- Glasgow-haskell-users

RE: -Werror Request
by Simon Peyton-Jones 20 Nov '02

20 Nov '02

Done! SImon | -----Original Message----- | From: Ashley Yakeley [mailto:ashley@semantic.org] | Sent: 14 November 2002 02:25 | To: GHC List | Subject: -Werror Request | | If it's not too much work, I'd like to request a "-Werror" option for GHC | that would turn warnings into errors. Sometimes warnings one would like | to catch get lost in a long make process. | | Thanks... | | -- | Ashley Yakeley, Seattle WA | | _______________________________________________ | Glasgow-haskell-users mailing list | Glasgow-haskell-users(a)haskell.org | http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

2 1

RE: adding new primitives to ghci...
by Simon Marlow 20 Nov '02

20 Nov '02

> I am trying to add a new primitive to ghc (5.04.1) ... > I followed > the instructions in primops.txt.pp: > > -- - or, for an out-of-line primop: > -- ghc/includes/PrimOps.h (just add the declaration) > -- ghc/rts/PrimOps.hc (define it here) > -- ghc/rts/Linker.c (declare the symbol for GHCi) > -- > > My new primitive works fine when I compile the programs that > use it but when I try to load ghci I get: > > lxdsg[137](2.05)> ./ghc-inplace --interactive > ___ ___ _ > / _ \ /\ /\/ __(_) > / /_\// /_/ / / | | GHC Interactive, version 5.04.1, > for Haskell > 98. > / /_\\/ __ / /___| | http://www.haskell.org/ghc/ > \____/\/ /_/\____/|_| Type :? for help. > > Loading package base ... linking ... > /data/dubois/ghc-5.04.1/libraries/base/HSbase.o: unknown symbol > `packcczh_fast' > ghc-5.04.1: panic! (the `impossible' happened, GHC version 5.04.1): > can't load package `base' > > Please report it as a compiler bug to > glasgow-haskell-bugs(a)haskell.org, > or http://sourceforge.net/projects/ghc/. You need to bootstrap GHCi with the compiler that contains the new primitive. Cheers, Simon

1 0

adding new primitives to ghci...
by Andre Rauber Du Bois 20 Nov '02

20 Nov '02

Hello! I am trying to add a new primitive to ghc (5.04.1) ... I followed the instructions in primops.txt.pp: -- - or, for an out-of-line primop: -- ghc/includes/PrimOps.h (just add the declaration) -- ghc/rts/PrimOps.hc (define it here) -- ghc/rts/Linker.c (declare the symbol for GHCi) -- My new primitive works fine when I compile the programs that use it but when I try to load ghci I get: lxdsg[137](2.05)> ./ghc-inplace --interactive ___ ___ _ / _ \ /\ /\/ __(_) / /_\// /_/ / / | | GHC Interactive, version 5.04.1, for Haskell 98. / /_\\/ __ / /___| | http://www.haskell.org/ghc/ \____/\/ /_/\____/|_| Type :? for help. Loading package base ... linking ... /data/dubois/ghc-5.04.1/libraries/base/HSbase.o: unknown symbol `packcczh_fast' ghc-5.04.1: panic! (the `impossible' happened, GHC version 5.04.1): can't load package `base' Please report it as a compiler bug to glasgow-haskell-bugs(a)haskell.org, or http://sourceforge.net/projects/ghc/. So, is there any other file that I have to change to have my new primitive working with ghci? Cheers, Andre. -- --------------------------------- André Rauber Du Bois dubois at macs.hw.ac.uk http://www.macs.hw.ac.uk/~dubois/

1 0

Re: Native Threads in the RTS
by Wolfgang Thaller 19 Nov '02

19 Nov '02

I've now written up a slightly more formal proposal for native threads. (OK, it's only a tiny bit more formal...) I doubt I have explained everything clearly, please tell me which points are unclear. And of course please tell me what you like/don't like about it. I have some rough ideas on how to implement the proposal. I would be ready to invest some time, but I don't have enough free time to make any promises here. The discussion has to be finished first, anyway. Cheers, Wolfgang ******************* Native Threads Proposal, version 1 Some "foreign" libraries (for example OpenGL) rely on a mechanism called thread-local storage. The meaning of an OpenGL call therefore usually depends on which OS thread it is called from. Therefore, some kind of direct mapping from Haskell threads to OS threads is necessary in order to use the affected foreign libraries. Executing every haskell thread in its own OS thread is not feasible for performance reasons. However, perfomance of native OS threads is not too bad as long as there aren't too many, so I propose that some threads get their own OS threads, and some don't: Every Haskell Thread can be either a "green" thread or a "native" thread. For each "native" thread, there is exactly one OS thread created by the RTS. For a green thread, it is unspecified which OS thread it is executed in. The main program and all haskell threads forked using forkIO are green threads. Threads forked using forkNativeThread :: IO () -> IO () are native threads. Execution of a green thread might move from one OS thread to another at any time. A "green" thread is never executed in an OS thread that is reserved for a "native" thread. A "native" haskell thread and all foreign imported functions that it calls are executed in its associated OS thread. A foreign exported callback that is called from C code executing in that OS thread is executed in the native haskell thread. A foreign exported callback that is called from C code executing in an OS thread that is not associated with a "native" haskell thread is executed in a new green haskell thread. Only one OS thread can execute Haskell code at any given time. If a "native" haskell thread enters a foreign imported function that is marked as "safe" or "threadsafe", all other Haskell threads keep running. If the imported function is marked as "unsafe", no other threads are executed until the call finishes. If a "green" haskell thread enters a foreign imported function marked as "threadsafe", a new OS thread is spawned that keeps executing other green haskell threads while the foreign function executes. Native haskell threads continue to run in their own OS threads. If a "green" haskell thread enters a foreign imported function marked as "safe", all other green threads are blocked. Native haskell threads continue to run in their own OS threads. If the imported function is marked as "unsafe", no other threads are executed until the call finishes. Finalizers are always run in green threads. Issues deliberately not addressed in this proposal: Some people may want to run several Haskell threads in a dedicated OS thread (this is what has been called "thread groups" before). Some people may want to run finalizers in specific OS threads (are finalizers predictable enough for this to be useful?). Everyone would want SMP if it came for free (but SMP seems to be too hard to do at the moment...) Other things I'm not sure about: What should we do get if a foreign function spawns a new OS thread and executes a haskell callback in that OS thread? Should a new native haskell thread that executes in the OS thread be created? Should the new OS thread be blocked and the callback executed in a green thread? What does the current threaded RTS do? (I assume the non-threaded RTS will just crash?)

1 0

RE: Strings are slow
by Simon Marlow 19 Nov '02

19 Nov '02

> module Main where > > import Text.Regex > import System.IO > import System.Environment > import Control.Monad > import Data.Maybe > > main = do [re] <- getArgs > let rx = mkRegex re > let loop = do line <- getLine > when (isJust (matchRegex rx line)) > (putStrLn line) > eof <- isEOF > unless eof loop > loop > > > It turned out that this is remarkably slow. The first problem was with > inlining. If this is compiled with ghc-5.04 -O -ddump-simpl, I get: > > case GHC.IOBase.unsafePerformIO > @ (Data.Maybe.Maybe > (GHC.Base.String, > GHC.Base.String, > GHC.Base.String, > [GHC.Base.String])) > (Text.Regex.Posix.regexec > (Text.Regex.mkRegex re) a731) > of wild4 { This is indeed an optimiser bug, but it's the result of a design decision: GHC is a bit laid back about inlining things inside the state lambda in the IO monad, because it often enables important optimisations. However, we're experimenting with modifying this "optimisation" so that it will be less likely to kill performance in the way it did in your example. In the meantime, you can add rx as an argument to loop, that will be enough to fool GHC into not inlining rx. > Ie. the regex is compiled anew every time a string is matched. A bug? > > Anyway, without optimization the code produced is reasonable, > but still > horrendously slow. Testing with a simple word as a pattern > from a 7.3MB, > 800kline file, the running time was 37.5 seconds. For > comparison, a similar > program in mzscheme (interpreted!) took 7.3 seconds while the system > grep, of course, took 0.4 seconds. > > I did some profiling by creating new top-level bindings for matchRegex > and getLine (is there a better way?): > > > total time = 53.34 secs (2667 ticks @ 20 ms) > total alloc = 1,172,482,496 bytes (excludes > profiling overheads) > > COST CENTRE MODULE %time %alloc > > match Main 69.7 56.8 > getl Main 23.9 40.2 > main Main 6.3 3.0 > > > So it seems like all the time is spent just converting ByteArrays to > char lists to C arrays. This makes me wonder how sensible it really is > to represent strings by char lists. Yes, it's nice and > uniform and lazy, but... String processing in Haskell is very slow, due to the list-of-characters representation. A more complete PackedString library with better integration with other libraries (like Text.Regex) would help a lot for these kind of examples. Cheers, Simon

1 0

RE: first stab at -ffunction-sections
by Simon Marlow 19 Nov '02

19 Nov '02

> I noticed a lot of not-obviously-used stuff brought in from various > libraries and wanted to nuke some of the unneeded things. Step 1 was > trying to compile the libraries with the option, which didn't quite > fly... it looks like ghc-asm is the primary sufferer, and I'm not sure > the compiler option is needed... Yes, indeed you could do this in ghc-asm. But don't forget the native code generator too... > -split-objs I didn't really realize was there. I see (tracing > through ghc5, whatever debian's latest shipping version is): > > ghc/compiler/main/DriverFlags.hs:250 > , ( "split-objs" , NoArg (if can_split > then do writeIORef > v_Split_object_files True > add v_Opt_C > "-fglobalise-toplev-name > s" > else hPutStrLn stderr > "warning: don't > know how to split \ > \object files on > this architecture" > ) ) -split-objs has to go to some trouble to make more symbols global so that they can still be resolved after the assembler file has been split into chunks. This is one rather ugly hack that it would be nice to get rid of. > Then in ghc/driver/split/ghc-split.lprl:287 (there's actually > one per arch): > > # strip the marker > > $str =~ s/(\.text\n\t\.align > .(,0x90)?\n)\.globl\s+.*_?__stg_split_marker.*\ > n/$1/; > $str =~ s/(\t\.align > .(,0x90)?\n)\.globl\s+.*_?__stg_split_marker.*\n/$1/; > > ... Yes, this is all particularly horrible too. The splitter has to go to some trouble to make sure that "local constants" (strings and the like) get duplicated in each assembler chunk that refers to them, subverting the commoning up of constants that gcc does. > So to me it looks feasible to figure out who's fooling with these > things, though it's probably not necessary to do any of this > within the > compiler except for whatever might circumvent ghc-asm, if anything. > > At any rate, I am finding the amount of unused code/data linked into > the generated executables significant... for instance, in a non- > concurrent program: > > 080a1c64 D MVar_modifyMVarzu_closure > 0805aeb8 T MVar_modifyMVarzu_entry > 0805aece T MVar_modifyMVarzu_fast3 > 0805aeb8 T MVar_modifyMVarzu_info The IO library requires MVars (it's thread-safe) so most programs will have some MVar bits linked in. I'm not sure why modifyMVar in particular is being included though. > ... and as it's a 9-line script to mangle patches, it's certainly not > using this: > > 0805b140 T __stginit_PosixDB > > The idea with -ffunction-sections or brewing up an equivalent is to > build the libraries with it so when the final executable is linked, it > imports only the code and statically-allocated data it uses from them. Ok, I'm convinced :) Let us know if you need any more help. Cheers, Simon

1 0

RE: unsafePerformIO and IORefs
by Simon Marlow 19 Nov '02

19 Nov '02

> Hal Daume III wrote: > > You can't. [...] > > Well, you can, but only for CAFs. This idiom/hack is used > quite happily throughout GHC, HOpenGL, H/Direct, ... I think "quite happily" is a bit strong ;-) We'd much rather have a safe way to do what is really quite a reasonable thing. Cheers, Simon

1 0

RE: unsafePerformIO and IORefs
by Simon Marlow 19 Nov '02

19 Nov '02

> At 2002-11-18 11:05, Sven Panne wrote: > > >global :: a -> IORef a > >global a = unsafePerformIO (newIORef a) > > This is useful, you can do this with it: > > ref = global Nothing > > convert :: a -> IO b > convert a = do > writeIORef ref (Just a) > Just b <- readIORef ref > return b This particular "flexibility" provided by unsafePerformIO is actually documented... http://www.haskell.org/ghc/docs/latest/html/base/System.IO.Unsafe.html#u nsafePerformIO Cheers, Simon

1 0

Strings are slow
by Lauri Alanko 19 Nov '02

19 Nov '02

Hello. Here's a grep program: module Main where import Text.Regex import System.IO import System.Environment import Control.Monad import Data.Maybe main = do [re] <- getArgs let rx = mkRegex re let loop = do line <- getLine when (isJust (matchRegex rx line)) (putStrLn line) eof <- isEOF unless eof loop loop It turned out that this is remarkably slow. The first problem was with inlining. If this is compiled with ghc-5.04 -O -ddump-simpl, I get: case GHC.IOBase.unsafePerformIO @ (Data.Maybe.Maybe (GHC.Base.String, GHC.Base.String, GHC.Base.String, [GHC.Base.String])) (Text.Regex.Posix.regexec (Text.Regex.mkRegex re) a731) of wild4 { Ie. the regex is compiled anew every time a string is matched. A bug? Anyway, without optimization the code produced is reasonable, but still horrendously slow. Testing with a simple word as a pattern from a 7.3MB, 800kline file, the running time was 37.5 seconds. For comparison, a similar program in mzscheme (interpreted!) took 7.3 seconds while the system grep, of course, took 0.4 seconds. I did some profiling by creating new top-level bindings for matchRegex and getLine (is there a better way?): total time = 53.34 secs (2667 ticks @ 20 ms) total alloc = 1,172,482,496 bytes (excludes profiling overheads) COST CENTRE MODULE %time %alloc match Main 69.7 56.8 getl Main 23.9 40.2 main Main 6.3 3.0 So it seems like all the time is spent just converting ByteArrays to char lists to C arrays. This makes me wonder how sensible it really is to represent strings by char lists. Yes, it's nice and uniform and lazy, but... How can I get this faster, then? PackedStrings are not very useful because they just don't support enough operations (getline, matchregex) alone, and having to convert them to Strings sort of defeats their purpose. _Any_ operation that provides only a String interface condemns us to a gazillion allocations. And by default all char*-based foreign interfaces are represented with Strings on the Haskell side. Maybe a generic Textual class with at least String and PackedString (and ByteArray?) as instances would help? Then the common string-based operations could all have separate implementations for separate representations. With heavy specialization, of course. This would be especially useful if the FFI (especially withCString) supported it. Or alternatively, maybe the foldr/build rewriting trick could be used to eliminate some redundant conversions between representations? Just throwing ideas in the air here. Lauri Alanko la(a)iki.fi

1 0

Native Threads in the RTS
by Wolfgang Thaller 19 Nov '02

19 Nov '02

Hello All, A while ago there was a discussion on the shortcomings of the threaded RTS (in short, it doesn't work with foreign APIs that use thread-local state, and that breaks HOpenGL). Back then, it was decided to just keep the threaded RTS off by default and to do something about it some time after 5.04. I believe it's time to think about it again, so I'll take the liberty of proposing an extension to the RTS that might solve the problem. I propose adding something like forkNativeThread :: IO () -> IO () which forks a new Haskell thread that has its own OS thread to execute in. Note that the fact that only one Haskell thread may execute at a time remains unchanged. Whenever the scheduler determines that a "native" haskell thread is next, it sends the OS worker thread to sleep and wakes up the OS thread corresponding to the "native" haskell thread. When the "native" haskell thread yields again, so does the corresponding OS thread. Foreign calls from "normal" (non-native) haskell threads should be handled in exactly the same way as they are currently. If a callback is entered and the current OS thread corresponds to a native haskell thread, the callback should be executed in the current OS thread. Other haskell threads continue to run in the worker thread or in their own dedicated OS thread. Programs that don't use forkNativeThread won't be affected by the change. Thread switching to and from native threads will be slower, but not painfully slow. Wrapping an entire HOpenGL program in forkNativeThread should solve the OpenGL/GLUT thread-local-state problem, for example, and who knows what else it is good for. Any comments? Opinions? Wolfgang

5 8