November 2002 - Glasgow-haskell-users

RE: Template Haskell
by Simon Peyton-Jones 27 Nov '02

27 Nov '02

If you look in the manual you'll see that it says you can only compile-time-call a function that is in a separate module. So put 'pr/gen/parse' in a separate module and you'll be fine. The manual may not be very clear... pls help me improve it. S | -----Original Message----- | From: Mike Thomas [mailto:miketh@brisbane.paradigmgeo.com] | Sent: 27 November 2002 06:19 | To: glasgow-haskell-users(a)haskell.org | Subject: Template Haskell | | Hi there. | | Could somebody please let me know where I've gone wrong in the program below | (yesterday's CVS HEAD stage 3 compiler on Windows)? | | ------------- TH - printf.hs --- | | module Main where | | import Language.Haskell.THSyntax | | data Format = D | S | L String | | main = putStrLn ( $(pr "Hello") ) | | parse :: String -> [Format] | parse s = [ L s ] | | gen :: [Format] -> Expr | gen [D] = [| \n -> show n |] | gen [S] = [| \s -> s |] | gen [L s] = string s | | pr :: String -> Expr | pr s = gen (parse s) | | | ------------- Command Line ----- | | /c/cvs/i386-unknown-mingw32/stage3/ghc/compiler/ghc-inplace -fglasgow-exts - | package haskell-src printf.hs -o printf.exe | | ------------- GHC output--------- | | printf.hs:7: | Stage error: `pr' is bound at stage 1 but used at stage 0 | In the first argument of `putStrLn', namely `($[splice](pr "Hello"))' | In a right-hand side of function `main': | putStrLn ($[splice](pr "Hello")) | In the definition of `main': main = putStrLn ($[splice](pr "Hello")) | | ----------------------------------- | | Thanks | | Mike Thomas. | | | _______________________________________________ | Glasgow-haskell-users mailing list | Glasgow-haskell-users(a)haskell.org | http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

2 1

RE: incremental linking?
by Simon Marlow 27 Nov '02

27 Nov '02

> On Wed, Nov 27, 2002 at 09:50:56AM -0000, Simon Marlow wrote: > > > > More fun with Haskell-in-the-large: linking time has become the > > > main bottleneck in our development cycle. The standard solution > > > would be to use an incremental linker, but it seems that gnu does > > > not yet support this:-| > > > > Hmm, I've never heard of linking being a bottleneck. > > The runtime loader stuff I'm working on[1] takes around 10 > seconds to compile ... and 3 minutes to link it with libHSbase > and libHSrts. (This is on a 500MHz PIII). Linking is a huge > bottleneck once you start linking in the Haskell libraries; ld > takes up enormous amounts of CPU time resolving symbols, > I think. > > 1. > http://www.algorithm.com.au/wiki/hacking/haske> ll.ghc_runtime_loading 3 minutes???!! I just downloaded your example code, did './configure && make' and the link step took about 3 seconds. This is also on a 500MHz PIII. Are you sure you're not getting libHSbase over NFS? There may be something that ld is doing that causes a lot of NFS traffic. Cheers, Simon

3 2

Re: Native Threads in the RTS
by Wolfgang Thaller 27 Nov '02

27 Nov '02

Simon Marlow wrote: > I don't see the problem with forking a new Haskell thread for each > foreign export, and associating it with the current native thread if > the > foreign export is marked "bound". It does mean we can get multiple > Haskell threads bound to the same native thread, but only one can be > runnable at any one time (this is an important invariant from the point > of view of the implementation, I believe). Of course, you're right. Simon Peyton-Jones wrote: > I offer myself as such a guinea pig. I'm afraid I don't understand it > yet. It's hard to describe precisely. Comments below. Yes, I do need a guinea pig ;-) . I really have trouble expressing my ideas accurately, and I keep changing them. > Better start with some definitions, for > 'Haskell thread' > and 'native thread' > > I think I know what you mean, but better to be sure. Well, yes. I didn't manage to come up with a decent definition for it yet. I intend "native thread" to be the thing you get using pthread_create on unix, and "Haskell thread" to be the same thing as it currently is in the GHC RTS. Can anyone think of a way of defining that in a way that is accurate, general and understandable? > | If there is one, this Haskell thread is used to execute the callback. > > OK, so this is where I get completely confused. A Haskell thread is > not > (currently) an execution platform. Oops... The best way to confuse other people is to be confused yourself ;-) --- I had a slight misconception about the RTS there --- I think I've corrected that. =============================== Threads Proposal, version 4 Goals ~~~~~ Since foreign libraries sometimes exploit thread local state, it is necessary to provide some control over which thread is used to execute foreign code. In particular, it is important that it should be possible for Haskell code to arrange that a sequence of calls to a given library are performed by the same native thread and that if an external library calls into Haskell, then any outgoing calls from Haskell are performed by the same native thread. This specification is intended to be implementable both by multithreaded Haskell implementations and by single-threaded implementations and so it does not comment on which particular OS thread is used to execute Haskell code. Definitions ~~~~~~~~~~~ A native thread (aka OS thread) is a thread as defined by the operating system. A Haskell thread is [*** FIXME - How shall I put this? ***] the thing you see from Haskell land. Design ~~~~~~ Haskell threads may be associated at thread creation time with either zero or one native threads. Each Native thread is associated with zero or more native threads. If a native thread is associated with one or more Haskell threads, exactly one of the following must be true: *) Exactly one Haskell thread associated with the native thread is executing. *) The native thread is executing foreign code. *) The native thread and all Haskell threads associated with it are blocked. The thread that main runs in, threads created using forkIO and threads created for running finalizers or signal handlers are not necessarily associated with a native thread. However, an implementation might choose to do so. There are now two kinds of foreign exported [and foreign import wrapped] functions: bound and free. The FFI syntax should be extended appropriately [which of the two should be the default, if any?]. When a "bound" foreign exported function is invoked [by foreign code], a new Haskell thread is created and associated with the native thread. The new associated Haskell thread is then used to execute the callback. When a "free" foreign exported function is invoked, the implementation may freely choose what kind of Haskell thread the function is executed in. It is not specified whether this thread is associated with a particular OS thread or not. When a foreign imported function is invoked [by Haskell code], the foreign code is executed in the native thread associated with the current Haskell thread, if an association exists. If the current Haskell thread is not associated to a native thread, the implementation may freely decide which thread to run the foreign function in. The existing distinction between unsafe, safe and threadsafe calls remains unchanged. A new library routine, forkNativeThread :: IO () -> IO ThreadID, should spawn a new Haskell Thread (like forkIO) and associate it with a new native thread (forkIO is not guaranteed to do this). It may be implemented using the FFI and an OS-specific thread creation routine. It would just pass a "bound" callback as an entry point for a new OS thread. Issues ~~~~~~ Finalizers and signal handlers cannot be associated with a particular native thread. If they have to trigger an action in a particular native thread, a message has to be sent manually (via MVars and friends) to the Haskell thread associated with the native thread in question. This introduces a change in the syntax for foreign export and foreign import "wrapper" declarations (a bound/free specifier is added). I think we should have a default option here. I'm not sure which, however. Also, the objection that "bound" and "free" can be confused with the lambda calculus terms still holds.

1 0

RE: incremental linking?
by Simon Marlow 27 Nov '02

27 Nov '02

> More fun with Haskell-in-the-large: linking time has become the > main bottleneck in our development cycle. The standard solution > would be to use an incremental linker, but it seems that gnu does > not yet support this:-| Hmm, I've never heard of linking being a bottleneck. Even GHC itself links in about 3-4 seconds here. One common problem is that linking on a network filesystem takes a *lot* longer than linking objects from a local disk. It's always a good idea to keep the build tree on the local disk, even if the sources are NFS-mounted. > Alternative a: use someone else's incremental linker, e.g., Sun's > ild (ghc's -pgml option appears to have its own idea about option > formatting, btw) - this doesn't seem to work - should it? You'd probably want to call the incremental linker directly rather than using GHC - what exactly does it do, BTW? What files does it generate? > Alternative b: convince ghc to link objects in stages, e.g., on a > per-directory basis - gnu's ld seems to support at least this kind > of partial linking (-i/-r). Not quite as nice as a fully incremental > linker, but would probably save our day.. Yes, this works fine. We use it to build the libraries for GHCi. Cheers, Simon

3 2

RE: incremental linking?
by Simon Marlow 27 Nov '02

27 Nov '02

> Unfortunately, we're not talking seconds, but coffee-breaks of > linking times on our Sun (yes, the stuff is in the range of a large > compiler - we're fortunate enough to be able to build on rather > substantial third-party packages, think haskell-in-haskell frontend > distributed over unusually many modules + strategic traversal > support + our own code). > > And yes, I was worried about NFS-mounting first, especially since > linking on our Sun takes even longer than on our PCs (long breaks > instead of short ones;-), but moving .hi and .o to local tmp-space > didn't speed things up (then again, it's a large machine, and our > disk setup is likely to be more complex than I know - I'll have to > check with our admins). It does sound like it's taking rather too long, I'd investigate further. > > > Alternative a: use someone else's incremental linker, e.g., Sun's > > > ild (ghc's -pgml option appears to have its own idea about option > > > formatting, btw) - this doesn't seem to work - should it? > > > > You'd probably want to call the incremental linker directly > rather than > > using GHC - what exactly does it do, BTW? What files does > it generate? > > Calling it via GHC seemed the best way to ensure that it gets > everything it needs (what else would be the purpose of -pgml?). Ah I see - it's just like a normal linker, except it takes a previous version of the executable and attempts to just re-link the bits it needs? GHC has some built-in assumptions about the linker: it must accept gcc-style command line options. This probably isn't true of the incremental linker, so you could either (a) write a wrapper around it, or (b) arrange to call the incremental linker through gcc. I can't remember if (b) is possible, but a quick scan of the docs suggested that gcc doesn't have the equivalent of -pgml so (b) might be out. BTW, -pgml is usually just used to select a different version of gcc (as with -pgmc). > > Alternative b: convince ghc to link objects in stages, e.g., on a > > per-directory basis - gnu's ld seems to support at least this kind > > of partial linking (-i/-r). Not quite as nice as a fully incremental > > linker, but would probably save our day.. > > Yes, this works fine. We use it to build the libraries for GHCi. > > Presumably directed via Makefiles? > Could this please be automated for ghc --make? It's easy: $ ld -r -o All.o A.o B.o C.o ... Cheers, Simon

1 0

Re: incremental linking?
by C.Reinke 27 Nov '02

27 Nov '02

> Hmm, I've never heard of linking being a bottleneck. Even GHC itself > links in about 3-4 seconds here. One common problem is that linking on > a network filesystem takes a *lot* longer than linking objects from a > local disk. It's always a good idea to keep the build tree on the local > disk, even if the sources are NFS-mounted. Unfortunately, we're not talking seconds, but coffee-breaks of linking times on our Sun (yes, the stuff is in the range of a large compiler - we're fortunate enough to be able to build on rather substantial third-party packages, think haskell-in-haskell frontend distributed over unusually many modules + strategic traversal support + our own code). And yes, I was worried about NFS-mounting first, especially since linking on our Sun takes even longer than on our PCs (long breaks instead of short ones;-), but moving .hi and .o to local tmp-space didn't speed things up (then again, it's a large machine, and our disk setup is likely to be more complex than I know - I'll have to check with our admins). > > Alternative a: use someone else's incremental linker, e.g., Sun's > > ild (ghc's -pgml option appears to have its own idea about option > > formatting, btw) - this doesn't seem to work - should it? > > You'd probably want to call the incremental linker directly rather than > using GHC - what exactly does it do, BTW? What files does it generate? Calling it via GHC seemed the best way to ensure that it gets everything it needs (what else would be the purpose of -pgml?). According to docs, ild just keeps more information and space in the linked object, so that on re-linking, it can (a) check for file-modification times and (b) replace and partially relink only those contributing objects that have changed. http://docs.sun.com/db/doc/802-5693/6i9edqka5?l=zh&a=view > > Alternative b: convince ghc to link objects in stages, e.g., on a > > per-directory basis - gnu's ld seems to support at least this kind > > of partial linking (-i/-r). Not quite as nice as a fully incremental > > linker, but would probably save our day.. > > Yes, this works fine. We use it to build the libraries for GHCi. Presumably directed via Makefiles? Could this please be automated for ghc --make? Thanks, Claus

1 0

Re: Native Threads in the RTS
by Ashley Yakeley 27 Nov '02

27 Nov '02

At 2002-11-26 09:37, Alastair Reid wrote: > 1) forkNativeThread :: IO () -> IO () > The fresh Haskell thread is bound to a fresh native thread. > > 2) forkIO :: IO () -> IO () > The fresh Haskell thread is not bound to a native thread. Are you sure you intend to change the type of forkIO? Currently it's forkIO :: IO () -> IO ThreadId -- Ashley Yakeley, Seattle WA

2 1

RE: Native Threads in the RTS
by Simon Marlow 27 Nov '02

27 Nov '02

> When a "bound" foreign exported function is invoked [by > foreign code], > the implementation checks whether a Haskell thread is associated with > the current OS thread. > If there is one, this Haskell thread is used to execute the callback. > If there is none, a new Haskell thread is created and associated with > the native thread. This is the only situation where a Haskell > thread is > associated with a native thread. The new associated Haskell thread is > then used to execute the callback. When the callback finishes, the > Haskell thread is terminated, the association is dissolved, > but the OS thread continues to run. This is the bit I have trouble with too. If the OS thread trying to call a foreign export has an associated Haskell thread, then it must be because the Haskell thread called out to C in the first place. This Haskell thread will be waiting for the call to return, and has all the state associated with its current execution context, so we can't just use that thread to run the foreign export. I don't see the problem with forking a new Haskell thread for each foreign export, and associating it with the current native thread if the foreign export is marked "bound". It does mean we can get multiple Haskell threads bound to the same native thread, but only one can be runnable at any one time (this is an important invariant from the point of view of the implementation, I believe). Cheers, Simon

1 0

incremental linking?
by Claus Reinke 27 Nov '02

27 Nov '02

More fun with Haskell-in-the-large: linking time has become the main bottleneck in our development cycle. The standard solution would be to use an incremental linker, but it seems that gnu does not yet support this:-| Alternative a: use someone else's incremental linker, e.g., Sun's ild (ghc's -pgml option appears to have its own idea about option formatting, btw) - this doesn't seem to work - should it? Alternative b: convince ghc to link objects in stages, e.g., on a per-directory basis - gnu's ld seems to support at least this kind of partial linking (-i/-r). Not quite as nice as a fully incremental linker, but would probably save our day.. other alternatives? In other words: could ghc (please;-) support incremental linking? Cheers, Claus

1 0

RE: Native Threads in the RTS
by Simon Peyton-Jones 27 Nov '02

27 Nov '02

| Nice design, Alastair. I've stolen lots of ideas and some text for the | complete rewrite of the proposal. I think it is a great idea to rewrite the proposal. A tiny minority will follow the details of the discussion, and it's essential to record the outcome in a way comprehensible by someone who reads only the proposal. I offer myself as such a guinea pig. I'm afraid I don't understand it yet. It's hard to describe precisely. Comments below. I wonder if some simple formal semantics would be useful. Simon | Haskell threads may be associated at thread creation time with either | zero or one native threads. Each Native thread is associated with zero | or one Haskell threads (no native thread may be associated with two | Haskell threads at a time). Better start with some definitions, for 'Haskell thread' and 'native thread' I think I know what you mean, but better to be sure. | An associated pair of a Haskell thread and a native thread can only | execute either foreign code or haskell code at any one time. Unclear. Can we have a name for a Haskell thread that is 'associated with' a native thread? (Or is that misleading?) Is that what you mean by a 'bound Haskell thread'? If so then I think the above sentence means A bound Haskell thread can be executed only by its associated native thread. | There are now two kinds of foreign exported [and foreign import | wrapped] functions: bound and free [I'm not happy with these names, for | the same reasons as given by Seth before]. Do you mean that the syntax for 'foreign export' etc is modified to let you say bound/free? | When a "bound" foreign exported function is invoked [by foreign code], | the implementation checks whether a Haskell thread is associated with | the current OS thread. | If there is one, this Haskell thread is used to execute the callback. OK, so this is where I get completely confused. A Haskell thread is not (currently) an execution platform. It is spawned to execute a single IO action, and then dies, completely and forever. Perhaps you are saying that a bound Haskell thread does not vanish when it completes its IO action instead it waits around in case its associated OS thread wants it to do something If that is what you mean, better say so, because it's quite new. Suppose the Haskell thread calls *out* to foreign code. Then the Haskell thread is not available for to compute anything because it's suspended in the middle of a call. What happens if a bound Haskell thread blocks? Does its associated OS thread block too? Please say so. | If there is none, a new Haskell thread is created and associated with | the native thread. | This is the only situation where a Haskell thread is | associated with a native thread. Do you mean "This is the only way in which an association between a native thread and a Haskell thread can be created"? | The new associated Haskell thread is | then used to execute the callback. When the callback finishes, the | Haskell thread is terminated, the association is dissolved, but the OS | thread continues to run. So, returning to "the implementation checks whether a Haskell thread is associated with the current OS thread", how could this ever happen? The Haskell thread dies when the foreign call terminates. | When a foreign imported function is invoked [by Haskell code], the | foreign code is executed in the native thread associated with the | current Haskell thread, if an association exists. If the current | Haskell thread is not associated to a native thread, the implementation | may freely decide which thread to run the foreign function in. | This must be done for all foreign imports, including "unsafe". What is "this"? | An unsafe call must not call back to Haskell or otherwise cause a | garbage collection. Other Haskell threads may be blocked while the | unsafe call executes. | A safe call allows callbacks to Haskell. Other Haskell threads may be | blocked while the unsafe call executes. | A threadsafe call additionally guarantees other Haskell threads not to | block. ...not to block, even if the foreign call blocks in foreign code. | forkNativeThread :: IO () -> IO ThreadID may be implemented using the | FFI and an OS-specific thread creation routine. Better say what you expect forkNativeThread to do, and what its implementation looks like.

1 0