- Glasgow-haskell-users

Re: incremental linking?
by C.Reinke 27 Nov '02

27 Nov '02

> Hmm, I've never heard of linking being a bottleneck. Even GHC itself > links in about 3-4 seconds here. One common problem is that linking on > a network filesystem takes a *lot* longer than linking objects from a > local disk. It's always a good idea to keep the build tree on the local > disk, even if the sources are NFS-mounted. Unfortunately, we're not talking seconds, but coffee-breaks of linking times on our Sun (yes, the stuff is in the range of a large compiler - we're fortunate enough to be able to build on rather substantial third-party packages, think haskell-in-haskell frontend distributed over unusually many modules + strategic traversal support + our own code). And yes, I was worried about NFS-mounting first, especially since linking on our Sun takes even longer than on our PCs (long breaks instead of short ones;-), but moving .hi and .o to local tmp-space didn't speed things up (then again, it's a large machine, and our disk setup is likely to be more complex than I know - I'll have to check with our admins). > > Alternative a: use someone else's incremental linker, e.g., Sun's > > ild (ghc's -pgml option appears to have its own idea about option > > formatting, btw) - this doesn't seem to work - should it? > > You'd probably want to call the incremental linker directly rather than > using GHC - what exactly does it do, BTW? What files does it generate? Calling it via GHC seemed the best way to ensure that it gets everything it needs (what else would be the purpose of -pgml?). According to docs, ild just keeps more information and space in the linked object, so that on re-linking, it can (a) check for file-modification times and (b) replace and partially relink only those contributing objects that have changed. http://docs.sun.com/db/doc/802-5693/6i9edqka5?l=zh&a=view > > Alternative b: convince ghc to link objects in stages, e.g., on a > > per-directory basis - gnu's ld seems to support at least this kind > > of partial linking (-i/-r). Not quite as nice as a fully incremental > > linker, but would probably save our day.. > > Yes, this works fine. We use it to build the libraries for GHCi. Presumably directed via Makefiles? Could this please be automated for ghc --make? Thanks, Claus

1 0

Re: Native Threads in the RTS
by Ashley Yakeley 27 Nov '02

27 Nov '02

At 2002-11-26 09:37, Alastair Reid wrote: > 1) forkNativeThread :: IO () -> IO () > The fresh Haskell thread is bound to a fresh native thread. > > 2) forkIO :: IO () -> IO () > The fresh Haskell thread is not bound to a native thread. Are you sure you intend to change the type of forkIO? Currently it's forkIO :: IO () -> IO ThreadId -- Ashley Yakeley, Seattle WA

2 1

RE: Native Threads in the RTS
by Simon Marlow 27 Nov '02

27 Nov '02

> When a "bound" foreign exported function is invoked [by > foreign code], > the implementation checks whether a Haskell thread is associated with > the current OS thread. > If there is one, this Haskell thread is used to execute the callback. > If there is none, a new Haskell thread is created and associated with > the native thread. This is the only situation where a Haskell > thread is > associated with a native thread. The new associated Haskell thread is > then used to execute the callback. When the callback finishes, the > Haskell thread is terminated, the association is dissolved, > but the OS thread continues to run. This is the bit I have trouble with too. If the OS thread trying to call a foreign export has an associated Haskell thread, then it must be because the Haskell thread called out to C in the first place. This Haskell thread will be waiting for the call to return, and has all the state associated with its current execution context, so we can't just use that thread to run the foreign export. I don't see the problem with forking a new Haskell thread for each foreign export, and associating it with the current native thread if the foreign export is marked "bound". It does mean we can get multiple Haskell threads bound to the same native thread, but only one can be runnable at any one time (this is an important invariant from the point of view of the implementation, I believe). Cheers, Simon

1 0

incremental linking?
by Claus Reinke 27 Nov '02

27 Nov '02

More fun with Haskell-in-the-large: linking time has become the main bottleneck in our development cycle. The standard solution would be to use an incremental linker, but it seems that gnu does not yet support this:-| Alternative a: use someone else's incremental linker, e.g., Sun's ild (ghc's -pgml option appears to have its own idea about option formatting, btw) - this doesn't seem to work - should it? Alternative b: convince ghc to link objects in stages, e.g., on a per-directory basis - gnu's ld seems to support at least this kind of partial linking (-i/-r). Not quite as nice as a fully incremental linker, but would probably save our day.. other alternatives? In other words: could ghc (please;-) support incremental linking? Cheers, Claus

1 0

RE: Native Threads in the RTS
by Simon Peyton-Jones 27 Nov '02

27 Nov '02

| Nice design, Alastair. I've stolen lots of ideas and some text for the | complete rewrite of the proposal. I think it is a great idea to rewrite the proposal. A tiny minority will follow the details of the discussion, and it's essential to record the outcome in a way comprehensible by someone who reads only the proposal. I offer myself as such a guinea pig. I'm afraid I don't understand it yet. It's hard to describe precisely. Comments below. I wonder if some simple formal semantics would be useful. Simon | Haskell threads may be associated at thread creation time with either | zero or one native threads. Each Native thread is associated with zero | or one Haskell threads (no native thread may be associated with two | Haskell threads at a time). Better start with some definitions, for 'Haskell thread' and 'native thread' I think I know what you mean, but better to be sure. | An associated pair of a Haskell thread and a native thread can only | execute either foreign code or haskell code at any one time. Unclear. Can we have a name for a Haskell thread that is 'associated with' a native thread? (Or is that misleading?) Is that what you mean by a 'bound Haskell thread'? If so then I think the above sentence means A bound Haskell thread can be executed only by its associated native thread. | There are now two kinds of foreign exported [and foreign import | wrapped] functions: bound and free [I'm not happy with these names, for | the same reasons as given by Seth before]. Do you mean that the syntax for 'foreign export' etc is modified to let you say bound/free? | When a "bound" foreign exported function is invoked [by foreign code], | the implementation checks whether a Haskell thread is associated with | the current OS thread. | If there is one, this Haskell thread is used to execute the callback. OK, so this is where I get completely confused. A Haskell thread is not (currently) an execution platform. It is spawned to execute a single IO action, and then dies, completely and forever. Perhaps you are saying that a bound Haskell thread does not vanish when it completes its IO action instead it waits around in case its associated OS thread wants it to do something If that is what you mean, better say so, because it's quite new. Suppose the Haskell thread calls *out* to foreign code. Then the Haskell thread is not available for to compute anything because it's suspended in the middle of a call. What happens if a bound Haskell thread blocks? Does its associated OS thread block too? Please say so. | If there is none, a new Haskell thread is created and associated with | the native thread. | This is the only situation where a Haskell thread is | associated with a native thread. Do you mean "This is the only way in which an association between a native thread and a Haskell thread can be created"? | The new associated Haskell thread is | then used to execute the callback. When the callback finishes, the | Haskell thread is terminated, the association is dissolved, but the OS | thread continues to run. So, returning to "the implementation checks whether a Haskell thread is associated with the current OS thread", how could this ever happen? The Haskell thread dies when the foreign call terminates. | When a foreign imported function is invoked [by Haskell code], the | foreign code is executed in the native thread associated with the | current Haskell thread, if an association exists. If the current | Haskell thread is not associated to a native thread, the implementation | may freely decide which thread to run the foreign function in. | This must be done for all foreign imports, including "unsafe". What is "this"? | An unsafe call must not call back to Haskell or otherwise cause a | garbage collection. Other Haskell threads may be blocked while the | unsafe call executes. | A safe call allows callbacks to Haskell. Other Haskell threads may be | blocked while the unsafe call executes. | A threadsafe call additionally guarantees other Haskell threads not to | block. ...not to block, even if the foreign call blocks in foreign code. | forkNativeThread :: IO () -> IO ThreadID may be implemented using the | FFI and an OS-specific thread creation routine. Better say what you expect forkNativeThread to do, and what its implementation looks like.

1 0

RE: Native Threads in the RTS
by Simon Marlow 27 Nov '02

27 Nov '02

Alastair Reid wrote: > Design > ~~~~~~ > > Haskell threads may be associated at thread creation time with either > zero or one native threads. There are only two ways to create Haskell > threads so there are two cases to consider: Umm, Alastair, I think you've got things a bit mixed up here. Did you mean two ways to create a native thread? There are currently three ways to create a Haskell thread (forkIO, foreign export, finalizers) and Wolfgang has proposed a fourth (forkNativeThread). Or perhaps you're proposing we do away with forkIO and only have forkNativeThread? That was discounted a while back because we want to continue to have light-weight threads independent of OS threads. > 1) forkNativeThread :: IO () -> IO () > > The fresh Haskell thread is associated with a fresh native thread. > > (ToDo: do we really need to use a fresh native thread or would a > pool of threads be ok? The issue could be avoided by separating > creation of the native thread from the 'associate' operation.) > > 2) Calls to a threadsafe foreign export allocate a fresh Haskell > thread which is then associated with the Haskell thread. I don't know what you mean by a "threadsafe foreign export". Did you mean "threadsafe foreign import" perhaps? And I'm not sure how a fresh Haskell thread is associated with a Haskell thread ;-) > Calls to threadsafe foreign imports by threads which have an > associated native thread are performed by that native thread. > > Calls to any other foreign imports (i.e., 'safe' or 'unsafe' calls) > may be made in other threads or, if it exists, in the associated > native thread at the implementation's discretion. > > [ToDo: can Haskell threads with no associated thread make foreign > calls using a thread associated with some other thread? er... overloading the word "thread" was maybe not such a good idea. I think you're asking whether a green thread can grab a native thread to make a foreign call. The current answer is no... but can you think of a reason we might want this feature? Cheers, Simon

4 4

Template Haskell
by Mike Thomas 27 Nov '02

27 Nov '02

Hi there. Could somebody please let me know where I've gone wrong in the program below (yesterday's CVS HEAD stage 3 compiler on Windows)? ------------- TH - printf.hs --- module Main where import Language.Haskell.THSyntax data Format = D | S | L String main = putStrLn ( $(pr "Hello") ) parse :: String -> [Format] parse s = [ L s ] gen :: [Format] -> Expr gen [D] = [| \n -> show n |] gen [S] = [| \s -> s |] gen [L s] = string s pr :: String -> Expr pr s = gen (parse s) ------------- Command Line ----- /c/cvs/i386-unknown-mingw32/stage3/ghc/compiler/ghc-inplace -fglasgow-exts - package haskell-src printf.hs -o printf.exe ------------- GHC output--------- printf.hs:7: Stage error: `pr' is bound at stage 1 but used at stage 0 In the first argument of `putStrLn', namely `($[splice](pr "Hello"))' In a right-hand side of function `main': putStrLn ($[splice](pr "Hello")) In the definition of `main': main = putStrLn ($[splice](pr "Hello")) ----------------------------------- Thanks Mike Thomas.

1 0

Re: Native Threads in the RTS
by Wolfgang Thaller 26 Nov '02

26 Nov '02

Nice design, Alastair. I've stolen lots of ideas and some text for the complete rewrite of the proposal. The concept of "associating" haskell threads to native threads proved to be a good way of explaining my original idea in a different way --- and then I found out that forkNativeThread needn't be a primitive, but can be implemented on top of the FFI. After that I found out what the bound/free exports buisiness was all about and why we might need it. As for the questions regarding my previous proposal, I think they are answered in the new one. If they aren't, let me know. [It's no use explaining what I meant to say if I already want to say something different ;-) ] Cheers, Wolfgang =============================== Threads Proposal, version 3 Goals ~~~~~ Since foreign libraries sometimes exploit thread local state, it is necessary to provide some control over which thread is used to execute foreign code. In particular, it is important that it should be possible for Haskell code to arrange that a sequence of calls to a given library are performed by the same native thread and that if an external library calls into Haskell, then any outgoing calls from Haskell are performed by the same native thread. This specification is intended to be implementable both by multithreaded Haskell implementations and by single-threaded implementations and so it does not comment on which particular OS thread is used to execute Haskell code. Design ~~~~~~ Haskell threads may be associated at thread creation time with either zero or one native threads. Each Native thread is associated with zero or one Haskell threads (no native thread may be associated with two Haskell threads at a time). An associated pair of a Haskell thread and a native thread can only execute either foreign code or haskell code at any one time. The thread that main runs in, threads created using forkIO and threads created for running finalizers or signal handlers are not associated with a native thread [Actually, we might make this an implementation detail: These Haskell threads are not guaranteed to be associated with a native thread. If it makes sense for some implementation, all threads might be associated with native threads]. There are now two kinds of foreign exported [and foreign import wrapped] functions: bound and free [I'm not happy with these names, for the same reasons as given by Seth before]. When a "bound" foreign exported function is invoked [by foreign code], the implementation checks whether a Haskell thread is associated with the current OS thread. If there is one, this Haskell thread is used to execute the callback. If there is none, a new Haskell thread is created and associated with the native thread. This is the only situation where a Haskell thread is associated with a native thread. The new associated Haskell thread is then used to execute the callback. When the callback finishes, the Haskell thread is terminated, the association is dissolved, but the OS thread continues to run. When a "free" foreign exported function is invoked, the implementation may freely choose what Haskell thread the function is executed in. It is not specified whether this thread is associated with a particular OS thread or not. When a foreign imported function is invoked [by Haskell code], the foreign code is executed in the native thread associated with the current Haskell thread, if an association exists. If the current Haskell thread is not associated to a native thread, the implementation may freely decide which thread to run the foreign function in. This must be done for all foreign imports, including "unsafe". An unsafe call must not call back to Haskell or otherwise cause a garbage collection. Other Haskell threads may be blocked while the unsafe call executes. A safe call allows callbacks to Haskell. Other Haskell threads may be blocked while the unsafe call executes. A threadsafe call additionally guarantees other Haskell threads not to block. forkNativeThread :: IO () -> IO ThreadID may be implemented using the FFI and an OS-specific thread creation routine. Issues ~~~~~~ Finalizers and signal handlers cannot be associated with a particular native thread. If they have to trigger an action in a particular native thread, a message has to be sent manually (via MVars and friends) to the Haskell thread associated with the native thread in question. This might be tedious, but if we want to avoid the nastiness of multiple Haskell threads associated with one OS thread (lots of unpredictable blocking), it looks like our only choice.

1 0

foldl' ?
by Mark Carroll 26 Nov '02

26 Nov '02

Where do I find foldl' in GHC? It's mentioned on http://www.haskell.org/ghc/docs/latest/html/base/Data.List.html but importing List and using "-package data" don't seem to make it appear. I'm using GHC 5.02.2. I must be making some simple mistake. -- Mark

3 5

RE: Native Threads in the RTS
by Simon Marlow 26 Nov '02

26 Nov '02

> ***************** > Native Threads Proposal, version 2 > > Some "foreign" libraries (for example OpenGL) rely on a mechanism > called thread-local storage. The meaning of an OpenGL call therefore > usually depends on which OS thread it is called from. > Therefore, some > kind of direct mapping from Haskell threads to OS threads is > necessary > in order to use the affected foreign libraries. > Executing every haskell thread in its own OS thread is not > feasible for > performance reasons. However, perfomance of native OS threads is not > too bad as long as there aren't too many, so I propose that some > threads get their own OS threads, and some don't: > > Every Haskell Thread can be either a "green" thread or a "native" > thread. > For each "native" thread, there is exactly one OS thread > created by the > RTS. Is there an invariant that there is only one Haskell thread for each native thread? > For a green thread, it is unspecified which OS thread it is > executed in. > The main program and all haskell threads forked using forkIO > are green > threads. Threads forked using forkNativeThread :: IO () -> IO () are > native threads. (Note: The type of the current haskell thread does > _not_ matter when forking new threads) > > Execution of a green thread might move from one OS thread to > another at > any time. A "green" thread is never executed in an OS thread that is > reserved for a "native" thread. > A "native" haskell thread and all foreign imported functions that it > calls are executed in its associated OS thread. A foreign exported > callback that is called from C code executing in that OS thread is > executed in the native haskell thread. > A foreign exported callback that is called from C code > executing in an > OS thread that is not associated with a "native" haskell thread is > executed in a new green haskell thread. > > Only one OS thread can execute Haskell code at any given time. > > If a "native" haskell thread enters a foreign imported > function that is > marked as "safe" or "threadsafe", all other Haskell threads keep > running. If the imported function is marked as "unsafe", no other > threads are executed until the call finishes. I'd change this paragraph as follows: If a "native" haskell thread enters a foreign imported function that is marked as "threadsafe", all other Haskell threads keep running. If the imported function is marked as "safe" or "unsafe", no other threads are executed until the call finishes. in other words, foreign imports for native threads behave just like they do for green threads. As Simon P.J. pointed out, we don't do any OS-thread synchronisation on a "safe" foreign import. > Other things I'm not sure about: > What should we do get if a foreign function spawns a new OS > thread and > executes a haskell callback in that OS thread? Should a new native > haskell thread that executes in the OS thread be created? Should the > new OS thread be blocked and the callback executed in a green > thread? > What does the current threaded RTS do? (I assume the > non-threaded RTS will just crash?) The Haskell callback will execute in a green thread, according to the proposal above, and I think that's the right thing to do. It's possible to get into a situation in which several Haskell threads in the system are associated with the same OS thread: - Haskell native thread calls a foreign import, which - invokes a Haskell callback, which - calls a foreign import, which - invokes another Haskell callback at this point the two Haskell threads created by the callbacks are both attached to the same native thread. One of them is blocked waiting for its foreign import to return, however. I'm not sure if this is important. Cheers, Simon

1 0