August 2002 - Glasgow-haskell-users

RE: RFC: ghc's dynamic linker
by Simon Peyton-Jones 28 Aug '02

28 Aug '02

| Going the final step and allowing linkage to the current | binary is possible, it just means the linker has to know how | to read the symbol table out of the binary, and you have to | avoid running 'strip'. I believe reading the symbol table is | quite straightforward, the main problem being that on Unix | you don't actually know where the binary lives, so you have | to wire it in or search the PATH. | | Another problem is that you don't normally link a *complete* | copy of the base package into your binary, you only link the | bits you need. Linking the whole lot would mean every binary | would be about 10M; but doing this on the basis of a flag | which you turn on when you want to do dynamic linking maybe | isn't so bad. There is another option too, which I find quite attractive. To the interface that Simon mentioned: | loadLibrary :: FilePath -> IO () | lookupEntity :: String -> IO a add one more call: addEntity :: String -> a -> IO () The effect is to extend GHCi's dynamic linker's symbol table with an extra binding. So that 'lookupEntity' of the same string would return the value you stuffed in with addEntity. Why is this useful? Because it enables the 'main Haskell program' to populate the symbol table with the symbols that the plugins need. I'm guessing that the dynamically loaded plugins will not need very many entry points to the main program, so the 'populate symbol table' code would look like do { addEntity "foo" foo ; addEntity "baz" baz; ..etc.. One nice feature is that this automatically makes sure that the main program mentions 'foo', 'baz' etc, even if only the plugin is meant to call them. So that makes sure that when linking the main program, all the relevant bits are linked in. (Simon mentioned this problem in his message.) This approach is v simple, it would satisfy Duncan's need (I believe), it could be made more secure using dynamic types (addEntity and lookupEntity are good moments to typecheck), and it doesn't suffer from the disadvantages that Simon M mentioned. Simon PS: I'm working on an upgrade to GHC which, among other things, happens to eliminate the need for initLinker

2 1

RE: addFinalizer & ThreadId
by Simon Marlow 28 Aug '02

28 Aug '02

> I remember that addFinalizer never did work quite the way people > expected it to. A current instantiation of this problem: adding > a finalizer to a ThreadId. Could somebody please give me a short > explanation why the following never prints "Finalize"? > [ghc is 5.04-cvs] Your finalizer runs right at the end of the program, after the stdout Handle has been closed, so it can't print anything :-) Cheers, Simon

1 0

addFinalizer & ThreadId
by Volker Stolz 28 Aug '02

28 Aug '02

I remember that addFinalizer never did work quite the way people expected it to. A current instantiation of this problem: adding a finalizer to a ThreadId. Could somebody please give me a short explanation why the following never prints "Finalize"? [ghc is 5.04-cvs] \begin{code} module Main where import IO import Concurrent import GHC.Weak main = do mv <- newEmptyMVar child <- forkIO $ do takeMVar mv print "terminated" addFinalizer child fin putMVar mv () yield yield yield threadDelay $ 1000*1000*30 print "done" fin :: IO () fin = do print "Finalize" \end{code} -- http://www-i2.informatik.rwth-aachen.de/stolz/ *** PGP *** S/MIME

1 0

RE: RFC: ghc's dynamic linker
by Simon Marlow 28 Aug '02

28 Aug '02

Ok, let's start with a possible API to the library I think you're asking for: loadLibrary :: FilePath -> IO () lookupEntity :: String -> IO a (there's no type checking at dynamic link time of course, so you get to claim the returned object is whatever type you like). Right, now what would it take to implement this. As Duncan points out, this is almost possible already using the GHCi dynamic linker, which is available to any program compiled with GHC via the FFI. The interface is fairly straightforward, eg: foreign import "initLinker" unsafe initLinker :: IO () foreign import "lookupSymbol" unsafe c_lookupSymbol :: CString -> IO (Ptr a) foreign import "loadObj" unsafe c_loadObj :: CString -> IO Int but the main problem is that the dynamic linker can't link new modules to symbols in the currently running binary. So, in order to link a new Haskell module, you first have to load up a fresh copy of the 'base' and 'haskell98' packages, just like GHCi does. It *almost* works to do this, except that you get strange effects, one of which is that you have two copies of stdout each with their own buffer. Going the final step and allowing linkage to the current binary is possible, it just means the linker has to know how to read the symbol table out of the binary, and you have to avoid running 'strip'. I believe reading the symbol table is quite straightforward, the main problem being that on Unix you don't actually know where the binary lives, so you have to wire it in or search the PATH. Another problem is that you don't normally link a *complete* copy of the base package into your binary, you only link the bits you need. Linking the whole lot would mean every binary would be about 10M; but doing this on the basis of a flag which you turn on when you want to do dynamic linking maybe isn't so bad. There are a couple of other options: - make your program into a collection of dynamically-linked libraries itself. i.e. have a little stub main() which links with the RTS, and loads up 'base' followed by your program when it starts. The startup cost would be high (we don't do lazy linking in Haskell), but you'd only get one copy of the base package and this is possible right now. - make GHC generate objects that play nicely with the standarad dynamic linker on your system. This is entirely non-trivial, I believe. See previous discussions on this list. However, it might get easier in the future; I'm currently working on removing the need to distinguish code from data in GHC's RTS, which will eliminate some of the problems. Summary: extending GHC's dynamic linker to be able to slurp in the symbol table from the currently running binary would be useful, and is a good bite-sized GHC hacker task. I can't guarantee that we'll get around to it in a timely fashion, but contributions are, as always, entirely welcome... Cheers, Simon

5 8

RE: RFC: ghc's dynamic linker
by Simon Marlow 28 Aug '02

28 Aug '02

> I just want to remark that anyone contemplating hacking on object > files should steer well clear of BFD which is the most inconsistently > implemented, poorly (though abundantly) documented obstacle to quality > programming it has ever been my misfortune to use. Particularily > galling is the fact that BFD provides little indication (in > documentation, compile-time errors, etc.) that certain functionality > is completely unimplemented on a particular platform. > > In contrast, libelf is widely available (multiple implementations > exist), well documented, a joy to use and covers everything (well, > everything that GHC runs on) but Windows. Thanks, I didn't know about libelf. That might be a good replacement for our custom linker (at least on ELF platforms) at some point. Cheers, Simon

2 1

RE: Hardware STG?
by Simon Peyton-Jones 28 Aug '02

28 Aug '02

| advantages. The cost of course is always an issue, but my | point is that the | difference in performance may in many cases be insignificant. | | I see a definite place for this hardware, if it can be engineered | successfully. For example, I could build a firewall that is provably | inpenetrable. I could build embedded medical devices that I | can prove will not fail mysteriously. Yes, you are right. One possible goal is performance, but another, as you point out, is reliability and correctness. The issues are quite different and I'm sure Keean will want to be clear which objective he is shooting for. If you could really build a device where you had a proof of correctness that stretched from the original source code right through to hardware execution, that would be a real tour de force. Simon

1 0

RE: Posix.Fd questions
by Simon Marlow 28 Aug '02

28 Aug '02

> 1) Is the Posix stuff for doing IO operations on Fd in the > new hierarchy? Not yet. > If so where? The current plan is to put it in its own System.Posix (or possibly System.Unix) sub-hierarchy. > If not about when will it be there? When someone gets around to doing it :-) > (I'm wondering when I should > schedule the job of changing over all > the imports in our existing source code to the new hierarchy.) > 2) Is there any way of finding out whether an Fd you are > reading from has characters available? You can do a non-blocking read of a single character, or a select (not sure if the Select library still works, though). Cheers, Simon

1 0

RE: RFC: ghc's dynamic linker
by Simon Marlow 28 Aug '02

28 Aug '02

> On Tue, Aug 27, 2002 at 05:19:15 +0100, Simon Marlow wrote: > > > 'haskell98' packages, just like GHCi does. It *almost* works to do > > this, except that you get strange effects, one of which is > that you have > > two copies of stdout each with their own buffer. > > If it's not too much trouble, do you mind explaining why this is > so? It's just to satisfy my curiosity; don't worry if it's too > long-winded or contains really heavy wizardry :). It's quite simple: there will be one stdout Handle in your program, and one in the base package that has been loaded. Each is a separate Handle in its own right, with its own buffer, although they both point to the same underlying file descriptor. This can lead to strange effects because different parts of the program will be using different stdout handles, and the output might not be ordered in the way you expect. For functions, it's ok to have two identical copies, because you can't observe which one is being used. For most normal CAFs, the same is true except that the program might take longer to run than expected. You would be able to observe that there are two copies of the random number stream from the Random library, however. > You've already got the symbol table in memory though, right? Is > it absolutely necessary to re-read the binary? not sure what you mean here. The symbol table isn't normally read when a binary is executed - the same binary works fine after it is stripped, after all. Cheers, Simon

1 0

RE: RFC: ghc's dynamic linker
by Simon Marlow 28 Aug '02

28 Aug '02

> On 28 Aug 2002 01:34:37 +0100 > Alastair Reid <alastair(a)reid-consulting-uk.ltd.uk> wrote: > > > > 2. The BFD way > > I was just about to ask about this. I was going to ask what > ghc uses for > manipulating .o files. I've come accross BFD through gdb but > have never had to use it seriously. We do it The Hard Way, i.e. we have our own complete object linker. It's a lot simpler than BFD of course, because it only has to handle static linking. It currently supports ELF and the Windows PE object format. Cheers, Simon

1 0

RE: RFC: ghc's dynamic linker
by Simon Marlow 28 Aug '02

28 Aug '02

> From: Duncan Coutts [mailto:duncan@coutts.uklinux.net] > Sent: 28 August 2002 00:51 > > Would it be easier/better to exlicitly specify a symbol map > file for the > linker to use to locate the appropriate points in the current binary. > Then perhaps we need a flag to ask ghc to spit out a symbol map along > with the .o. Alternatively there tools to extract the map from a .o, I > don't know - I'm not a bin-utils guru! Possibly, but our linker already has most of the machinery necessary to read the symbol table out of a binary (the binary format is largely the same as an object file, it just has a fixed load address), so I'm guessing that using a symbol map will be extra work. > > Summary: extending GHC's dynamic linker to be able to slurp in the > > symbol table from the currently running binary would be > useful, and is a > > good bite-sized GHC hacker task. I can't guarantee that we'll get > > around to it in a timely fashion, but contributions are, as always, > > entirely welcome... > > Having made the suggestion, I'ts only right that I contribute > my (limited) > skills. I have done some gdb hacking before (not out of choice you > understand!) so I ought to know a bit about .o's ELF sections > and such. Take a look in ghc/rts/Linker.c if you're interested. Cheers, Simon

1 0