I've finished a good sized Haskell project which I now must expose as a library (along with a lot of C code) to my fellow engineers. I had originally devoped my code on Windows XP and managed to learn how to wrap my Haskell code in a DLL and create an MSVC linkable library stub for the DLL. Although involved, the output in windows is relatively transparent in that as long as the paths are available to the Haskell runtime libraries and my DLL, linking is straightforward. Now however I find I must export the same libraries to linux. I decided to get 'smart' and repackage my build tools using CMAKE and cabal. Building the C object files is no problem and even configuring Cabal to generate my executables is not too painful. However the pain begins when one tries to link any Haskell object files or libraries to an external C program. Using ghc -v during the build reveals that gcc has about 50 -u flag arguments and links to 9 or 10 different libraries. Given that my little component is just a tiny part of a large software project involving many more engineers and many man years of work, it's not going to be fun to link my little library using all these crazy build flags to a large external project, especially when order is important. Wouldn't it be nice if there were a single run time library I could link to, (or even generate on the fly) that didn't have this intricate build step. How is ghc/Haskell supposed to integrate with the rest of the software world? A related issue in the building of my export libraries is cabal itself. I want to package my own library with some external object files etc and some Haskell code. It's easily done with ar and the correct object files. Unfortunately it is not clear from the documentation how I can force cabal to place the object files and generated stub files in a place where I can use them. (CMake is kind of forcing me to have them reside with the source for reasons I don't want to go into here.) I instead find myself copying the object files (*.o) back into the source directory from an obscure tmp directory several layers down from the source directory. This is not particularly portable and is probably quite fragile once someone decides to change the directory hiearcharchy in cabal. I guess I am surprised that on linux, with all of it's amazing software development tools, that Haskell export libraries would be this tricky to develop. -- View this message in context: http://www.nabble.com/Export-Haskell-Libraries-tf3569747.html#a9972866 Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.
On Thu, 2007-04-12 at 21:48 -0700, SevenThunders wrote:
I guess I am surprised that on linux, with all of it's amazing software development tools, that Haskell export libraries would be this tricky to develop.
I guess it's because most of the existing Haskell hackers are going the other direction, they build Haskell programs that use external C libs, but the top level of the system is always Haskell. It's much more rare for these people to want to include a Haskell component into another system. So it's easy if you link the system using ghc. If you want to link it all using gcc instead, yeah, that's a bit harder. You can see most of the flags ghc passes to gcc as they're just in the package configuration for the rts and base packages (ghc-pkg display rts / base). It should be fairly straightforward to generate a gnu linker script from all this that you could use with gcc when linking your whole system. By tucking the ghc flags away in a linker scrupt you will not terrify your fellow developers with all the odd -u flags. As for the issue of cabal putting generated files in a directory other than the source tree, you can tell cabal exactly which directory to use, so it's not that non-portable to go grubbing around in it to find the .o files you need to link into the archive file. Alternatively you could just let cabal build the .a file. It can include externally generated .o files into the archive. Alternatively you can just use two .a files, keeping your other .o's in a separate file. Or you could even combine the two archives together as a later build step. Actually it's not too bad if you ignore all the 50 -u flags. Apart from that, the "single runtime library" you want is just three: HSbase, HSbase_cbits and HSrts. Those also depend on some system C libs: m, gmp, dl and rt. There is a project for this year's Google Summer of Code to use dynamic libraries on Linux. Perhaps this would make the situation better for you since dynamic libs record their own dependencies much better. Ideally you would only have to link to your own Haskell package built as a .so and that would pull in the dependencies on HSbase.so, HSrts.so and the other system libs. Duncan
Duncan Coutts wrote:
So it's easy if you link the system using ghc. If you want to link it all using gcc instead, yeah, that's a bit harder. You can see most of the flags ghc passes to gcc as they're just in the package configuration for the rts and base packages (ghc-pkg display rts / base). It should be fairly straightforward to generate a gnu linker script from all this that you could use with gcc when linking your whole system. By tucking the ghc flags away in a linker scrupt you will not terrify your fellow developers with all the odd -u flags.
That was my first thought and in fact I did write such a script. The only problem is I'm afraid that the link stages for the software I have integrate to may be rather complex and I thought that maybe this would not be the best approach if there were order dependencies etc. But maybe it's not so bad. In the end I managed to capture all the dependencies in CMake so I'm hoping that will make it a little easier to do the final integration.
As for the issue of cabal putting generated files in a directory other than the source tree, you can tell cabal exactly which directory to use, so it's not that non-portable to go grubbing around in it to find the .o files you need to link into the archive file.
I saw a lot of options for places to put sources and targets, but I couldn't quite figure out how to configure it to place the object file output. No doubt it's there, I just couldn't find it in the 45 min.s or so that I looked for it.
Alternatively you could just let cabal build the .a file. It can include externally generated .o files into the archive. Alternatively you can just use two .a files, keeping your other .o's in a separate file. Or you could even combine the two archives together as a later build step.
Yes, this would be an attractive approach I think. Is it a matter of passing the correct flags to ghc, Ghc-options: -? At first glance, looking at the basic tutorial it seemed like to build a library one uses a line like Exposed Modules: A B C However I thought this would build Haskell only libraries. Then there is the business of merging libraries, which I suppose is done with ar and ranlib by extracting all the object files from one library and then adding them back in to the other. If it had to portable to windows as well I wonder if this would work.
Actually it's not too bad if you ignore all the 50 -u flags. Apart from that, the "single runtime library" you want is just three: HSbase, HSbase_cbits and HSrts. Those also depend on some system C libs: m, gmp, dl and rt.
running ghc -v for all my haskell code forced me to link to these libraries ultimately: HShaskell98 HSregex-compat HSregex-posix HSregex-base HSparsec HSbase HSbase_cbits HSrts m gmp dl rt
There is a project for this year's Google Summer of Code to use dynamic libraries on Linux. Perhaps this would make the situation better for you since dynamic libs record their own dependencies much better. Ideally you would only have to link to your own Haskell package built as a .so and that would pull in the dependencies on HSbase.so, HSrts.so and the other system libs.
Duncan
Then it would be very similar to the windows build steps and probably a bit easier since one wouldn't have to mess with dlltools and converting libraries to ms vc formats etc. Really all that's needed though is a tool that can automagically wrap a homegrown static or even dynamic library that contains the needed components of the GHC run time library along with the additional user code. I know all the object files are available as are of course the libraries themselves, so such a script is not impossible. It seems that ghc itself is doing some kind of dependency analysis to determine the final call to gcc. _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe -- View this message in context: http://www.nabble.com/Export-Haskell-Libraries-tf3569747.html#a9973642 Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.
It seems that what you want is a standalone .a file that you can then link into other programs without any special options. In principle this should be possible: you just need to include .o files for the RTS and libraries, and fortunately we already have those (HSrts.o, HSbase.o etc.) because they're needed by GHCi. You won't need the -u options because the whole base package will be linked in anyway. I can't immediately think of a reason this wouldn't work, but I could be wrong... Cheers, Simon SevenThunders wrote:
Duncan Coutts wrote:
So it's easy if you link the system using ghc. If you want to link it all using gcc instead, yeah, that's a bit harder. You can see most of the flags ghc passes to gcc as they're just in the package configuration for the rts and base packages (ghc-pkg display rts / base). It should be fairly straightforward to generate a gnu linker script from all this that you could use with gcc when linking your whole system. By tucking the ghc flags away in a linker scrupt you will not terrify your fellow developers with all the odd -u flags.
That was my first thought and in fact I did write such a script. The only problem is I'm afraid that the link stages for the software I have integrate to may be rather complex and I thought that maybe this would not be the best approach if there were order dependencies etc. But maybe it's not so bad. In the end I managed to capture all the dependencies in CMake so I'm hoping that will make it a little easier to do the final integration.
As for the issue of cabal putting generated files in a directory other than the source tree, you can tell cabal exactly which directory to use, so it's not that non-portable to go grubbing around in it to find the .o files you need to link into the archive file.
I saw a lot of options for places to put sources and targets, but I couldn't quite figure out how to configure it to place the object file output. No doubt it's there, I just couldn't find it in the 45 min.s or so that I looked for it.
Alternatively you could just let cabal build the .a file. It can include externally generated .o files into the archive. Alternatively you can just use two .a files, keeping your other .o's in a separate file. Or you could even combine the two archives together as a later build step.
Yes, this would be an attractive approach I think. Is it a matter of passing the correct flags to ghc, Ghc-options: -? At first glance, looking at the basic tutorial it seemed like to build a library one uses a line like Exposed Modules: A B C However I thought this would build Haskell only libraries. Then there is the business of merging libraries, which I suppose is done with ar and ranlib by extracting all the object files from one library and then adding them back in to the other. If it had to portable to windows as well I wonder if this would work.
Actually it's not too bad if you ignore all the 50 -u flags. Apart from that, the "single runtime library" you want is just three: HSbase, HSbase_cbits and HSrts. Those also depend on some system C libs: m, gmp, dl and rt.
running ghc -v for all my haskell code forced me to link to these libraries ultimately: HShaskell98 HSregex-compat HSregex-posix HSregex-base HSparsec HSbase HSbase_cbits HSrts m gmp dl rt
There is a project for this year's Google Summer of Code to use dynamic libraries on Linux. Perhaps this would make the situation better for you since dynamic libs record their own dependencies much better. Ideally you would only have to link to your own Haskell package built as a .so and that would pull in the dependencies on HSbase.so, HSrts.so and the other system libs.
Duncan
Then it would be very similar to the windows build steps and probably a bit easier since one wouldn't have to mess with dlltools and converting libraries to ms vc formats etc. Really all that's needed though is a tool that can automagically wrap a homegrown static or even dynamic library that contains the needed components of the GHC run time library along with the additional user code. I know all the object files are available as are of course the libraries themselves, so such a script is not impossible. It seems that ghc itself is doing some kind of dependency analysis to determine the final call to gcc.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
While we're on the topic, does anyone know if there exists a similarly
simple solution like the (last) section "Using both Python & Haskell
with ctypes" at
http://wiki.python.org/moin/PythonVsHaskell
that works on Linux to easily link Haskell libraries/functions into Python?
Cheers,
Jason
On 4/13/07, Simon Marlow
It seems that what you want is a standalone .a file that you can then link into other programs without any special options. In principle this should be possible: you just need to include .o files for the RTS and libraries, and fortunately we already have those (HSrts.o, HSbase.o etc.) because they're needed by GHCi. You won't need the -u options because the whole base package will be linked in anyway. I can't immediately think of a reason this wouldn't work, but I could be wrong...
Cheers, Simon
SevenThunders wrote:
Duncan Coutts wrote:
So it's easy if you link the system using ghc. If you want to link it all using gcc instead, yeah, that's a bit harder. You can see most of the flags ghc passes to gcc as they're just in the package configuration for the rts and base packages (ghc-pkg display rts / base). It should be fairly straightforward to generate a gnu linker script from all this that you could use with gcc when linking your whole system. By tucking the ghc flags away in a linker scrupt you will not terrify your fellow developers with all the odd -u flags.
That was my first thought and in fact I did write such a script. The only problem is I'm afraid that the link stages for the software I have integrate to may be rather complex and I thought that maybe this would not be the best approach if there were order dependencies etc. But maybe it's not so bad. In the end I managed to capture all the dependencies in CMake so I'm hoping that will make it a little easier to do the final integration.
As for the issue of cabal putting generated files in a directory other than the source tree, you can tell cabal exactly which directory to use, so it's not that non-portable to go grubbing around in it to find the .o files you need to link into the archive file.
I saw a lot of options for places to put sources and targets, but I couldn't quite figure out how to configure it to place the object file output. No doubt it's there, I just couldn't find it in the 45 min.s or so that I looked for it.
Alternatively you could just let cabal build the .a file. It can include externally generated .o files into the archive. Alternatively you can just use two .a files, keeping your other .o's in a separate file. Or you could even combine the two archives together as a later build step.
Yes, this would be an attractive approach I think. Is it a matter of passing the correct flags to ghc, Ghc-options: -? At first glance, looking at the basic tutorial it seemed like to build a library one uses a line like Exposed Modules: A B C However I thought this would build Haskell only libraries. Then there is the business of merging libraries, which I suppose is done with ar and ranlib by extracting all the object files from one library and then adding them back in to the other. If it had to portable to windows as well I wonder if this would work.
Actually it's not too bad if you ignore all the 50 -u flags. Apart from that, the "single runtime library" you want is just three: HSbase, HSbase_cbits and HSrts. Those also depend on some system C libs: m, gmp, dl and rt.
running ghc -v for all my haskell code forced me to link to these libraries ultimately: HShaskell98 HSregex-compat HSregex-posix HSregex-base HSparsec HSbase HSbase_cbits HSrts m gmp dl rt
There is a project for this year's Google Summer of Code to use dynamic libraries on Linux. Perhaps this would make the situation better for you since dynamic libs record their own dependencies much better. Ideally you would only have to link to your own Haskell package built as a .so and that would pull in the dependencies on HSbase.so, HSrts.so and the other system libs.
Duncan
Then it would be very similar to the windows build steps and probably a bit easier since one wouldn't have to mess with dlltools and converting libraries to ms vc formats etc. Really all that's needed though is a tool that can automagically wrap a homegrown static or even dynamic library that contains the needed components of the GHC run time library along with the additional user code. I know all the object files are available as are of course the libraries themselves, so such a script is not impossible. It seems that ghc itself is doing some kind of dependency analysis to determine the final call to gcc.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
jasonm wrote:
While we're on the topic, does anyone know if there exists a similarly simple solution like the (last) section "Using both Python & Haskell with ctypes" at http://wiki.python.org/moin/PythonVsHaskell that works on Linux to easily link Haskell libraries/functions into Python?
Cheers, Jason
Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
What I would do is go through the 'usual' export process of exporting Haskell to C. You could use the ***_stub.h files that are produced, but it's usually a bit easier to make your own .h files. After this use swig to wrap the C callable result into Python. A lot of the build tools seem to support SWIG these days. I know that CMake does and probably SCONS does to. What would be nice would be just a touch more automation to handle this. Maybe we can write some Haskell code to script up the process of linking together the correct object files for export on the Haskell side. Or perhaps it suffices to automate it within CMake or SCONS or provide macros or python scripts to do the same. -- View this message in context: http://www.nabble.com/Export-Haskell-Libraries-tf3569747.html#a9984796 Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.
In the GHC docs: http://www.haskell.org/ghc/docs/6.4.1/html/users_guide/sec-ffi-ghc.html#usin... "There can be multiple calls to hs_init(), but each one should be matched by one (and only one) call to hs_exit()[8]." What exactly happens with nested calls? Is there only one runtime created, with a simple counter to know which hs_exit should shut it down? If so, is there a way of having multiple interpreters open safely at the same time? Or does each hs_init() create a new separate concurrent runtime (the preferable alternative)? And what is the cost of creating and destructing the GHC runtime anyway? Can the Haskell interpreter be in a Linux shared-object library, so long as I make sure to call hs_init() after loading and hs_exit() before unloading it? My experiments so far show this working flawlessly, but I vaguely remember an e-mail thread saying GHC couldn't be linked in dynamically. Dan Simon Marlow wrote:
It seems that what you want is a standalone .a file that you can then link into other programs without any special options. In principle this should be possible: you just need to include .o files for the RTS and libraries, and fortunately we already have those (HSrts.o, HSbase.o etc.) because they're needed by GHCi. You won't need the -u options because the whole base package will be linked in anyway. I can't immediately think of a reason this wouldn't work, but I could be wrong...
Cheers, Simon
SevenThunders wrote:
Duncan Coutts wrote:
So it's easy if you link the system using ghc. If you want to link it all using gcc instead, yeah, that's a bit harder. You can see most of the flags ghc passes to gcc as they're just in the package configuration for the rts and base packages (ghc-pkg display rts / base). It should be fairly straightforward to generate a gnu linker script from all this that you could use with gcc when linking your whole system. By tucking the ghc flags away in a linker scrupt you will not terrify your fellow developers with all the odd -u flags.
That was my first thought and in fact I did write such a script. The only problem is I'm afraid that the link stages for the software I have integrate to may be rather complex and I thought that maybe this would not be the best approach if there were order dependencies etc. But maybe it's not so bad. In the end I managed to capture all the dependencies in CMake so I'm hoping that will make it a little easier to do the final integration.
As for the issue of cabal putting generated files in a directory other than the source tree, you can tell cabal exactly which directory to use, so it's not that non-portable to go grubbing around in it to find the .o files you need to link into the archive file.
I saw a lot of options for places to put sources and targets, but I couldn't quite figure out how to configure it to place the object file output. No doubt it's there, I just couldn't find it in the 45 min.s or so that I looked for it.
Alternatively you could just let cabal build the .a file. It can include externally generated .o files into the archive. Alternatively you can just use two .a files, keeping your other .o's in a separate file. Or you could even combine the two archives together as a later build step.
Yes, this would be an attractive approach I think. Is it a matter of passing the correct flags to ghc, Ghc-options: -? At first glance, looking at the basic tutorial it seemed like to build a library one uses a line like Exposed Modules: A B C However I thought this would build Haskell only libraries. Then there is the business of merging libraries, which I suppose is done with ar and ranlib by extracting all the object files from one library and then adding them back in to the other. If it had to portable to windows as well I wonder if this would work.
Actually it's not too bad if you ignore all the 50 -u flags. Apart from that, the "single runtime library" you want is just three: HSbase, HSbase_cbits and HSrts. Those also depend on some system C libs: m, gmp, dl and rt.
running ghc -v for all my haskell code forced me to link to these libraries ultimately: HShaskell98 HSregex-compat HSregex-posix HSregex-base HSparsec HSbase HSbase_cbits HSrts m gmp dl rt
There is a project for this year's Google Summer of Code to use dynamic libraries on Linux. Perhaps this would make the situation better for you since dynamic libs record their own dependencies much better. Ideally you would only have to link to your own Haskell package built as a .so and that would pull in the dependencies on HSbase.so, HSrts.so and the other system libs.
Duncan
Then it would be very similar to the windows build steps and probably a bit easier since one wouldn't have to mess with dlltools and converting libraries to ms vc formats etc. Really all that's needed though is a tool that can automagically wrap a homegrown static or even dynamic library that contains the needed components of the GHC run time library along with the additional user code. I know all the object files are available as are of course the libraries themselves, so such a script is not impossible. It seems that ghc itself is doing some kind of dependency analysis to determine the final call to gcc.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Dan Weston wrote:
In the GHC docs: http://www.haskell.org/ghc/docs/6.4.1/html/users_guide/sec-ffi-ghc.html#usin...
"There can be multiple calls to hs_init(), but each one should be matched by one (and only one) call to hs_exit()[8]."
What exactly happens with nested calls? Is there only one runtime created, with a simple counter to know which hs_exit should shut it down? If so, is there a way of having multiple interpreters open safely at the same time?
Or does each hs_init() create a new separate concurrent runtime (the preferable alternative)?
And what is the cost of creating and destructing the GHC runtime anyway?
Can the Haskell interpreter be in a Linux shared-object library, so long as I make sure to call hs_init() after loading and hs_exit() before unloading it? My experiments so far show this working flawlessly, but I vaguely remember an e-mail thread saying GHC couldn't be linked in dynamically.
Dan
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Thats a good question. I know that on windows multiple calls to loadLibrary end up returning memory to the same shared area, although different processes have their own data block (but still share the code block). Unfortunately the same process with different threads share the same data block, potentially causing issues if static memory gets altered. Thus it's up to the programmer to create and initialize any static memory that should be unique to a thread. Perhaps hs_init() does this, I do not know. -- View this message in context: http://www.nabble.com/Export-Haskell-Libraries-tf3569747.html#a9985245 Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.
Dan Weston wrote:
In the GHC docs: http://www.haskell.org/ghc/docs/6.4.1/html/users_guide/sec-ffi-ghc.html#usin...
"There can be multiple calls to hs_init(), but each one should be matched by one (and only one) call to hs_exit()[8]."
What exactly happens with nested calls? Is there only one runtime created, with a simple counter to know which hs_exit should shut it down?
Yes. But the runtime doesn't currently support restarting (hs_exit() followed by hs_init()).
If so, is there a way of having multiple interpreters open safely at the same time?
I'm not sure exactly what you mean by "multiple interpreters". The runtime is only designed to support a single instance of itself (it uses global static storage everywhere).
Or does each hs_init() create a new separate concurrent runtime (the preferable alternative)?
Nope, see above. To do that you'd need a "runtime handle" returned by hs_init() and passed to every foreign exported function, somehow.
And what is the cost of creating and destructing the GHC runtime anyway?
Not much.
Can the Haskell interpreter be in a Linux shared-object library, so long as I make sure to call hs_init() after loading and hs_exit() before unloading it? My experiments so far show this working flawlessly, but I vaguely remember an e-mail thread saying GHC couldn't be linked in dynamically.
Perhaps, although the shared library won't really be shared - it'll be linked in place each time you use it, because we don't currently have support for PIC everywhere (well, we have partial support and there's a SoC project to finish it off). Cheers, Simon.
On Thu, 2007-04-12 at 23:38 -0700, SevenThunders wrote:
I saw a lot of options for places to put sources and targets, but I couldn't quite figure out how to configure it to place the object file output. No doubt it's there, I just couldn't find it in the 45 min.s or so that I looked for it.
runghc Setup.hs configure --help give the full list. The one you want is --scratchdir (or just -b).
It seems that ghc itself is doing some kind of dependency analysis to determine the final call to gcc.
Yes, ghc knows which packages are required and the description for each package lists the packages it depends on and any C libs and other linker flags and search paths it needs. Duncan
SevenThunders wrote:
... Really all that's needed though is a tool that can automagically wrap a homegrown static or even dynamic library that contains the needed components of the GHC run time library along with the additional user code. I know all the object files are available as are of course the libraries themselves, so such a script is not impossible. It seems that ghc itself is doing some kind of dependency analysis to determine the final call to gcc.
To which Duncan replied:
Yes, ghc knows which packages are required and the description for each package lists the packages it depends on and any C libs and other linker flags and search paths it needs.
Automated sounds great. Has anything been done in this regard? I'm new to Haskell and to GHC in particular; how difficult is this? Where would I start in coding it myself (it seems as though I would need some inside knowledge into the workings of GHC)? Otherwise I'll try to make SevenThunders' approach work. Did you have success with the HS*.o files? Many thanks, Joe
participants (6)
-
Dan Weston -
Duncan Coutts -
Jason Morton -
Joe -
SevenThunders -
Simon Marlow