Re: Glasgow-haskell-users Digest, Vol 35, Issue 23

Brian, The standard method of skirting the LGPL restriction and saving your source code is to link dynamically in a separate step and then distribute your program along with the dynamically linked LGPL'd library. Compile with ghc -c (or with ghc -c -odir 'separate directory where you want to store the object files') and pass specific lines to the linker through gcc with -optc. Then link the object files for your program separately using ld and distribute the ghc runtime libraries you need to dynamically link along with your program. Some of these runtime libraries are big but on average libHSrts_dyn, libHSbase_dyn and libHSbase_cbits_dyn do the trick (I have needed cbits up for programs that use -ffi). Bulgat, Yes, GMP allows the fastest implementation but the problem I posed with GHC concerned "native" Integers, meaning that the actual implementation of Integer would be internal to the language implementation--i.e., not written in Haskell. I guess--I don't know GHC's internals apart from reading papers, such as "Haskell on a Shared-Memory Multiprocessor"--the memory problem would involve adding "carry" tag to chain thunks and allow lazy arbitrary precision arithmetic without resorting to program-level constructs like lists or big arrays. My point concerning other functional languages (OCaml and ML through MLton) was that gmp (or gmp-style) programs seem to be a common way of implementing the representation. Evolution means getting beyond that: is there another possibility? Esa,
I think the idea is to get rid of annoying dependencies (GMP), and not to create new ones (C++). There's few other big ifs with APREC as far as I see: Fiddling with FPU precision regs in x86 and possibly problems in Windows environment. It might require a bit fiddling to get memory handling right when using APREC, as well.
Right on. I like ARPREC for some programs; it is easy and fast but I
can imagine encountering problems when building it on Windows. ARPREC
also keeps its own internal memory representation: to use it from c
you essentially have to pass the numbers to its own runtime system
(i.e., you have to initialise ARPREC and allocate memory for a
variable using c_mpnew() before you can use it). It was just a
suggestion to think about...
-Peter
On 7/30/06, glasgow-haskell-users-request@haskell.org
Send Glasgow-haskell-users mailing list submissions to glasgow-haskell-users@haskell.org
To subscribe or unsubscribe via the World Wide Web, visit http://www.haskell.org/mailman/listinfo/glasgow-haskell-users or, via email, send a message with subject or body 'help' to glasgow-haskell-users-request@haskell.org
You can reach the person managing the list at glasgow-haskell-users-owner@haskell.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Glasgow-haskell-users digest..."
Today's Topics:
1. Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL (Seth Kurtzberg) 2. (Repost) Replacement for GMP as Bignum: ARPREC? Haskell?; OS X and OpenSSL (p.tanski@gmail.com) 3. Re: Re: Rebuilding GHC on Mac OSX PPC (p.tanski@gmail.com) 4. Re: (Repost) Replacement for GMP as Bignum: ARPREC? Haskell?; OS X and OpenSSL (Bulat Ziganshin) 5. Re: (Repost) Replacement for GMP as Bignum: ARPREC? Haskell?; OS X and OpenSSL (Esa Ilari Vuokko) 6. Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL (Brian Hulley) 7. Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL (Esa Ilari Vuokko) 8. Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL (Duncan Coutts) 9. Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL (Brian Hulley)
----------------------------------------------------------------------
Message: 1 Date: Sat, 29 Jul 2006 22:09:45 -0400 From: Seth Kurtzberg
Subject: Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL To: p.tanski@gmail.com Cc: glasgow-haskell-users@haskell.org Message-ID: <20060729220945.92f3f263.seth@cql.com> Content-Type: text/plain; charset=US-ASCII On Sat, 29 Jul 2006 21:45:21 -0400 p.tanski@gmail.com wrote:
GHC Task Ticket # 601 suggests replacing GMP with OpenSSL's Bignum library, BN. I have two questions concerning this:
(1) Why not use the ARbitrary PRECision Computation Package (ARPREC) by David Bailey, Yozo Hida, Karthik Jeyabalan, Xiaoye Li and Brandon Thompson? Here is a reference web page: http://crd.lbl.gov/~dhbailey/mpdist/
ARPREC is written in C++ but supports calls from C (see include/arprec/c_mp.h in the distribution directory). ARPREC is very fast and supports more complex mathematics than BN. The Licensing for ARPREC is not a problem: essentially similar to BSD3.
Note that there would be a basic configuration fix for building OpenSSL's BN library on OS 10.4: you would probably--I had to, myself--have to temporarily move the default installation of /usr/lib/libssl0.9.dylib (and variants) if you want to create a shared (.a) library because Apple's ln links dynamic libraries in preference to shared libraries.
I think you meant "if you want to create a static (.a) library", not "if you want to create a shared (.a) library"
Those of you who have some experience with
Apple's gcc also know that Apple's gcc does not recognise the -shared flag.
(2) A much more aesthetic solution would be to replace any external Bignum library with a pure Haskell library. There are many problems with building a truly efficient and fast Bignum library in Haskell but challenges like that are how languages evolve...
-Peter Tanski _______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
------------------------------
Message: 2 Date: Sat, 29 Jul 2006 23:21:07 -0400 From: p.tanski@gmail.com Subject: (Repost) Replacement for GMP as Bignum: ARPREC? Haskell?; OS X and OpenSSL To: glasgow-haskell-users@haskell.org Message-ID:
Content-Type: text/plain; charset=ISO-8859-1; format=flowed (Sorry for the repost: I needed to correct a few typos--thanks Seth--and clarify a few things.)
GHC Task Ticket # 601 suggests replacing GMP with OpenSSL's Bignum library, BN. I have two questions concerning this:
(1) Why not use the ARbitrary PRECision Computation Package (ARPREC) by David Bailey, Yozo Hida, Karthik Jeyabalan, Xiaoye Li and Brandon Thompson? Here is a reference web page: http://crd.lbl.gov/~dhbailey/mpdist/
ARPREC is written in C++ but supports calls from C (see include/arprec/c_mp.h in the distribution directory). ARPREC is very fast and supports more complex mathematics than BN. The Licensing for ARPREC is not a problem: essentially similar to BSD3.
Note that there would be a basic configuration fix for building OpenSSL's BN library on OS 10.4: you would probably--I had to, myself--have to temporarily move the default installation of /usr/lib/libssl0.9.dylib (and variants) if you want to create a static (.a) library because Apple's ln links dynamic libraries in preference to static libraries.
(2) A much more aesthetic solution would be to replace any external Bignum library with a pure Haskell library. There are many problems with building a truly efficient and fast Bignum library in Haskell but challenges like that are how languages evolve... Both GHC and nhc98 seem to use GMP, OCaml uses a custom bignum library written by Xavier Leroy and even MLton uses GMP. Robert Dockins wrote a type-based library for arbitrary that can (slowly) handle big numbers (up to 10^15), see http://www.eecs.tufts.edu/~rdocki01/typenats.html I don't know what the constraints on making such native Integer support work in GHC but my comment on the evolution of Haskell was not meant badly: native Integer support would require modifications to how Integers are handled in memory. (I would be glad to lend whatever support I could on this, but it seems like a big project.)
-Peter Tanski
------------------------------
Message: 3 Date: Sun, 30 Jul 2006 00:29:43 -0400 From: p.tanski@gmail.com Subject: Re: Re: Rebuilding GHC on Mac OSX PPC To: glasgow-haskell-users@haskell.org Message-ID:
Content-Type: text/plain; charset=ISO-8859-1; format=flowed Sorry to flood your mailboxes with my junk, but I had a few questions on the configuration and Makefiles for building GHC on OS 10.4:
The odd memory bugs (wierd malloc errors) seem to remind me of errors using pthreads...
in Joel Reymont's email, I noticed that ghc is given the -threaded option:
../../ghc/compiler/stage1/ghc-inplace -o stage2/ghc-6.4.3 ... ... -threaded -package readline -DUSE_READLINE -
I have not had any luck using -threaded with GHC on my own programs, but I do not know what options GHC passes to gcc with -threaded. If GHC -threaded means it passes gcc the -pthread option when compiling -fvia-C, the problem might be that Apple's gcc does not recognize the -pthread flag (you have to link -lpthread manually, preferably after the object files).
If I can get the configuration correct--or whatever parameters you think you might want--I will get a clean distribution and try to build ghc-6.4.3 tomorrow (I am using a 1.25 MHz PPC).
-Peter Tanski
------------------------------
Message: 4 Date: Sun, 30 Jul 2006 10:35:20 +0400 From: Bulat Ziganshin
Subject: Re: (Repost) Replacement for GMP as Bignum: ARPREC? Haskell?; OS X and OpenSSL To: p.tanski@gmail.com Cc: glasgow-haskell-users@haskell.org Message-ID: <664568392.20060730103520@gmail.com> Content-Type: text/plain; charset=us-ascii Hello p,
Sunday, July 30, 2006, 7:21:07 AM, you wrote:
Leroy and even MLton uses GMP. Robert Dockins wrote a type-based library for arbitrary that can (slowly) handle big numbers (up to 10^15), see http://www.eecs.tufts.edu/~rdocki01/typenats.html I don't
this don't have anything common with our problem of implementing efficient runtime library
know what the constraints on making such native Integer support work in GHC but my comment on the evolution of Haskell was not meant badly: native Integer support would require modifications to how Integers are handled in memory. (I would be glad to lend whatever support I could on this, but it seems like a big project.)
imho, you are wrong. current representation allows fastest operations. but anyway Haskell code will be several times slower than C one - even for GHC, much more for Haskell interpreters
-- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
------------------------------
Message: 5 Date: Sun, 30 Jul 2006 13:00:24 +0300 From: "Esa Ilari Vuokko"
Subject: Re: (Repost) Replacement for GMP as Bignum: ARPREC? Haskell?; OS X and OpenSSL To: "p.tanski@gmail.com" Cc: glasgow-haskell-users@haskell.org Message-ID: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi
On 7/30/06, p.tanski@gmail.com
wrote: (Sorry for the repost: I needed to correct a few typos--thanks Seth--and clarify a few things.)
GHC Task Ticket # 601 suggests replacing GMP with OpenSSL's Bignum library, BN. I have two questions concerning this:
(1) Why not use the ARbitrary PRECision Computation Package (ARPREC) by David Bailey, Yozo Hida, Karthik Jeyabalan, Xiaoye Li and Brandon Thompson? Here is a reference web page: http://crd.lbl.gov/~dhbailey/mpdist/
ARPREC is written in C++ but supports calls from C (see include/arprec/c_mp.h in the distribution directory). ARPREC is very fast and supports more complex mathematics than BN. The Licensing for ARPREC is not a problem: essentially similar to BSD3.
I think the idea is to get rid of annoying dependencies (GMP), and not to create new ones (C++). There's few other big ifs with APREC as far as I see: Fiddling with FPU precision regs in x86 and possibly problems in Windows environment. It might require a bit fiddling to get memory handling right when using APREC, as well.
In past, I tried to get rid of GMP by replacing it with libtommath http://math.libtomcrypt.com/ But I have given up for now - because of related and unrelated problems.
But, whoever starts the work probably gets to pick their lib, so OpenSSL's BN, APREC, LibTomMath or anything else...it's open :-)
Best regards, Esa
------------------------------
Message: 6 Date: Sun, 30 Jul 2006 11:53:30 +0100 From: "Brian Hulley"
Subject: Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL To: , Message-ID: <002801c6b3c7$5cab93d0$c8ec2950@osmet> Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response p.tanski@gmail.com wrote:
GHC Task Ticket # 601 suggests replacing GMP with OpenSSL's Bignum library, BN. I have two questions concerning this:
From the ticket, this looks very scary:
but its LGPL license is problematic for users of GHC (it prohibits static linking of GHC-compiled programs, for example).
Does this mean I can't distribute my Haskell app as a commercial application? I certainly don't want to distribute source code and I've got no idea how to compile a Haskell app other than using ghc --make, which creates a single exe ie with static linkage.
If I only use Int's (not Integer's) in my code, is there a way to get rid of GMP so that I won't have an impossible licencing problem when trying to distribute my application binary? If not, then it would seem absolutely vital to get rid of GMP as soon as possible or else try to persuade the authors to add the usual static linking exception to their LGPL licence for it.
Thanks, Brian.
-- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us.
------------------------------
Message: 7 Date: Sun, 30 Jul 2006 14:18:30 +0300 From: "Esa Ilari Vuokko"
Subject: Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL To: "Brian Hulley" Cc: p.tanski@gmail.com, glasgow-haskell-users@haskell.org Message-ID: Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 7/30/06, Brian Hulley
wrote: p.tanski@gmail.com wrote:
GHC Task Ticket # 601 suggests replacing GMP with OpenSSL's Bignum library, BN. I have two questions concerning this:
From the ticket, this looks very scary:
but its LGPL license is problematic for users of GHC (it prohibits static linking of GHC-compiled programs, for example).
Does this mean I can't distribute my Haskell app as a commercial application? I certainly don't want to distribute source code and I've got
I am not a lawyer, and you should consult lawyer if you want real advice about licensing issues.
Strict answer to first question: No. You can distribute statically compiled binaries.
no idea how to compile a Haskell app other than using ghc --make, which creates a single exe ie with static linkage.
But in this case, yes, you can't. You need to provide way to relink gmp in you program. As far as I know, this is simply possible by taking all the objs --make generates, other objs and libs and giving them on - users need to have ghc distro+their own gmp set in.
If I only use Int's (not Integer's) in my code, is there a way to get rid of GMP so that I won't have an impossible licencing problem when trying to
I am fairly certain you cannot force or rely GMP getting dropped by linker.
distribute my application binary? If not, then it would seem absolutely vital to get rid of GMP as soon as possible or else try to persuade the authors to add the usual static linking exception to their LGPL licence for it.
GMP being rather GNU ;-) replacing is probably the only option.
There's other reason for dropping GMP as well: The ghc rts steals GMP memory allocation (sort of integrating it with garbage collector) which makes it impossible for other parts of program use GMP (in practice).
HTH, --Esa
------------------------------
Message: 8 Date: Sun, 30 Jul 2006 12:58:09 +0100 From: Duncan Coutts
Subject: Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL To: Brian Hulley Cc: p.tanski@gmail.com, glasgow-haskell-users@haskell.org Message-ID: <1154260689.14437.26.camel@localhost> Content-Type: text/plain On Sun, 2006-07-30 at 11:53 +0100, Brian Hulley wrote:
p.tanski@gmail.com wrote:
GHC Task Ticket # 601 suggests replacing GMP with OpenSSL's Bignum library, BN. I have two questions concerning this:
From the ticket, this looks very scary:
but its LGPL license is problematic for users of GHC (it prohibits static linking of GHC-compiled programs, for example).
Does this mean I can't distribute my Haskell app as a commercial application? I certainly don't want to distribute source code and I've got no idea how to compile a Haskell app other than using ghc --make, which creates a single exe ie with static linkage.
GHC only statically links Haskell code. It dynamically links to GMP and the system C library and other C libs.
So you're fine.
On unix you can check for yourself with ldd. It lists all the shared libs that your program needs. For example:
$ ldd /usr/lib/ghc-6.4.2/ghc-6.4.2 libreadline.so.5 => /lib/libreadline.so.5 (0x00002b568fca6000) libncurses.so.5 => /lib/libncurses.so.5 (0x00002b568fde3000) libdl.so.2 => /lib/libdl.so.2 (0x00002b568ff3f000) libm.so.6 => /lib/libm.so.6 (0x00002b5690042000) libgmp.so.3 => /usr/lib/libgmp.so.3 (0x00002b569019a000) libpthread.so.0 => /lib/libpthread.so.0 (0x00002b56902cf000) libc.so.6 => /lib/libc.so.6 (0x00002b56903e4000) /lib64/ld-linux-x86-64.so.2 (0x00002b568fb8e000)
Duncan
------------------------------
Message: 9 Date: Sun, 30 Jul 2006 13:21:12 +0100 From: "Brian Hulley"
Subject: Re: Replacement for GMP as Bignum: ARPREC? Haskell?; OS-X and OpenSSL To: "Esa Ilari Vuokko" Cc: glasgow-haskell-users@haskell.org Message-ID: <003901c6b3d2$a6c01cb0$c8ec2950@osmet> Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Esa Ilari Vuokko wrote:
On 7/30/06, Brian Hulley
wrote: p.tanski@gmail.com wrote:
GHC Task Ticket # 601 suggests replacing GMP with OpenSSL's Bignum library, BN. I have two questions concerning this:
From the ticket, this looks very scary:
but its LGPL license is problematic for users of GHC (it prohibits static linking of GHC-compiled programs, for example). [snip] But in this case, yes, you can't. You need to provide way to relink gmp in you program. As far as I know, this is simply possible by taking all the objs --make generates, other objs and libs and giving them on - users need to have ghc distro+their own gmp set in.
Hi Esa - Thanks for reminding me about the distinction between source and object files. I keep forgetting that object files exist (!) and that it's sufficient for LGPL to just make them available.
Still, a slight problem is that since there is one object file per source file, the names of the object files give quite a lot of information away about the structure of the program especially when they are arranged in a module hierarchy, so I'll be glad when GMP is replaced by something without such a burdensome licence. (Although perhaps I can bundle my object files into a single library file but I don't know how to do this yet, or if it would really help in the goal to make the code completely obfuscated, impenetrable, and unavailable to any rival company... ;-) )
Thanks again, Brian. -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us.
------------------------------
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
End of Glasgow-haskell-users Digest, Vol 35, Issue 23 *****************************************************

p.tanski@gmail.com wrote:
Brian,
The standard method of skirting the LGPL restriction and saving your source code is to link dynamically in a separate step and then distribute your program along with the dynamically linked LGPL'd library. Compile with ghc -c (or with ghc -c -odir 'separate directory where you want to store the object files') and pass specific lines to the linker through gcc with -optc. Then link the object files for your program separately using ld and distribute the ghc runtime libraries you need to dynamically link along with your program. Some of these runtime libraries are big but on average libHSrts_dyn, libHSbase_dyn and libHSbase_cbits_dyn do the trick (I have needed cbits up for programs that use -ffi).
Hi - I think the main problem here is that I'm using Windows, so there is no way to dynamically link with the runtime libraries - the GHC implementations available for Windows only produce statically linked executables. Perhaps Windows support was just an afterthought from the main development of GHC on Unix, but I think it's quite a serious nusiance that the GHC runtime incorporates LGPL'd components in the light of the absence of the facility to dynamically link with it on this platform. Regards, Brian. -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us. http://www.metamilk.com

On Sun, 2006-07-30 at 17:33 +0100, Brian Hulley wrote:
I think the main problem here is that I'm using Windows, so there is no way to dynamically link with the runtime libraries - the GHC implementations available for Windows only produce statically linked executables.
Perhaps Windows support was just an afterthought from the main development of GHC on Unix, but I think it's quite a serious nusiance that the GHC runtime incorporates LGPL'd components in the light of the absence of the facility to dynamically link with it on this platform.
I think part of the issue is that static linking is very convenient and dynamic linking in this case would lead to some tricky administrative problems. Suppose for a moment that GHC did dynamically link gmp.dll, or indeed HSbase.dll. Where exactly would these files go? Currently I can cd to a random directory and compile something: ghc --make Foo.hs -o Foo.exe and then expect that Foo.exe to actually work. Now as far as I understand Windows dll technology this requires either that: 1. gmp.dll be copied into the directory with Foo.exe 2. gmp.dll exist in the windows or system directories 3. gmp.dll exist somewhere on the windows search %PATH% None of these are attractive or scalable solutions. 1 is ugly and doesn't scale if we dynamically link all GHC packages. 2 and 3 are remarkably fragile. See dll hell. For example they do not allow multiple versions of ghc to co-exist happily. Windows is very well set up for distributing isolated applications but I can't see how to do it for libraries. COM works by a runtime loading mechanism and a single entry point per dll and registering the location in the win32 registry. MSVC now supports a delayed loading mechanism that might work. It allows the loading of the dll to be delayed until the first call into it is made. So it might be possible to delay load all the dlls and then on startup explicitly load them (assuming you do have some mechanism for locating them). On Unix this isn't a problem because it's possible to embed the dynamic library search path into an executable (and indeed into a dynamic library) using the -rpath linker directive. Duncan

On Sun, 2006-07-30 at 19:03 +0100, Duncan Coutts wrote:
On Sun, 2006-07-30 at 17:33 +0100, Brian Hulley wrote:
I think part of the issue is that static linking is very convenient and dynamic linking in this case would lead to some tricky administrative problems.
Suppose for a moment that GHC did dynamically link gmp.dll, or indeed HSbase.dll. Where exactly would these files go?
Now as far as I understand Windows dll technology this requires either that: 1. gmp.dll be copied into the directory with Foo.exe 2. gmp.dll exist in the windows or system directories 3. gmp.dll exist somewhere on the windows search %PATH%
None of these are attractive or scalable solutions.
Nor is static linkage ;) FWIW: Windows has obsoleted your solutions above. It now uses a thing called 'assemblies'. This involves shipping all the dependent dll's with an application or library, and embedding a 'manifest' in each executable object. On installation, the dll's are copied into a cache if necessary, and the dynamic loader knows how to find them there. Versions of a library are kept distinct. In particular note this applies to the C library (MSVCR80.DLL) which is not considered a system library on XP anymore, XP64 in particular. Exactly how MinGW version of gcc will cope with all this I do not know. I personally couldn't figure out how to make it all work ;(
On Unix this isn't a problem because it's possible to embed the dynamic library search path into an executable (and indeed into a dynamic library) using the -rpath linker directive.
Which has problems of its own -- and is strongly discouraged by systems like Debian. Don't even think about it, rpath is evil. -- John Skaller <skaller at users dot sf dot net> Felix, successor to C++: http://felix.sf.net

On Mon, 2006-07-31 at 05:00 +1000, skaller wrote:
On Sun, 2006-07-30 at 19:03 +0100, Duncan Coutts wrote:
On Sun, 2006-07-30 at 17:33 +0100, Brian Hulley wrote:
I think part of the issue is that static linking is very convenient and dynamic linking in this case would lead to some tricky administrative problems.
Suppose for a moment that GHC did dynamically link gmp.dll, or indeed HSbase.dll. Where exactly would these files go?
Now as far as I understand Windows dll technology this requires either that: 1. gmp.dll be copied into the directory with Foo.exe 2. gmp.dll exist in the windows or system directories 3. gmp.dll exist somewhere on the windows search %PATH%
None of these are attractive or scalable solutions.
Nor is static linkage ;)
True :-)
FWIW: Windows has obsoleted your solutions above. It now uses a thing called 'assemblies'.
This involves shipping all the dependent dll's with an application or library, and embedding a 'manifest' in each executable object. On installation, the dll's are copied into a cache if necessary, and the dynamic loader knows how to find them there. Versions of a library are kept distinct.
Sounds good. I've been reading the MSDN documentation on assemblies for some time now and it's not yet clear to me if assemblies can be used without also using the .NET common language runtime. It seems like one needs .NET installed to use assemblies and that the global assemblies cache is specific to a version of the .NET CLR. Does this sound right? I was hoping that I'd be able to package Gtk+'s collection of .dlls as a single assembly and get that into the global cache and then get ghc to mark .exe's that use the Gtk2Hs Haskell packages as needing that Gtk+ assembly.
In particular note this applies to the C library (MSVCR80.DLL) which is not considered a system library on XP anymore, XP64 in particular.
That's interesting because presumably MSVCR80.DLL is not a .NET dll. I mean it's 'unmanaged' native code, not MSIL.
Exactly how MinGW version of gcc will cope with all this I do not know. I personally couldn't figure out how to make it all work ;(
Well there have been suggestions that GHC should use the MS C compiler and or linker. Sadly that would not help for my Gtk+ example which almost certainly needs gcc. Duncan

Duncan Coutts wrote:
On Sun, 2006-07-30 at 17:33 +0100, Brian Hulley wrote:
I think the main problem here is that I'm using Windows, so there is no way to dynamically link with the runtime libraries - the GHC implementations available for Windows only produce statically linked executables.
Perhaps Windows support was just an afterthought from the main development of GHC on Unix, but I think it's quite a serious nusiance that the GHC runtime incorporates LGPL'd components in the light of the absence of the facility to dynamically link with it on this platform.
I think part of the issue is that static linking is very convenient and dynamic linking in this case would lead to some tricky administrative problems.
Suppose for a moment that GHC did dynamically link gmp.dll, or indeed HSbase.dll. Where exactly would these files go?
I'd install them in the same directory as ghc.exe because this directory has to be on the %PATH% for the command ghc to work so Windows would find these DLLs the same way.
Currently I can cd to a random directory and compile something: ghc --make Foo.hs -o Foo.exe
and then expect that Foo.exe to actually work.
I know - I like this too. Also static linking seems to give more opportunity for optimization. If it wasn't for the LGPL restrictions there would be no problem (except that GMP might not have been written :-( (but then some nice person at GHC would probably have written or found a BSD3 lib instead... ;-) ))
Now as far as I understand Windows dll technology this requires either that: 1. gmp.dll be copied into the directory with Foo.exe 2. gmp.dll exist in the windows or system directories 3. gmp.dll exist somewhere on the windows search %PATH%
None of these are attractive or scalable solutions.
1 is ugly and doesn't scale if we dynamically link all GHC packages. 2 and 3 are remarkably fragile. See dll hell. For example they do not allow multiple versions of ghc to co-exist happily.
For developing apps, it would surely be fine if the DLLs were just stored in the ghc directory ie c:\ghc\ghc-6.4.2\bin then when you come to distribute your app, you'd simply just copy all the dlls the app needs as well as the exe into the installation directory of the app. When the app runs, Windows will first look in the app's directory and therefore find the correct set of DLLs. It's true that multiple versions of ghc could not coexist, but this already is the case at the moment: if you type "ghc" at a Windows command prompt Windows searches the one and only PATH variable to locate it so you can't have more than one version in any case, unless you create multiple user accounts and modify the user's PATH variables to point to the various versions of ghc.
Windows is very well set up for distributing isolated applications but I can't see how to do it for libraries. COM works by a runtime loading mechanism and a single entry point per dll and registering the location in the win32 registry.
Shared components are a bad idea imho because it's almost certain that someone elses's app would use a different version of a DLL - this is where real DLL Hell comes in! (and probably why .NET abandoned DLLs and (raw) COM in favour of assemblies)
MSVC now supports a delayed loading mechanism that might work. It allows the loading of the dll to be delayed until the first call into it is made. So it might be possible to delay load all the dlls and then on startup explicitly load them (assuming you do have some mechanism for locating them).
As long as the DLLs are stored in the same directory as the exe everything should be ok - it's only when people try to share DLLs by putting them somewhere else that versioning problems can arise. Also, the only things that need to be in DLLs are the LGPL libs - everything else could be statically linked. (Am I correct that the only LGPL component of GHC is GMP?)
On Unix this isn't a problem because it's possible to embed the dynamic library search path into an executable (and indeed into a dynamic library) using the -rpath linker directive.
It's also possible to call the Platform SDK function LoadLibrary with an explicit path to a DLL, but there are some restrictions on the use of LoadLibrary so I wouldn't recommend this - it's getting too complicated ;-) Regards, Brian. -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us. http://www.metamilk.com

On Sun, 2006-07-30 at 20:02 +0100, Brian Hulley wrote:
Suppose for a moment that GHC did dynamically link gmp.dll, or indeed HSbase.dll. Where exactly would these files go?
I'd install them in the same directory as ghc.exe because this directory has to be on the %PATH% for the command ghc to work so Windows would find these DLLs the same way.
As you say, it doesn't work for more than one version of GHC. And it doesn't work when names clash. If there is some lingering old version of gmp.dll in the windows system directory or in any other directory on the %PATH% (earlier than ghc's bin dir) then that dll gets picked up instead and everything breaks.
For developing apps, it would surely be fine if the DLLs were just stored in the ghc directory ie c:\ghc\ghc-6.4.2\bin then when you come to distribute your app, you'd simply just copy all the dlls the app needs as well as the exe into the installation directory of the app. When the app runs, Windows will first look in the app's directory and therefore find the correct set of DLLs.
Yes, it's fine for distributing applications but not much good for libraries.
It's true that multiple versions of ghc could not coexist, but this already is the case at the moment: if you type "ghc" at a Windows command prompt Windows searches the one and only PATH variable to locate it so you can't have more than one version in any case, unless you create multiple user accounts and modify the user's PATH variables to point to the various versions of ghc.
I use several versions of GHC at once on windows and switch between them in a console window just by doing: set PATH=...
Windows is very well set up for distributing isolated applications but I can't see how to do it for libraries. COM works by a runtime loading mechanism and a single entry point per dll and registering the location in the win32 registry.
Shared components are a bad idea imho because it's almost certain that someone elses's app would use a different version of a DLL - this is where real DLL Hell comes in! (and probably why .NET abandoned DLLs and (raw) COM in favour of assemblies)
They are pretty inevitable if you want to distribute Haskell libraries rather than applications. I'm not sure we can just say "don't do that".
As long as the DLLs are stored in the same directory as the exe everything should be ok - it's only when people try to share DLLs by putting them somewhere else that versioning problems can arise.
Indeed. This issue is the main source of problems that users have with the Gtk2Hs packages that I build for Windows. I do not yet have any satisfactory solution. Gtk+ comes as a collection of .dll files. Remember that I am distributing a Haskell library here, not an application. Users want to use my library to compile their own applications, eg for tasks on a university course. So where do I put the Gtk+ .dlls? At the moment I seem to have no choice but to put them on the %PATH%. And thus the breakage ensues. Old versions of Gtk+ that people have on their systems can interfere. Lots of naughty apps seem to install libz.dll into the windows system directory. The list seems endless. Each of these dll problems results in the Haskell GUI apps that users compile failing to start due to missing or mismatched dll imports. wxHaskell does the same. It installs it's wx-*.dll files into the windows directory (which is probably even worse). It's all sufficiently unreliable that I do not dare tell my students that they can do their practical on their home machines because I would get too many support requests. I am very interested at John Skaller's suggestion about assemblies. I will have to look into it. It certainly can't be worse than the current situation! :-) Duncan

Duncan Coutts wrote:
On Sun, 2006-07-30 at 20:02 +0100, Brian Hulley wrote: [snip]
app. When the app runs, Windows will first look in the app's directory and therefore find the correct set of DLLs.
Yes, it's fine for distributing applications but not much good for libraries.
Hi Duncan - That's true. I think I didn't catch onto the word "library" 'cause I was selfishly thinking too much about my poor unfinished app floundering helplessly in a stormy LGPL sea!!!
It's true that multiple versions of ghc could not coexist, but this already is the case at the moment: if you type "ghc" at a Windows command prompt Windows searches the one and only PATH variable to locate it so you can't have more than one version in any case, unless you create multiple user accounts and modify the user's PATH variables to point to the various versions of ghc.
I use several versions of GHC at once on windows and switch between them in a console window just by doing: set PATH=...
I hadn't thought of that (though I haven't used multiple versions of ghc at once yet).
Shared components are a bad idea imho because it's almost certain that someone elses's app would use a different version of a DLL - this is where real DLL Hell comes in! (and probably why .NET abandoned DLLs and (raw) COM in favour of assemblies)
They are pretty inevitable if you want to distribute Haskell libraries rather than applications. [snip]
I am very interested at John Skaller's suggestion about assemblies. I will have to look into it. It certainly can't be worse than the current situation! :-)
Absolutely - shared DLLs are an absolute nightmare (I became totally infamous at my last paid employment for introducing them to the workplace!!!) ANYTHING is better :-) Best regards, Brian -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us. http://www.metamilk.com

Duncan Coutts wrote:
On Sun, 2006-07-30 at 20:02 +0100, Brian Hulley wrote:
Suppose for a moment that GHC did dynamically link gmp.dll, or indeed HSbase.dll. Where exactly would these files go?
I'd install them in the same directory as ghc.exe because this directory has to be on the %PATH% for the command ghc to work so Windows would find these DLLs the same way.
As you say, it doesn't work for more than one version of GHC.
And it doesn't work when names clash. If there is some lingering old version of gmp.dll in the windows system directory or in any other directory on the %PATH% (earlier than ghc's bin dir) then that dll gets picked up instead and everything breaks.
For developing apps, it would surely be fine if the DLLs were just stored in the ghc directory ie c:\ghc\ghc-6.4.2\bin then when you come to distribute your app, you'd simply just copy all the dlls the app needs as well as the exe into the installation directory of the app. When the app runs, Windows will first look in the app's directory and therefore find the correct set of DLLs.
Yes, it's fine for distributing applications but not much good for libraries.
It's true that multiple versions of ghc could not coexist, but this already is the case at the moment: if you type "ghc" at a Windows command prompt Windows searches the one and only PATH variable to locate it so you can't have more than one version in any case, unless you create multiple user accounts and modify the user's PATH variables to point to the various versions of ghc.
I use several versions of GHC at once on windows and switch between them in a console window just by doing: set PATH=...
Windows is very well set up for distributing isolated applications but I can't see how to do it for libraries. COM works by a runtime loading mechanism and a single entry point per dll and registering the location in the win32 registry.
Shared components are a bad idea imho because it's almost certain that someone elses's app would use a different version of a DLL - this is where real DLL Hell comes in! (and probably why .NET abandoned DLLs and (raw) COM in favour of assemblies)
They are pretty inevitable if you want to distribute Haskell libraries rather than applications.
I'm not sure we can just say "don't do that".
As long as the DLLs are stored in the same directory as the exe everything should be ok - it's only when people try to share DLLs by putting them somewhere else that versioning problems can arise.
Indeed.
This issue is the main source of problems that users have with the Gtk2Hs packages that I build for Windows. I do not yet have any satisfactory solution.
Gtk+ comes as a collection of .dll files. Remember that I am distributing a Haskell library here, not an application. Users want to use my library to compile their own applications, eg for tasks on a university course.
So where do I put the Gtk+ .dlls? At the moment I seem to have no choice but to put them on the %PATH%. And thus the breakage ensues. Old versions of Gtk+ that people have on their systems can interfere. Lots of naughty apps seem to install libz.dll into the windows system directory. The list seems endless. Each of these dll problems results in the Haskell GUI apps that users compile failing to start due to missing or mismatched dll imports.
The "solution" to this problem is to use COM - COM assigns a unique identifier to an API, and you're only supposed to replace a DLL if you implement all the interfaces that the old DLL implemented, plus possibly some new ones. The reason libz & co. cause problems is because they are changing APIs without any kind of version tracking at all. .NET has a different solution to these issues. Assemblies can be installed globally, but when binding to an assembly the .NET runtime checks various constraints on which versions are compatible with each other before linking. So, in the absence of any runtime version checking like COM or .NET, it seems the only safe thing to do is put a GUID in the name of the DLL itself, or at the very least include a version number in the filename. Cheers, Simon

On Tue, 2006-08-01 at 17:05 +0100, Simon Marlow wrote:
Duncan Coutts wrote:
So where do I put the Gtk+ .dlls? At the moment I seem to have no choice but to put them on the %PATH%. And thus the breakage ensues. Old versions of Gtk+ that people have on their systems can interfere. Lots of naughty apps seem to install libz.dll into the windows system directory. The list seems endless. Each of these dll problems results in the Haskell GUI apps that users compile failing to start due to missing or mismatched dll imports.
The "solution" to this problem is to use COM - COM assigns a unique identifier to an API, and you're only supposed to replace a DLL if you implement all the interfaces that the old DLL implemented, plus possibly some new ones.
I've been talking to John Skaller about assemblies which do seem like a potentially sane solution to this problem As John pointed out in WinXP and above there is this assemblies mechanism that works not just for .NET but for native stuff too. If you look in your Windows/SxS/ directory you'll find a number of shared "side by side" assemblies. An assembly consists of a set of dlls with one dll containing an embedded xml manifest file that describes the name and version and member files of the assembly. Then any other .exe on the system is allowed to have an associated or embedded manifest that specifies dependent assemblies. The runtime linker then looks for these assemblies in the apps' private dir (ie subdirectories of the dir containing the .exe) or in the shared Windows/SxS assembly collection. The tools to embed manifests are not particularly complex. There is a mt.exe that embeds a manifest in a .exe or .dll after it's already been linked, or the resource compiled rc.exe can embed it during linking. I've not yet found out if mt.exe works on gcc-compiled .dll or .exe files. You can read about mt.exe and side by side assemblies here: http://windowssdk.msdn.microsoft.com/en-us/library/ms716972.aspx So, supposing this works, how might it apply to ghc? We could add an assemblies section to the .cabal/package.conf info, much like the OSX frameworks thing. However it'd need ghc to call some tool to embed an xml manifest specifying these dependencies in the generated .exe. It's not clear if such a tool is freely available. As for Gtk2Hs on Windows, I'd need to turn Gtk+ into an side by side assembly by producing and embedding a manifest in a representative gtk .dll and then specify this as a framework dependency in the cabal/package.conf file. Not trivial. Duncan

Brian, Sorry, I smash out emails without thinking and forgot GHC is distributed with static archives in Windows. No more. Even if you build the GHC runtime library from source as DLLs you will run into another problem, as noted in the DLL-NOTES file (see http:// cvs.haskell.org/cgi-bin/cvsweb.cgi/fptools/ghc/compiler/DLL-NOTES? rev=1.1;content-type=text%2Fplain or the file compiler/DLL-NOTES in the distribution): some of the runtime system must be statically linked into your main executable in order to resolve all the symbols at compile-time. The only way around this is to treat those runtime libraries like plugins. A good reference is http:// edll.m2osw.com/ . I have honestly not used EDLL on GHC runtime libraries, but it should be possible. One of the main goals for the next major release of GHC is to make it Windows-native and use Microsoft's CL. I think that is another big project.... The disadvantage to "free" software is that it often feels like you are trying to build a car with spare parts; either you spend your time porting and fixing things yourself--an almost daily task, these days--or you wait for someone with more experience or time than you have to fix it for you (which may never happen, or may not happen the way you want it). The advantage to free software is that, like the Haskell language, you get to use some of the most advanced programming available. So here I am, trying to figure out what I can do to help GHC, since right now GHC is the only actively maintained, current Haskell compiler available. (In any case, nhc98 uses GMP as well, so even if you use nhc98 you will still have the DLL-NOTES problem to deal with.) Best Regards, Peter On Jul 30, 2006, at 12:33 PM, Brian Hulley wrote:
p.tanski@gmail.com wrote:
Brian,
The standard method of skirting the LGPL restriction and saving your source code is to link dynamically in a separate step and then distribute your program along with the dynamically linked LGPL'd library. Compile with ghc -c (or with ghc -c -odir 'separate directory where you want to store the object files') and pass specific lines to the linker through gcc with -optc. Then link the object files for your program separately using ld and distribute the ghc runtime libraries you need to dynamically link along with your program. Some of these runtime libraries are big but on average libHSrts_dyn, libHSbase_dyn and libHSbase_cbits_dyn do the trick (I have needed cbits up for programs that use -ffi).
Hi - I think the main problem here is that I'm using Windows, so there is no way to dynamically link with the runtime libraries - the GHC implementations available for Windows only produce statically linked executables.
Perhaps Windows support was just an afterthought from the main development of GHC on Unix, but I think it's quite a serious nusiance that the GHC runtime incorporates LGPL'd components in the light of the absence of the facility to dynamically link with it on this platform.
Regards, Brian. -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us.

Hi Brian,
So here I am, trying to figure out what I can do to help GHC, since right now GHC is the only actively maintained, current Haskell compiler available. nhc98 and Yhc are both compilers which are actively maintained.
(In any case, nhc98 uses GMP as well, so even if you use nhc98 you will still have the DLL-NOTES problem to deal with.) However, Yhc has a bytecode/runtime split, the runtime is Yhc and is linked with GMP. The bytecode is not linked to GMP. In fact, even if GMP was GPL (rather than just LGPL), Yhc would still allow you to have entirely closed source bytecodes. So with Yhc you can have big numbers and closed source :)
Thanks Neil

Neil Mitchell wrote:
Hi Brian,
So here I am, trying to figure out what I can do to help GHC, since right now GHC is the only actively maintained, current Haskell compiler available. nhc98 and Yhc are both compilers which are actively maintained.
(In any case, nhc98 uses GMP as well, so even if you use nhc98 you will still have the DLL-NOTES problem to deal with.) However, Yhc has a bytecode/runtime split, the runtime is Yhc and is linked with GMP. The bytecode is not linked to GMP. In fact, even if GMP was GPL (rather than just LGPL), Yhc would still allow you to have entirely closed source bytecodes. So with Yhc you can have big numbers and closed source :)
Hi Neil - thanks for the info about Yhc. Best regards, Brian. -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us. http://www.metamilk.com

Peter Tanski wrote: [snip]
The advantage to free software is that, like the Haskell language, you get to use some of the most advanced programming available. So here I am, trying to figure out what I can do to help GHC, since right now GHC is the only actively maintained, current Haskell compiler available. (In any case, nhc98 uses GMP as well, so even if you use nhc98 you will still have the DLL-NOTES problem to deal with.)
Hi Peter - It seems a workable solution is just to follow what Esa recommended (similar to what you said also) - namely linking all the obj's (for user source) together, stripping out debug info, and distributing this (now rather obfuscated object file) along with the usual exe to satisfy the demands of LGPL, so although a hypothetical implementation of GHC which put the runtime in DLLs might make things easier, it's ok as things are at the moment. Apologies if I seemed to be too negative about the current state of affairs... ;-) Best regards, Brian. -- Logic empowers us and Love gives us purpose. Yet still phantoms restless for eras long past, congealed in the present in unthought forms, strive mightily unseen to destroy us. http://www.metamilk.com
participants (7)
-
Brian Hulley
-
Duncan Coutts
-
Neil Mitchell
-
p.tanski@gmail.com
-
Peter Tanski
-
Simon Marlow
-
skaller