- Glasgow-haskell-users

Re: Locating shared libraries
by Peter Tanski 16 Jun '07

16 Jun '07

Spencer Janssen wrote: > On Sat, 16 Jun 2007 08:21:50 +1000 > skaller <skaller(a)users.sourceforge.net> wrote: > >> One way to measure this is: if you removed GHC and applications, >> and there are (necessarily) no users of the remaining library >> package .. the library package shouldn't be in the global public >> place (/usr/lib+include etc). > > As I understand it, the entire point of this effort (shared libraries > in GHC) is to allow dynamically linked Haskell executables. In this > case, applications outside the GHC toolchain will in fact depend on > these shared objects. As a concrete case, a binary darcs package > could > be a user of libghc66-base.so and libghc66-mtl.so -- with no > dependencies on the GHC compiler package itself. > > Does this pass your litmus test? If a binary darcs package requires GHC libraries to function it would be a GHC "application," so it would not pass the test. John Skaller's point is actually very good, though it would require special changes to the current system: distinguishing between GHC- compiled _shared_libraries_ and the GHC compiler itself. Under the current system ghc sits in its own subdirectory, typically / usr/local/lib, /opt/local/lib, etc., depending on your distribution, with only the compiler and "base" libraries. Relatively few Cabal packages install into this directory--HaXml does, if i remember correctly. Most Cabal packagers leave the default install, so applications compiled by GHC are installed as subdirectories in the user library directory (/usr/local/lib). So we have this layout: /usr/local/lib/ghc-6.6.1 <--compiler, ghc-distribution libraries /usr/local/lib/ghc-6.6.1/imports <-- .hi files after installing HaXml: /usr/local/lib/ghc-6.6.1 <- HaXml libraries /usr/local/lib/ghc-6.6.1/imports <-- HaXml .hi files after installing a default Cabal distribution (here, HXT): /usr/local/lib/hxt-$(version) <-- libraries /usr/local/lib/hxt-$(version)/imports <-- hxt .hi files With the current setup is it is a real pain to go through and uninstall all those extra libraries when you upgrade GHC: you must either delete the directories completely (for directories based on version, such as hxt-$(version), above); or, you must go into each directory and delete the old libraries contained there. The ghc-pkg program only handles finding these libraries for the ghc compile process; it does not help with uninstalling them except by tracking location. I would extend John's idea--and reference to gcc's install layout: everything should go into the generic "ghc" directory, say, /usr/ local/lib/ghc; _under_ that should go the compiler version (including arch-platform); _under_ that may go user-libraries. That structure would make it much easier for users to uninstall or upgrade ghc versions. It also gives distributors the flexibility to include a ghc-library-only distribution (no compiler) for packaging applications--say, a basic darcs install (with shared libraries) includes only the shared libraries it needs. A basic "ghc" directory on top--no version until the next subdirectory--also preserves prime real estate toward the top levels of the directory structure: a user may install more than one version of GHC into their home directory without flooding their home directory with ghc-$(version)s. There is a fundamental flaw with all these "new" languages: ABI changes between compiler versions force software vendors to give customers upgrades of the entire program--you can't send a patch if you need to compile all the libraries again. Static libraries are horrible for the same problem, and one of the main reasons I will not use OCaml for any distributable application. Even in-house, a hassle like that means time and lost productivity. Patches are inevitable for new features, security, etc., and changes to the compiler are inevitable just to get around bugs in the compiler! GHC is a research effort so it is a relatively fast moving target--how much Haskell code has bit rotted simply due to changes in the GHC compiler? In the past _three_ years? How much simply due to changes in the _interface_ file (.hi) format? A lot. Too much to mention here. Even if we can't standardise the ABI a bit the solution of where to put libraries should place some real consideration on how much time and trouble it is to uninstall and recompile everything, without resorting to a script or package system to do it for us--you may imagine the bug bath from, say, modifying ghc-pkg to handle uninstallation. Cheers, Pete

1 0

Locating shared libraries
by Clemens Fruhwirth 16 Jun '07

16 Jun '07

Hi, I'm hacking on shared library support for GHC and it's coming along quite nicely. http://hpaste.org/192 My initial hacks are available from: http://clemens.endorphin.org/patches/ghc-20070605-initial-shared-libs.patch (works only with x86-64 atm, on i386 the NCG dies in the register allocator when compiling cmm files RTS) http://clemens.endorphin.org/patches/cabal-20070528-initial-shared-library.… libtool usually takes care of creating shared libraries under *nix system. libtool solves a few minor problems associated with: 1) creating shared libraries 2) linking programs that depend on shared libraries 3) running programs that depend on shared libraries libtool is tailored to C compilers and the general opinion from #ghc towards libtool seems to be: "hands off". From the list above, I will try to sketch solutions without libtool. 1) creating shared libraries: At the moment, my second patch teaches Cabal how to build shared libraries. Basically, this is: * add -fPIC to the compiler invocation (and -optc-fPIC for c-sources), * invoke "ld -shared -Bsymbolic -o foo.so obj1.o obj2.o ...". ATM, ld is not invoked with the inter-library dependencies for the shared library being built. This is not problematic as the final executable will include all dependencies due to the ghc package dependency tracking. But DT_NEEDED on ELF influences the sequence in which shared library initializers are run. I have not yet investigated if this leads to any problems. To solve this little shortcoming, the ld-invocation could be delegated to GHC. "ghc -o libHSfoo.so Foo1.o Foo2.o". We already have a similar facility for DLLs (see MkDLL in DriverPipeline.hs). This could be abstracted into MkShared, and platform specific knowledge could be encapsulated in GHC. The benefit would be that we could easily access the package information and we could create shared libraries that contain proper DT_NEEDED sections. 2) Linking programs Linking should work out of the box: "ghc -dynamic -o HelloWorld HelloWorld.o" creates dynamically linked executable. 3) Running programs This a typical problem: ./HelloWorld ./HelloWorld: error while loading shared libraries: libHShaskell98-1.0_dyn.so: cannot open shared object file: No such file or directory There are several ways to add search paths for dynamic linking: either we do it temporarily or we encode the search paths into the executables. On ELF platforms, this works by adding -rpath to the linker flags. This adds two new entries in the .dynamic section (DT_RPATH, DT_RUNPATH) both responsible for signalling additional search paths to the dynamic linker, ld.so. According to Simon Marlow, Windows has similar mechanism via manifest files. Let's see how libtool handles this situation. libtool differentiates between installed and uninstalled libraries. When linking against installed libraries not in the standard search path, libtool uses -rpath to add these search paths to the created executable. When linking against uninstalled libraries, libtool still uses -rpath but pointing to the directory the uninstalled library is going to be installed in. libtool derives this information from the .la files+Makefiles. In any case, libtool creates a wrapper in the build directory that takes care of executing the program linked against uninstalled shared libraries. There are two strategies for accomplishing this: * add the paths of the uninstalled shared libraries to LD_LIBRARY_PATH * relink the executable with additional -rpath's libtool chooses the second strategy. How do we translate these solutions to GHC? The first question is whether we expect ghc -dynamic -package uninstalled-package -o Hello Hello.o ./Hello to work or whether we require manual intervention in these cases. If we expect this to work without intervention, we have the same options as libtool: * create a wrapper that takes care of locating the uninstalled shared libraries and sets LD_LIBRARY_PATH. * create a binary with rpath of the uninstalled libraries, and create an additional executable for deployment without these rpaths. In any case we have to modify the installer scripts to either know about where to locate the real binary, either ask ghc where to find the real binary, or either delegate the installation to ghc. The last option is basically the libtool way, "libtool --mode=install ..." When we decide to create a deployable executable at the "-o" spot, we need to * modify the invocation to manually pick up the libraries by modifying LD_LIBRARY_PATH.. this is pretty unpractical. * delegate invocation to ghc. Maybe "ghc --execute HelloWorld". libtool has a similar mechanism for executing 3rd party programs in the "dynamic environment" of the compiler executable. For instance, "gdb HelloWorld" would fails for libtool as HelloWorld is a wrapper, but "libtool --mode=execute gdb HelloWorld" works, as libtool rewrites to HelloWorld to .libs/lt-HelloWorld. And now something completely different: Create a custom ELF program interpreter for Haskell programs. Using INTERP in the ELF program header, loads up this interpreter and delegates control to it. Usually this is /lib/ld-linux.so.2, the dynamic linker, but we can replace that. Haskell has its own idea of libraries/packages. We have package.conf which gives us the location of the installed libraries. This is ok for static linking, as at link time ghc is running and knows how to invoke gcc with the correct paths. It does not matter, if package.conf is updated afterwards as the statically linked programs contain a copy of the library anyway. For dynamic linking this phase is delayed and when we encode rpath such as "/usr/lib/network-2.0/ghc-6.6/", we can not update to network-2.1 without breaking this executables. A custom programming loading stub could access the global and local package.conf and extract the library path for the dependencies and execve /lib64/ld-linux.so.2 --library-path=<paths of the dependencies> HelloWorld <args> This certainly gives us more flexibility than encoding all these rpaths statically into HelloWorld. To solve the inplace execution directly from the build directory, we might create .HelloWorld.package.conf in case a non-standard package.conf is used (non-standard=different from global and local) and have the stub loader to check for this file. I agree that the last scheme sounds a bit wild, but I argue that that's what ELF designers had in mind when they specified the INTERP header. Of course, this is only a solution for ELF platforms. Opinions :) ? -- Fruhwirth Clemens - http://clemens.endorphin.org

9 19

DoCon soft announcement
by Serge D. Mechveliani 15 Jun '07

15 Jun '07

Software Announcement --------------------- The Algebraic Domain Constructor DoCon, Version 2.09, written in Haskell, is available, together with its source program, on Internet at the following addresses: http://www.botik.ru/~mechvel/ click at docon-2.09 (Russian site), ftp.botik.ru/pub/local/Mechveliani/docon/ (same site), http://www.haskell.org/docon/distrib/ (USA site) The difference to previous (2.08) release is small: * it is ported to the latest (on 2007) Glasgow Haskell version of 6.6.1, * it has a simpler installation code, due to that it relies on the commands `configure', `build', `install' of the Cabal support for `making', * several bugs are fixed, * the implementation for decomposing to elementary symmetrics is improved. ----------------- Serge Mechveliani mechvel(a)botik.ru

1 0

Running multithreaded GHC on Itanium
by Srđan Stipić 14 Jun '07

14 Jun '07

I am currently trying to modify GHC 6.6.1 to run on Itanium in multithreaded mode (+RTS -N). Is it possible to do those modifications without generating back-end for Itanium? (If I am correct GHC on Itanium uses gcc to generate the code) Thanks Srdjan

2 1

GHC 6.6 panics when compiling HSet from collections package
by Daniel McAllansmith 13 Jun '07

13 Jun '07

Hi. I just got the collections package from http://darcs.haskell.org/packages/collections-ghc6.6 When trying to build it with GHC 6.6 on an amd64 linux machine using Cabal I got the following: [12 of 57] Compiling Data.Tree.AVL.IntMap.Internals.HSet ( Data.Tree.AVL.IntMap/Data/Tree/AVL/IntMap/Internals/HSet.hs, dist/build/Data/Tree/AVL/IntMap/Internals/HSet.o ) ghc-6.6: panic! (the 'impossible' happened) (GHC version 6.6 for x86_64-unknown-linux): cgPanic tpl{v s2zg} [lid] static binds for: collections-0.3:Data.Tree.AVL.IntMap.Internals.HSet.intersectionMaybeH{v rji} [gid] collections-0.3:Data.Tree.AVL.IntMap.Internals.HSet.$Lr2jfforkL{v r2jf} [gid] collections-0.3:Data.Tree.AVL.IntMap.Internals.HSet.$Lr2jhforkR{v r2jh} [gid] local binds for: SRT label collections-0.3:Data.Tree.AVL.IntMap.Internals.HSet.intersectionWithH'{v rjg}_srt Anyone seen this before? A real GHC bug, a problem with my GHC installation, or a problem with the collections package? Cheers Daniel

2 1

> 2^32 b, 6.6 for Itanium
by Serge D. Mechveliani 13 Jun '07

13 Jun '07

Dear GHC developers, I have the two questions. 1. How ghc-6.6 runs on 64-bit machines having more that 2^32 byte memory ? Can it run into "Segmentation fault" ? 2. Can you prepare a reliable ghc-6.6 binary for Itanium a64 Linux ? Or maybe, someone can contribute ? One of my colleagues says the following. ------------------------------------- He has taken such a binary from > (http://www.haskell.org/ghc/distribution_packages.html#debian) > somehow installed it and tried to make from source. It reports >/tmp/ghc12430_0/ghc12430_0.s: Assembler messages: > >/tmp/ghc12430_0/ghc12430_0.s:51:0: > Error: Operand 1 of `mov' should be a general register ------------------------------------- (though, I have not looked so far, in how all this was done). Regards, ----------------- Serge Mechveliani mechvel(a)botik.ru

2 1

unneeded laziness example
by Serge D. Mechveliani 12 Jun '07

12 Jun '07

People, I wrote recently > And now unneeded laziness bites. > [..] > In ghc-6.6, it needs at least 100 Mb of heap to perform. > If I set seq infoK' between `shows' and `p', then it suffices 14 Mb. > [..] > Now: how to program real problems in Haskell ? > Should I try to set seq everywhere and see whether it helps ? I am sorry. This was false alarm. I continued the investigation and have found that Haskell and GHC are NOT guilty in this example (uh! how good). This is the specifics of the method. Probably, one needs to think of seq only if one applies such a method with expenses depending heavily on the computation order. An interesting thing, though. The simplified and contrived example is --------------------------------------------------- main = putStr $ test 18 test :: Integer -> String test w = shows (f tM) "\n" where f = Prelude.minimum . map Prelude.minimum mM = makeMatrix w strictnessM = Prelude.minimum $ map Prelude.minimum mM tM = -- seq strictnessM $ transpose mM --------------------------------------------------- Here mM is n by n matrix of small integers, n = 385 for w = 18. The program prints f $ transpose mM. makeMatrix computes the rows of mM in a complex way, finding m(i,j) uses the table of m(i',j') for other pairs (i',j'). And the time and space costs depend heavily on the order in which m(i,j) are computed. Applying `transpose' changes (due to Haskell laziness) this order into more expensive one: 5 times more of heap needed. And adding `seq ...' before `transpose' returns it to initial order. So, everything is natural. Regards, ----------------- Serge Mechveliani mechvel(a)botik.ru

1 0

Seemingly subtle change causes large performance variation
by Matthew Danish 12 Jun '07

12 Jun '07

Hello, I've been playing with the INTEST problem on SPOJ which demonstrates the ability to write a program which processes large quantities of input data. http://www.spoj.pl/problems/INTEST/ I came across some curious behavior while cleaning up the program. The original program, which runs fast (enough), is: module Main(main) where import Control.Monad import Data.Maybe import qualified Data.ByteString.Char8 as B divisibleBy :: Int -> Int -> Bool a `divisibleBy` n = a `rem` n == 0 main :: IO () main = do [n,k] <- (map int . B.split ' ') `fmap` B.getLine :: IO [Int] let doLine :: Int -> Int -> IO Int doLine r _ = B.getLine >>= testDiv r testDiv r l | int l `divisibleBy` k = return (r + 1) | otherwise = return r foldM doLine 0 [1..n] >>= print where int :: B.ByteString -> Int int = fst . fromJust . B.readInt But when I make a slight modification, the program chews up a ton more memory and takes more time: module Main(main) where import Control.Monad import Data.Maybe import qualified Data.ByteString.Char8 as B divisibleBy :: Int -> Int -> Bool a `divisibleBy` n = a `rem` n == 0 main :: IO () main = do [n,k] <- (map int . B.split ' ') `fmap` B.getLine :: IO [Int] let doLine :: Int -> Int -> IO Int doLine r _ = B.getLine >>= return . testDiv r -- 'return' moved here ^^ testDiv r l | int l `divisibleBy` k = r + 1 | otherwise = r foldM doLine 0 [1..n] >>= print where int :: B.ByteString -> Int int = fst . fromJust . B.readInt This program will generate sample data: import System.Random import System.Environment import Control.Monad main = do [n] <- map read `fmap` getArgs :: IO [Int] k <- randomRIO (1, 100) putStrLn $ unwords [show n, show k] replicateM_ n $ randomRIO (1, 10^9) >>= print Note that the same behavior occurs even if I manually inline the local function and try: return (if .. then .. else). Some sample runs: $ ghc/compiler/ghc-inplace ghc-6.7.20070601: no input files $ ghc/compiler/ghc-inplace --make -O2 intest.hs [1 of 1] Compiling Main ( intest.hs, intest.o ) Linking intest ... $ ghc/compiler/ghc-inplace --make -O2 intest_slow.hs [1 of 1] Compiling Main ( intest_slow.hs, intest_slow.o ) Linking intest_slow ... $ time ./intest +RTS -tstderr -RTS < test1 ./intest +RTS -tstderr 8830 <<ghc: 134876896 bytes, 248 GCs, 28672/28672 avg/max bytes residency (1 samples), 2M in use, 0.00 INIT (0.00 elapsed), 0.12 MUT (0.12 elapsed), 0.00 GC (0.00 elapsed) :ghc>> real 0m0.129s user 0m0.124s sys 0m0.006s $ time ./intest_slow +RTS -tstderr -RTS < test1 ./intest_slow +RTS -tstderr 8830 <<ghc: 144278584 bytes, 269 GCs, 7030784/21843968 avg/max bytes residency (6 samples), 38M in use, 0.00 INIT (0.00 elapsed), 0.13 MUT (0.14 elapsed), 0.10 GC (0.15 elapsed) :ghc>> real 0m0.296s user 0m0.238s sys 0m0.058s -- -- Matthew Danish -- user: mrd domain: cmu.edu -- OpenPGP public key: C24B6010 on keyring.debian.org

2 1

unneeded laziness case
by Serge D. Mechveliani 11 Jun '07

11 Jun '07

People, A couple of times I suffered of unneeded strictness in Haskell. And now unneeded laziness bites. Consider the following program: ------------------------------------------------------- main = let mK = <formed in a certain complex way> :: Map.Map Partition Integer row = [0, 1, 0 ... 0] (mK', p) = decompose mK row infoK' = sum $ map sum $ mapmap (\ n -> rem n 2) $ Map.elems mK' in putStr (shows p -- seq infoK' "\n") ------------------------------------------------------- In ghc-6.6, it needs at least 100 Mb of heap to perform. If I set seq infoK' between `shows' and `p', then it suffices 14 Mb. If I set there seq mK', then again it needs 100 Mb. 7 times as large. mK is a triangular n by n matrix, n is about 900 (so, there are about 1 million of elements in mK). mK has small integers in it. p is found actually by solving the linear system X x mK = row. Now: how to program real problems in Haskell ? Should I try to set seq everywhere and see whether it helps ? Also note: seq mK' was not sufficient. I do not know, probably, as mK' has structure, seq does not evaluate all its levels. Therefore the programmer needs to invent a function that gathers much of information of mK' and puts it into one Integer (infoK' = sum ... mK'). Then, seq infoK' forces most of evaluation of mK'. Is this a generic style to program real problems ? To write all this strange code instead of plain let (_, p) = decompose mK row in putStr (shows p "\n") ? The story was as follows. My friend asked me to decompose certain large symmetric polynomial into elementary symmetric ones. This was needed for a real mathematical problem. I applied my DoCon program, which is written in Haskell. 1000 Mb RAM was not enough. I started to investigate, because I suspected that this case needs several times smaller memory. I found a fragment of the type shown above. And I never use `seq'. I have put a bit smaller example to make it 300 Mb sufficient. The effect was discovered by occasion. I have forgotten that `decompose' returns a pair with a large matrix as fst and wrote let p = decompose mK row in putStr $ show p It has computed in a relatively small memory, and fast, but before the p value, it printed several pages of the matrix value. I said "Oh! we need to improve this, we only need p, the second part": let (_, p) = decompose mK row in putStr $ show p And this takes 7 times more of memory and 2 times more of time! Then, I understood, what is happening and verified the idea by introducing seq. And now, I wonder ... Thank you in advance for your help and comments. Regards, ----------------- Serge Mechveliani mechvel(a)botik.ru

1 0

Failure building HEAD
by Gregory Wright 08 Jun '07

08 Jun '07

Hi, Today I was building HEAD using macports and had the error: configure: Using compiler: ../../compiler/ghc-inplace configure: Compiler flavor: GHC configure: Compiler version: 6.7.20070607 configure: Using package tool: ../../utils/ghc-pkg/ghc-pkg-inplace checking for gcc... gcc checking for C compiler default output file name... configure: error: C compiler cannot create executables See `config.log' for more details. (this was in libraries/base). The config.log showed the problem: configure:1680: checking for C compiler default output file name configure:1683: gcc -I/opt/local/include --configure-option=-O2 -I/ opt/local/include --configure-option=-I/opt/local/include -L/opt/ local/lib --configure-option=-L/opt/local/lib conftest.c >&5 cc1: error: unrecognized command line option "-fconfigure-option=-O2" cc1: error: unrecognized command line option "-fconfigure-option=-I/ opt/local/include"cc1: error: unrecognized command line option "- fconfigure-option=-L/opt/local/lib"configure:1686: $? = 1 It looks like cabal is messed up, passing "--configure-option=" when it should be stripping it out. Is this a problem with the HEAD branch or do I need a development version of cabal from the darcs repository to build the latest ghc? Best Wishes, Greg

2 4