- Glasgow-haskell-users

RE: POLL: GC options
by Simon Marlow 07 Aug '01

07 Aug '01

> In local.glasgow-haskell-users, you wrote: > > Issue 1: should the maximum heap size be unbounded by default? > > Currently the maximum heap size is bounded at 64M. > Arguments for: this > > stops programs with a space leak eating all your swap > space. Arguments > > against: it's annoying to have to raise the limit when you > legitimately > > need more space. > > I'm for boundless wasting of memory. If I'd cared, I'd set the > ulimits correspondingly. You really don't want to get to work > on Monday morning finding that your favourite prime-finder stopped > Friday five minutes after you left :-) The discussion is somewhat moot, since I removed the default limit anyway, but I just thought I'd point out that there's an important difference between GHC's max heap size and the ulimit: GHC will adjust the sizes of the generations to try to stay within the max heap size (thus trading performance for space), whereas it doesn't do that for the ulimit. Anyway, the upshot is that you'll get more failures setting the ulimit to X than when setting the max heap size to X. I certainly recommend setting either a ulimit or a maximum heap size, especially when doing development. In my experience, modern OS's which do memory overcommit tend to behave very badly when something eats all the swap: if you're lucky, your Haskell process will get killed, but if you're unlucky, some other random process(es) will get killed or the machine will crash altogether. Besides, if you're on a multi-user box using up all the swap isn't very sociable :-) An interesting idea is for GHC to read the current ulimit settings and set the max heap size accordingly; however this doesn't seem to be straightforward. There are four settings: the data seg size, the resident set size, the virtual memory size and the locked-in memory size. The data seg size doesn't affect GHC-compiled programs, because we mmap() the heap. The resident set size similarly doesn't seem to have any effect on Linux. The locked-in memory size doesn't affect us (we don't use mlock()), which leaves only the virtual memory size. This one sets a bound on the whole process size, including the size of the binary itself any memory that has been malloc()ed, so it isn't easy to figure out how much is left for the heap. Cheers, Simon

1 0

POLL: GC options
by Simon Marlow 07 Aug '01

07 Aug '01

Folks, There is some disagreement over how the GC options should be specified for Haskell programs. I've identified a couple of issues below, comments and opinions are greatly appreciated. If there's a concensus that things should be changed, then I'll make the changes for the next release. Issue 1: should the maximum heap size be unbounded by default? Currently the maximum heap size is bounded at 64M. Arguments for: this stops programs with a space leak eating all your swap space. Arguments against: it's annoying to have to raise the limit when you legitimately need more space. Options: 1. remove the default limit altogether 2. raise the default limit 3. no change (any others?) Issue 2: Should -M be renamed to -H, and -H renamed to something else? The argument for this change is that GHC's -M option is closer to the traditional meaning of -H. Issue 3: (suggestion from Julian S.) Perhaps there should be two options to specify "optimise for memory use" or "optimise for performance", which have the effect of setting the defaults for various GC options to appropriate values. Optimising for memory use might enable the compacting collector all the time, for instance. Optimising for performance is hard - we may be able to change some of the defaults to trade space for time, but it's unlikely to be entirely reliable (eg. turning on the compacting collector sometimes improves performance, sometimes reduces it). Cheers, Simon > -----Original Message----- > From: Julian Seward (Intl Vendor) > Sent: Monday, August 06, 2001 10:54 AM > To: Simon Marlow; Marcin 'Qrczak' Kowalczyk; cvs-ghc(a)haskell.org > Subject: RE: cvs commit: fptools/ghc/compiler/stranal DmdAnal.lhs > > > | It appears that no-one understands the GC options except me. > | Perhpas they should be redesigned, but here's the current story: > | > | -H<size> is the *minimum* heap size. The heap will be grown > | as necessary, starting with the minimum, and up to the > | maximum heap size. > | > | -M<size> is the *maximum* heap size, by default 64M. The > | heap is not allowed to grow beyond this size [...] > > In days long since gone (before the 4.XX RTS) -H simply set > the heap size, and because the heap was fixed-size, that was > the max heap size too. > > How about: > > * Renaming current -M to -H, and current -H to -HS. > * Fixing up the sizing calculations a bit so that the > max heap size is more closely observed. > > Result: -H means what it meant originally, but you can still > set an initial heap size with -HS if you want. That means > for the most part we can forget about -M and use -H instead. > > Also having an auto-fallback to compacting collection when > heap gets full. Overall aim is to reduce, ideally to zero, > the number of flags users have to give to the RTS in order to > get reasonable performance yet efficient use of memory. > People simply won't use the compacting collector if you have > to ask for it specially. > > J >

5 4

RE: GC options. Reply
by Simon Marlow 07 Aug '01

07 Aug '01

> "Simon Marlow" <simonmar(a)microsoft.com> writes: > > > Ok, I've done the following: > > > - disabled the maximum heap > > > - RTS options are taken from the GHCRTS environment variable > > in addition to the command line (the command line has > precedence). > > (I haven't followed this debate closely, so I may be missing out on > something, but:) > > Perhaps it could be possible to embed a default in > the executable as well, through a command line option to the > compiler/linker? You know, for the occasional program that actually > requires more heap than one is comfortable declaring in GHCRTS, but > one is too lazy to write a shell wrapper for? You can do this already, though not with a compiler option. It would be possible to do it with a compiler option, but it would be tricky to implement - basically the compiler would have to generate a small C file, compile it, and link it into the excecutable, and somehow avoid conflicting with any other hooks that the programmer had already defined. So, I'm enclined not to do this on the grounds that the payoff to effort ratio is small. But, as usual, if there's strong demand then we'll probably do it anyway. Cheers, Simon

1 0

Re: GC options. Reply
by Ketil Malde 07 Aug '01

07 Aug '01

"Simon Marlow" <simonmar(a)microsoft.com> writes: > Ok, I've done the following: > - disabled the maximum heap > - RTS options are taken from the GHCRTS environment variable > in addition to the command line (the command line has precedence). (I haven't followed this debate closely, so I may be missing out on something, but:) Perhaps it could be possible to embed a default in the executable as well, through a command line option to the compiler/linker? You know, for the occasional program that actually requires more heap than one is comfortable declaring in GHCRTS, but one is too lazy to write a shell wrapper for? -kzm -- If I haven't seen further, it is by standing in the footprints of giants

1 0

Re: POLL: GC options
by Marcin 'Qrczak' Kowalczyk 07 Aug '01

07 Aug '01

Mon, 06 Aug 2001 15:40:50 -0700, Thomas Hallgren <hallgren(a)cse.ogi.edu> pisze: > Regarding the maximum heap size, to avoid letting the heap grow too > large, you could perhaps take into account the number of page faults > that occur during garbage collection, or the ratio between CPU time and > real time... A disadvantage of taking many factors into account is that the same program will non-deterministically run or fail. IMHO a default maximum heap size should be well-defined: either based on an environment variable or without limits or a fixed value. It would be bad for example to *automatically* set it depending on the free physical memory, because it would lead to the following: - your program doesn't compile - sorry, works for me - compilation dies with out of memory - you must have low physical memory; please set a flag: it will swap a lot but will finally compile - thanks, I don't know why but now it compiled without setting any flags. It would not be a problem if the limit was reached very rarely. Unfortunately it's not the case. -- __("< Marcin Kowalczyk * qrczak(a)knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK

1 0

RE: Dependency Generation
by Steinitz, Dominic J 07 Aug '01

07 Aug '01

I didn't realise I also had to put this on the line that works out the dependencies. In hindsight it's obvious. Perhaps the documentation could be updated with a note? I'd be happy to supply some text. Dominic. simonmar(a)microsoft.com on 07/08/2001 10:07:00 To: Dominic Steinitz glasgow-haskell-users cc: bcc: Subject: RE: Dependency Generation > I'm getting the following error using the instructions in > 4.9.5.1 Dependency > Generation. > > [dominic@r341-02 webldap]$ make depend > ghc -M -cpp Add.hs EnvPassed.hs ParseError.hs Search.hs > ASNv1.hs Counter.hs GetO > pt.hs ParseExpr.hs StdTokenDef.hs Basev1.hs DecodeAll.hs > Hello.hs ParseLib.hs Ta > gsv1.hs CGI.hs Decode.hs HTML.hs Parser.hs TokenDef.hs > CGISystem.hs Delete.hs HT > MLWizard.hs ParseToken.hs UrlEncoded.hs Classv1.hs DoAdd.hs > ldapHTML.hs Pretty.h > s WebSite.hs CL.hs DoDelete.hs LdapTypesv2.hs QL.hs > X690Utilsv2.hs Client2.hs Do > Modify.hs Mime.hs query.hs client.hs DoSearch.hs Modify.hs > Request.hs Client.hs > Env.hs oldDecodeAll.hs Response.hs > Basev1.hs: can't find one of the following: `Word.hi' > `Word.hs' `Word.lhs' > make: *** [depend] Error 1 > [dominic@r341-02 webldap]$ Just add the same -package options when you use ghc -M as you would when compiling any of the modules. i.e. in this case adding -package lang will help GHC find the Word module (but it won't be included in the dependency output). Cheers, Simon ------------------------------------------------------------------------------------------------- 21st century air travel http://www.britishairways.com

1 0

RE: Dependency Generation
by Simon Marlow 07 Aug '01

07 Aug '01

> I'm getting the following error using the instructions in > 4.9.5.1 Dependency > Generation. > > [dominic@r341-02 webldap]$ make depend > ghc -M -cpp Add.hs EnvPassed.hs ParseError.hs Search.hs > ASNv1.hs Counter.hs GetO > pt.hs ParseExpr.hs StdTokenDef.hs Basev1.hs DecodeAll.hs > Hello.hs ParseLib.hs Ta > gsv1.hs CGI.hs Decode.hs HTML.hs Parser.hs TokenDef.hs > CGISystem.hs Delete.hs HT > MLWizard.hs ParseToken.hs UrlEncoded.hs Classv1.hs DoAdd.hs > ldapHTML.hs Pretty.h > s WebSite.hs CL.hs DoDelete.hs LdapTypesv2.hs QL.hs > X690Utilsv2.hs Client2.hs Do > Modify.hs Mime.hs query.hs client.hs DoSearch.hs Modify.hs > Request.hs Client.hs > Env.hs oldDecodeAll.hs Response.hs > Basev1.hs: can't find one of the following: `Word.hi' > `Word.hs' `Word.lhs' > make: *** [depend] Error 1 > [dominic@r341-02 webldap]$ Just add the same -package options when you use ghc -M as you would when compiling any of the modules. i.e. in this case adding -package lang will help GHC find the Word module (but it won't be included in the dependency output). Cheers, Simon

1 0

Dependency Generation
by Steinitz, Dominic J 07 Aug '01

07 Aug '01

I'm getting the following error using the instructions in 4.9.5.1 Dependency Generation. [dominic@r341-02 webldap]$ make depend ghc -M -cpp Add.hs EnvPassed.hs ParseError.hs Search.hs ASNv1.hs Counter.hs GetO pt.hs ParseExpr.hs StdTokenDef.hs Basev1.hs DecodeAll.hs Hello.hs ParseLib.hs Ta gsv1.hs CGI.hs Decode.hs HTML.hs Parser.hs TokenDef.hs CGISystem.hs Delete.hs HT MLWizard.hs ParseToken.hs UrlEncoded.hs Classv1.hs DoAdd.hs ldapHTML.hs Pretty.h s WebSite.hs CL.hs DoDelete.hs LdapTypesv2.hs QL.hs X690Utilsv2.hs Client2.hs Do Modify.hs Mime.hs query.hs client.hs DoSearch.hs Modify.hs Request.hs Client.hs Env.hs oldDecodeAll.hs Response.hs Basev1.hs: can't find one of the following: `Word.hi' `Word.hs' `Word.lhs' make: *** [depend] Error 1 [dominic@r341-02 webldap]$ Here's the relevant bit of the makefile. depend: ghc -M $(HC_OPTS) $(SRCS) SRCS = Add.hs EnvPassed.hs ParseError.hs Search.hs\ ASNv1.hs Counter.hs GetOpt.hs ParseExpr.hs StdTokenDef.hs\ Basev1.hs DecodeAll.hs Hello.hs ParseLib.hs Tagsv1.hs\ CGI.hs Decode.hs HTML.hs Parser.hs TokenDef.hs\ CGISystem.hs Delete.hs HTMLWizard.hs ParseToken.hs UrlEncoded.hs\ Classv1.hs DoAdd.hs ldapHTML.hs Pretty.hs WebSite.hs\ CL.hs DoDelete.hs LdapTypesv2.hs QL.hs X690Utilsv2.hs\ Client2.hs DoModify.hs Mime.hs query.hs\ client.hs DoSearch.hs Modify.hs Request.hs\ Client.hs Env.hs oldDecodeAll.hs Response.hs HC_OPTS= -cpp and here are the imports in Basev1.hs import Int import Char import Bits import Word import Tagsv1 It's not clear from the documentation what I should add to stop ghc -M trying to find Word. I tried -exclude-module=Word.hs but got the following [dominic@r341-02 webldap]$ make depend ghc -M -cpp -exclude-module=Word.hs Add.hs EnvPassed.hs ParseError.hs Search.hs ASNv1.hs Counter.hs GetOpt.hs ParseExpr.hs StdTokenDef.hs Basev1.hs DecodeAll.hs Hello.hs ParseLib.hs Tagsv1.hs CGI.hs Decode.hs HTML.hs Parser.hs TokenDef.hs C GISystem.hs Delete.hs HTMLWizard.hs ParseToken.hs UrlEncoded.hs Classv1.hs DoAdd .hs ldapHTML.hs Pretty.hs WebSite.hs CL.hs DoDelete.hs LdapTypesv2.hs QL.hs X690 Utilsv2.hs Client2.hs DoModify.hs Mime.hs query.hs client.hs DoSearch.hs Modify. hs Request.hs Client.hs Env.hs oldDecodeAll.hs Response.hs ghc-5.00.2: unrecognised flag: -exclude-module=Word.hs Usage: For basic information, try the `--help' option. make: *** [depend] Error 1 [dominic@r341-02 webldap]$ ------------------------------------------------------------------------------------------------- 21st century air travel http://www.britishairways.com

1 0

GC options. Reply
by S.D.Mechveliani 07 Aug '01

07 Aug '01

Hello, here are my votes on Simon Marlow's questions. > Issue 1: should the maximum heap size be unbounded by default? > Currently the maximum heap size is bounded at 64M. > [..] > 1. remove the default limit altogether > 2. raise the default limit > 3. no change Put the default to 30M. > Issue 2: Should -M be renamed to -H, and -H renamed to something else? > The argument for this change is that GHC's -M option is closer to the > traditional meaning of -H. Yes. > Issue 3: (suggestion from Julian S.) Perhaps there should be two options > to specify "optimise for memory use" or "optimise for performance", > which have the effect of setting the defaults for various GC options to > appropriate values. > [..] All right. But let the meaning of -O remain as it was. ----------------- Serge Mechveliani mechvel(a)botik.ru

3 2

RE: GHC FFI Return Type Bug
by Julian Seward (Intl Vendor) 06 Aug '01

06 Aug '01

Hmm, we're looking at this. However, I don't really know what C is or is not supposed to do here. Given char fooble ( ... ) { return 'z'; } on an x86, 'z' will be returned at the lowest 8 bits in %eax. What I don't know is, is the C compiler obliged to clear the upper 24 bits of %eax, or does that onus fall on the callee? Also, I don't even know where to look for the specification of details of the C calling conventions. Anyone got a clue? J | -----Original Message----- | From: Ashley Yakeley [mailto:ashley@semantic.org] | Sent: Monday, August 06, 2001 1:34 AM | To: GHC List | Subject: GHC FFI Return Type Bug | | | Has anyone else come across this? I've only tried this with | Word8 but I | suspect this is a problem with all return types smaller than | four bytes. | | I've entered this as GHC bug #448104. | <http://sourceforge.net/tracker/index.php?func=detail&aid=4481 | 04&group_id=8 | 032&atid=108032> | | Here's the C code: | | extern "C" | { | unsigned char foo(); | } | | inline void use(const char* s) | { | } | | unsigned char foo() | { | const char* s = ""; | use(s); | return 0; | } | | ...and the Haskell looks something like this: | | foreign import foo :: IO Word8 | | do { | w8 <- foo; | putStrLn (show (w8 :: Word8)); | } | | ...and the result is this: | 1074803712 | | Kind of an unusual Word8 value! But note that the low byte of | this value | is zero. | | My setup: | | $ ghc -v | Glasgow Haskell Compiler, Version 5.00, for Haskell 98, | compiled by GHC | version 5.00 | Using package config file: /usr/lib/ghc-5.00/package.conf | Hsc static flags: -static -fignore-interface-pragmas | -fomit-interface-pragmas -fdo-lambda-eta-expansion -flet-no-escape | | $ uname -a | Linux server 2.2.19pre17 #1 Tue Mar 13 22:37:59 EST 2001 i686 unknown | | | -- | Ashley Yakeley, Seattle WA | | | _______________________________________________ | Glasgow-haskell-users mailing list | Glasgow-haskell-users(a)haskell.org | http://www.haskell.org/mailman/listinfo/glasgo| w-haskell-users |

2 1