- Glasgow-haskell-users

RE: "foreign export dynamic" in GHC
by Simon Marlow 08 Aug '01

08 Aug '01

[ moved to glasgow-haskell-users ] > When I make a function generator with "foreign export > dynamic", GHC spits > out stub files which it apparently then needs for the link. What are > these for? The stub files contain the helper function which invokes the Haskell code when called from C. Each time the foreign export dynamic is invoked to create a new callback function, a small piece of code has to be dynamically generated. It is the address of this dynamically generated bit of code that is returned as the Addr (or Ptr). When called from C, the dynamically generated code must somehow invoke the Haskell function which was originally passed to the f.e.d. function - it does this by invoking the helper function, passing it a StablePtr to the Haskell function. It's split this way for two reasons: the same helper function can be used each time the f.e.d. function is called, and to keep the amount of dynamically generated code to a minimum. Cheers, Simon

1 0

RE: adding isWHNF primop to 5.00.2 native code generator
by Julian Seward (Intl Vendor) 08 Aug '01

08 Aug '01

| In all the cases I tried, the info pointer in the closure | header pointed to the end of the info table. Although a | comment in ghc/includes/ClosureMacros.h worries me: | | "info pointer" The first word of the closure. Might point | to either the end or the beginning of the | info table, depending on whether we're using | the mini interpretter or not. GET_INFO(c) | retrieves the info pointer of a closure. | | Can someone please clarify when the info pointer will point | to the beginning or end of the table? Normally, the info pointer points at the first word following the info table. This word is the start of the closure's entry code. That means that, given an info pointer, we can either evaluate the closure by executing from that address, or look up info about it by indexing backward from it into the info table proper. However, for portable (-unreg) compilation, we can't control the relative placement of code and data. In this case the first word of the itbl points to the entry code. For nativeGen hacking you only need to consider the fast (pointer to end) case. | My problem is that this is a dodgy assumption since the design of the | info table may change in the future, and you get a nasty | unchecked dependency between the primops in the NCG and the | design of the rts. This is a major design problem (IMO) with the NCG. It afflicts every single macro and structure layout which the NCG has to deal with, and has already cost us some hours tracing bugs to do with divergences between the two implementations. I had a plan to derive all the structure layouts and macro defs from a single source to avoid this kind of breakage, but I don't know if it will ever be implemented. | It seems hard to get the kind of automatic offset calculation in the | NCG as you get with the C version of primops. My current idea | is to make some C primops that do the structure and array | indexing for me, and then use them in the NCG. i.e. primops | that essentially export 'closure_flags[type]' and | 'get_itbl(c)->type' to the NCG. It's a problem. StIndex is sometimes helpful, particularly since you can specify a PrimRep which indicates how the index should be scaled. Exactly like adding an integer to a pointer in C. J

1 0

Re: GHC FFI Return Type Bug
by Ashley Yakeley 07 Aug '01

07 Aug '01

At 2001-08-07 11:13, Carl R. Witty wrote: >You can see that the code for f is: > call g > andl $255,%eax > ret >So gcc believes that a function which returns a value of type unsigned >char is not responsible for clearing the high 3 bytes of %eax. This is to be expected; in most cases the caller will not be doing any kind of extension and so the high three bytes won't need to be masked at all. Only the caller knows whether extension is necessary, and (as Sigbjorn points out) whether it needs to do signed or unsigned extension. -- Ashley Yakeley, Seattle WA

1 0

Re: GHC FFI Return Type Bug
by cwitty＠newtonlabs.com 07 Aug '01

07 Aug '01

"Sigbjorn Finne" <sof(a)galconn.com> writes: > "Julian Seward (Intl Vendor)" <v-julsew(a)microsoft.com> writes: > > > > Hmm, we're looking at this. However, I don't really know what > > C is or is not supposed to do here. Given > > > > char fooble ( ... ) > > { > > return 'z'; > > } > > > > on an x86, 'z' will be returned at the lowest 8 bits in %eax. > > What I don't know is, is the C compiler obliged to clear the > > upper 24 bits of %eax, or does that onus fall on the callee? > > Yes, the callee is required to narrow <expr> in 'return <expr>;' to > fit that of the specified return type -- see 9.8 of Harbison and > Steele. So, a C compiler that cause f() to return 0x7fffffff for > the following, > > unsigned char g() > { > return 0x7fffffff; > } > > unsigned int f() > { > return g(); > } > > is in the wrong. [Notice that narrowing for signed integral types > is undefined in ISO C, but most current-day compilers implement > such narrowing ops the same way, i.e., by masking off excess bits.] It's certainly true that an ISO C compiler on a typical machine must return 255 from f() (on an atypical machine, it's possible to have unsigned char be a 32-bit type). However, this is essentially unrelated to the question of whether the x86 ABI allows g() to return a value in %eax that has the upper 3 bytes non-zero. When I compile the following file: -------------------- abitest.c -------------------- unsigned int g_val; unsigned char g() { return g_val; } unsigned int f() { return g(); } --------------------------------------------------- with the command gcc -Wall -O2 -fomit-frame-pointer -S abitest.c I get the output: -------------------- abitest.S -------------------- .file "abitest.c" .version "01.01" gcc2_compiled.: .text .align 16 .globl g .type g,@function g: movzbl g_val,%eax ret .Lfe1: .size g,.Lfe1-g .align 16 .globl f .type f,@function f: call g andl $255,%eax ret .Lfe2: .size f,.Lfe2-f .comm g_val,4,4 .ident "GCC: (GNU) 2.7.2.3" --------------------------------------------------- You can see that the code for f is: call g andl $255,%eax ret So gcc believes that a function which returns a value of type unsigned char is not responsible for clearing the high 3 bytes of %eax. Carl Witty

1 0

RE: POLL: GC options
by Simon Marlow 07 Aug '01

07 Aug '01

> In local.glasgow-haskell-users, you wrote: > > Issue 1: should the maximum heap size be unbounded by default? > > Currently the maximum heap size is bounded at 64M. > Arguments for: this > > stops programs with a space leak eating all your swap > space. Arguments > > against: it's annoying to have to raise the limit when you > legitimately > > need more space. > > I'm for boundless wasting of memory. If I'd cared, I'd set the > ulimits correspondingly. You really don't want to get to work > on Monday morning finding that your favourite prime-finder stopped > Friday five minutes after you left :-) The discussion is somewhat moot, since I removed the default limit anyway, but I just thought I'd point out that there's an important difference between GHC's max heap size and the ulimit: GHC will adjust the sizes of the generations to try to stay within the max heap size (thus trading performance for space), whereas it doesn't do that for the ulimit. Anyway, the upshot is that you'll get more failures setting the ulimit to X than when setting the max heap size to X. I certainly recommend setting either a ulimit or a maximum heap size, especially when doing development. In my experience, modern OS's which do memory overcommit tend to behave very badly when something eats all the swap: if you're lucky, your Haskell process will get killed, but if you're unlucky, some other random process(es) will get killed or the machine will crash altogether. Besides, if you're on a multi-user box using up all the swap isn't very sociable :-) An interesting idea is for GHC to read the current ulimit settings and set the max heap size accordingly; however this doesn't seem to be straightforward. There are four settings: the data seg size, the resident set size, the virtual memory size and the locked-in memory size. The data seg size doesn't affect GHC-compiled programs, because we mmap() the heap. The resident set size similarly doesn't seem to have any effect on Linux. The locked-in memory size doesn't affect us (we don't use mlock()), which leaves only the virtual memory size. This one sets a bound on the whole process size, including the size of the binary itself any memory that has been malloc()ed, so it isn't easy to figure out how much is left for the heap. Cheers, Simon

1 0

POLL: GC options
by Simon Marlow 07 Aug '01

07 Aug '01

Folks, There is some disagreement over how the GC options should be specified for Haskell programs. I've identified a couple of issues below, comments and opinions are greatly appreciated. If there's a concensus that things should be changed, then I'll make the changes for the next release. Issue 1: should the maximum heap size be unbounded by default? Currently the maximum heap size is bounded at 64M. Arguments for: this stops programs with a space leak eating all your swap space. Arguments against: it's annoying to have to raise the limit when you legitimately need more space. Options: 1. remove the default limit altogether 2. raise the default limit 3. no change (any others?) Issue 2: Should -M be renamed to -H, and -H renamed to something else? The argument for this change is that GHC's -M option is closer to the traditional meaning of -H. Issue 3: (suggestion from Julian S.) Perhaps there should be two options to specify "optimise for memory use" or "optimise for performance", which have the effect of setting the defaults for various GC options to appropriate values. Optimising for memory use might enable the compacting collector all the time, for instance. Optimising for performance is hard - we may be able to change some of the defaults to trade space for time, but it's unlikely to be entirely reliable (eg. turning on the compacting collector sometimes improves performance, sometimes reduces it). Cheers, Simon > -----Original Message----- > From: Julian Seward (Intl Vendor) > Sent: Monday, August 06, 2001 10:54 AM > To: Simon Marlow; Marcin 'Qrczak' Kowalczyk; cvs-ghc(a)haskell.org > Subject: RE: cvs commit: fptools/ghc/compiler/stranal DmdAnal.lhs > > > | It appears that no-one understands the GC options except me. > | Perhpas they should be redesigned, but here's the current story: > | > | -H<size> is the *minimum* heap size. The heap will be grown > | as necessary, starting with the minimum, and up to the > | maximum heap size. > | > | -M<size> is the *maximum* heap size, by default 64M. The > | heap is not allowed to grow beyond this size [...] > > In days long since gone (before the 4.XX RTS) -H simply set > the heap size, and because the heap was fixed-size, that was > the max heap size too. > > How about: > > * Renaming current -M to -H, and current -H to -HS. > * Fixing up the sizing calculations a bit so that the > max heap size is more closely observed. > > Result: -H means what it meant originally, but you can still > set an initial heap size with -HS if you want. That means > for the most part we can forget about -M and use -H instead. > > Also having an auto-fallback to compacting collection when > heap gets full. Overall aim is to reduce, ideally to zero, > the number of flags users have to give to the RTS in order to > get reasonable performance yet efficient use of memory. > People simply won't use the compacting collector if you have > to ask for it specially. > > J >

5 4

RE: GC options. Reply
by Simon Marlow 07 Aug '01

07 Aug '01

> "Simon Marlow" <simonmar(a)microsoft.com> writes: > > > Ok, I've done the following: > > > - disabled the maximum heap > > > - RTS options are taken from the GHCRTS environment variable > > in addition to the command line (the command line has > precedence). > > (I haven't followed this debate closely, so I may be missing out on > something, but:) > > Perhaps it could be possible to embed a default in > the executable as well, through a command line option to the > compiler/linker? You know, for the occasional program that actually > requires more heap than one is comfortable declaring in GHCRTS, but > one is too lazy to write a shell wrapper for? You can do this already, though not with a compiler option. It would be possible to do it with a compiler option, but it would be tricky to implement - basically the compiler would have to generate a small C file, compile it, and link it into the excecutable, and somehow avoid conflicting with any other hooks that the programmer had already defined. So, I'm enclined not to do this on the grounds that the payoff to effort ratio is small. But, as usual, if there's strong demand then we'll probably do it anyway. Cheers, Simon

1 0

Re: GC options. Reply
by Ketil Malde 07 Aug '01

07 Aug '01

"Simon Marlow" <simonmar(a)microsoft.com> writes: > Ok, I've done the following: > - disabled the maximum heap > - RTS options are taken from the GHCRTS environment variable > in addition to the command line (the command line has precedence). (I haven't followed this debate closely, so I may be missing out on something, but:) Perhaps it could be possible to embed a default in the executable as well, through a command line option to the compiler/linker? You know, for the occasional program that actually requires more heap than one is comfortable declaring in GHCRTS, but one is too lazy to write a shell wrapper for? -kzm -- If I haven't seen further, it is by standing in the footprints of giants

1 0

Re: POLL: GC options
by Marcin 'Qrczak' Kowalczyk 07 Aug '01

07 Aug '01

Mon, 06 Aug 2001 15:40:50 -0700, Thomas Hallgren <hallgren(a)cse.ogi.edu> pisze: > Regarding the maximum heap size, to avoid letting the heap grow too > large, you could perhaps take into account the number of page faults > that occur during garbage collection, or the ratio between CPU time and > real time... A disadvantage of taking many factors into account is that the same program will non-deterministically run or fail. IMHO a default maximum heap size should be well-defined: either based on an environment variable or without limits or a fixed value. It would be bad for example to *automatically* set it depending on the free physical memory, because it would lead to the following: - your program doesn't compile - sorry, works for me - compilation dies with out of memory - you must have low physical memory; please set a flag: it will swap a lot but will finally compile - thanks, I don't know why but now it compiled without setting any flags. It would not be a problem if the limit was reached very rarely. Unfortunately it's not the case. -- __("< Marcin Kowalczyk * qrczak(a)knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK

1 0

RE: Dependency Generation
by Steinitz, Dominic J 07 Aug '01

07 Aug '01

I didn't realise I also had to put this on the line that works out the dependencies. In hindsight it's obvious. Perhaps the documentation could be updated with a note? I'd be happy to supply some text. Dominic. simonmar(a)microsoft.com on 07/08/2001 10:07:00 To: Dominic Steinitz glasgow-haskell-users cc: bcc: Subject: RE: Dependency Generation > I'm getting the following error using the instructions in > 4.9.5.1 Dependency > Generation. > > [dominic@r341-02 webldap]$ make depend > ghc -M -cpp Add.hs EnvPassed.hs ParseError.hs Search.hs > ASNv1.hs Counter.hs GetO > pt.hs ParseExpr.hs StdTokenDef.hs Basev1.hs DecodeAll.hs > Hello.hs ParseLib.hs Ta > gsv1.hs CGI.hs Decode.hs HTML.hs Parser.hs TokenDef.hs > CGISystem.hs Delete.hs HT > MLWizard.hs ParseToken.hs UrlEncoded.hs Classv1.hs DoAdd.hs > ldapHTML.hs Pretty.h > s WebSite.hs CL.hs DoDelete.hs LdapTypesv2.hs QL.hs > X690Utilsv2.hs Client2.hs Do > Modify.hs Mime.hs query.hs client.hs DoSearch.hs Modify.hs > Request.hs Client.hs > Env.hs oldDecodeAll.hs Response.hs > Basev1.hs: can't find one of the following: `Word.hi' > `Word.hs' `Word.lhs' > make: *** [depend] Error 1 > [dominic@r341-02 webldap]$ Just add the same -package options when you use ghc -M as you would when compiling any of the modules. i.e. in this case adding -package lang will help GHC find the Word module (but it won't be included in the dependency output). Cheers, Simon ------------------------------------------------------------------------------------------------- 21st century air travel http://www.britishairways.com

1 0