April 2005 - Libraries

RE: GC trouble [was Re: Binary IO]
by Simon Marlow 23 Apr '05

23 Apr '05

On 23 April 2005 20:12, Jan-Willem Maessen wrote: > Hmm; this sounds like a lot of full-heap collections, which is exactly > what generational GC is trying to avoid. A very large old generation > (like, say, 500+Mb) might help a lot in this instance; I have no idea > how GHC decides generation sizes. It might also help to set a very > large allocation area to reduce promotion rate to the second > generation, and give the gobs of transient data some time to die---or, > similarly, to increase the number of generations to increase the time > it takes things to get to the old generation. Fundamentally, though, > when you run really close to your memory limits GC tends to be > unhappy. For a program with this much data, GHC's default allocation area size of 256k is pretty small. In fact, it's pretty small full stop - I recently bumped it to 512k. I suggest trying -A1m at least. I'd be interested to know whether you get better results by tweaking any of the other GC settings: more generatsions and more steps in particular. Cheers, Simon

1 0

RE: Binary IO
by Simon Marlow 23 Apr '05

23 Apr '05

On 21 April 2005 18:36, Aaron Denney wrote: > On 2005-04-21, Simon Marlow <simonmar(a)microsoft.com> wrote: >> On 21 April 2005 11:16, John Meacham wrote: >> >>> but a useful stopgap, because their use would guarentee your code >>> wouldn't suddenly stop working when ghc changes its IO system. >> >> That isn't going to happen, but I take your point. > > You mean we're not going to fix it? Not at all - just that we don't plan to break existing code if/when a replacement for the current I/O library is introduced. Sorry for the misunderstanding. Does anyone have any *objections* to introducing System.IO.hGetWord8 :: Handle -> IO Word8 System.IO.hPutWord8 :: Word8 -> Handle -> IO Word8 (I prefer these names, please speak up if you prefer hGetByte/hPutByte instead, or something else like hGetOctet/hPutOctet). One constraint would be that the Handle has to be in Binary mode. Lifting this constraint in GHC is a bit tricky because we currenly use the OS's text<->binary translation to do I/O (doesn't matter on Unix, right now). I presume that hPutWord8 (fromIntegral (ord '\n')) should not flush a line-buffered Handle, which is another slight complication. Perhaps line-buffering should be the same as block-buffering on binary Handles. Cheers, Simon

9 10

Cabal Issues
by Ashley Yakeley 23 Apr '05

23 Apr '05

I've just started using Cabal, for my time library implementation (<http://semantic.org/TimeLib/>). I have some issues: 1. If I do "runghc Setup.hs haddock" and then "runghc Setup.hs install", the documentation doesn't appear to get installed anywhere. 2. Likewise, "runghc Setup.hs clean" doesn't delete the documentation. I had expected it to delete the whole dist directory. 3. There ought to be a mechanism for including tests. Something like "runghc Setup.hs check". Right now I have two tests that are executables that are run and the output compared to a reference output file (and another test that must be verified manually). But perhaps this is difficult. -- Ashley Yakeley, Seattle WA

3 3

RE: Binary IO
by Simon Marlow 23 Apr '05

23 Apr '05

On 22 April 2005 21:56, Glynn Clements wrote: > David Brown wrote: > >>>> And hGet/PutWord8 are fast enough for most situations. >>> >>> Are you certain? Which interesting applications do you know >>> that read and write one byte at a time? I can't speak for >>> "most" situations, but in those situations where I needed >>> binary I/O this API would have been impracticable. >> >> I'm going to chime in agreement with the disagreement. >> >> I have one application which uses single character binary I/O. >> Because it works on small files, it borders on usefuleness. As time >> goes on, the size of these inputs grows, and now the tool has become >> nearly worthless. >> >> I seem to run into this kind of thing a lot with Haskell. Dominic's >> ASN library is useful to me, however I'm finding I'll probably just >> have to use it as a template for new code. It works fine for what >> designed for, parsing keys and such, but I'm looking to use it to >> represent gigabytes of data. Processing data like that one Word8 at >> a time isn't practical. > > Personally, I doubt that Haskell will ever be practical for processing > very large amounts of data (e.g. larger than your system's RAM). I hope that's not true - whatever techniques you use in other languages for handling large amounts of data should translate straightforwardly into Haskell. Cheers, SImon

4 5

RE: Cabal Issues
by Simon Marlow 23 Apr '05

23 Apr '05

On 16 April 2005 02:23, Isaac Jones wrote: > Ashley Yakeley <ashley(a)semantic.org> writes: > >> I've just started using Cabal, for my time library implementation >> (<http://semantic.org/TimeLib/>). I have some issues: >> >> 1. If I do "runghc Setup.hs haddock" and then "runghc Setup.hs >> install", the documentation doesn't appear to get installed anywhere. > > Where do folks think that the docs should go? Or should we add a > haddock-prefix flag to configure? GHC puts them in /usr/local/share/<package>/html Perhaps /usr/local/share/doc/<package>/html would be better? Cheers, Simon

2 1

RE: Cabal vs Haskell [sic]
by Simon Peyton-Jones 22 Apr '05

22 Apr '05

| > Moreover, I think Simon does too. He proposes that we should lift the | > overlap restriction by having module names map to (cabal-package, | > module) pairs. | | Well, the overlap restriction exists in the language already. | If you want to lift it, you really need to change the language. | Simon's proposal (which I support) is a hack to permit the restriction | to be lifted without changing existing code, by re-interpreting all | module imports to have an implicit (disambiguating) prefix. It is | just an engineering tradeoff to ensure that huge mounds of legacy code | are not broken purely for the sake of some rather rare overlap case. My motivation is actually this. In a function definition, the caller of the function need not care what the function's local variables are called. But in a Haskell program, the importer of a module M *does* need to care about the names of modules that are imported by M (i.e. are part of M's internal implementation). He needs to care because he can't use those module names. Indeed two modules M and N might simply be unusable in the same program, merely because the implementers of M and N happened to choose the same name for an internal sub-module. I think that's undesirable. The "solution" I was advocating arranges that a *package* must use unique module names, but two packages need not care about the modules used inside each other. But this solution only works smoothly for internal modules. What if packages P and Q both expose a module called M? Then you can't import M-from-P and M-from-Q into the same importing module, because the importing module has no way to say which M he means. And this observation points towards a simpler solution: rather than invisibly pre-pend the package name, just get the programmer to do so. So package P exposes a module called P.M and package Q exposes Q.M. All P's internal modules are called P.something, and similarly for Q. (We rely on some social mechanism for allocating new package names, as now.) Now of course you can import P.M and Q.M in a single module. That would be simple. It might be pretty inconvenient to say 'import Base.Data.List' rather than just 'import Data.List'. But nothing forces you to do this -- and indeed we don't do it for the current 'base' package. The point is that it's an easy way for a package author to ensure their package won't conflict with others. If they choose not to avail themselves of it, it's more likely that their package will be unusable because of accidental name conflicts. Bottom line: the current story is pretty defensible. I'm not sure that keeping names unique by implicitly using package-ids is worth the bother. Simon

2 1

RE: Binary IO
by Simon Marlow 22 Apr '05

22 Apr '05

On 22 April 2005 11:35, Benjamin Franksen wrote: > On Friday 22 April 2005 10:38, Simon Marlow wrote: >> Lifting this constraint in GHC is a bit tricky because we currenly >> use the OS's text<->binary translation to do I/O (doesn't matter on >> Unix, right now). > > So what? If the handle is in text mode, you won't get the exact bytes > as they are in the file, when calling hGetWord8 (at least on some > systems). But that is exactly what I would expect if the handle is in > text mode. If I need the exact binary representation of a text file, I > have to use binary, of course. So if you open a file in text mode and read it with hGetWord8, what bytes do you expect to get, exactly? Perhaps each byte of the UCS-4 representation? Big endian or little endian order? You could specify some meaning, but I don't think it would be a useful interface. And if you can't specify it precisely, it shouldn't be there. >> I presume that hPutWord8 (fromIntegral (ord '\n')) should not flush a >> line-buffered Handle, > > I can't see a reason not to flush it. Could you explain why you think > it should not? What have terminal settings to do with file handle > modes?? Well, I have the notion that EOL is a text concept, so doesn't belong in binary I/O. On the other hand, it's easy enough to specify that writing the byte 0xa to a line-buffered binary handle has the effect of flushing it, just as we say that writing the character '\n' to a line-buffered text handle triggers a flush. So I'm happy to leave things as they are. Cheers, Simon

3 2

Re: [Haskell] haddock -cpp ? Cabal support for haddock ?
by Isaac Jones 22 Apr '05

22 Apr '05

(CC'ing libraries) (snip) >> > I'd certainly welcome Cabal support for other haddock features as >> > well (--source, --read-interface). I am not sure where to put all >> > these arguments in the .cabal file. >> >> Cabal doesn't support these yet, though. Maybe in the future. > > Dear Isaac, > > for the next release, I think *every* external program used by Cabal > should get a xyz-options (free form) tag to give additional options. We > already have them for linker, c-compiler, and hs-compiler, but not yet > for preprocessors and doc generators (haddock). This is very easy to > implement, does no harm at all, and greatly increases cabal's > flexibility as a build tool. (BTW, I can send you a darcs patch if you > are too busy at the moment.) And in the other thread you said: >> > I made the necessary changes for hsc2hs-options in a few minutes You added hsc2hs-options to the package description? Cool. I'm happy to get a patch to add options fields to all the preprocessors and haddock and anything else we may have missed. There are basically 3 ways that people can customize their packages: - the .cabal file - the Setup script with UserHooks - flags to configure I was originally thinking of these extra flags as something to pass to configure, but actually putting them in the description file would be more consistent with what we have already... One trick, though, is to make sure that the parser test cases for cabal still run when you make the changes. It's all too common for someone to add a field and break the parser or pretty printer. The important thing is that when you parse it, pretty print it, and parse it again it comes out the same. Check out tests/ModuleTest.hs > (BTW, I can send you a darcs patch if you are too busy at the > moment.) I like patches whether I'm busy or not. It'll definitely get done faster ;) BTW, I'm tracking this feature request in the Debian bug tracking system for the haskell-cabal package. peace, isaac

2 1

RE: Binary IO
by Simon Marlow 22 Apr '05

22 Apr '05

On 22 April 2005 11:42, Henning Thielemann wrote: > On Fri, 22 Apr 2005, Simon Marlow wrote: > >> Does anyone have any *objections* to introducing >> >> System.IO.hGetWord8 :: Handle -> IO Word8 >> System.IO.hPutWord8 :: Word8 -> Handle -> IO Word8 > > Sorry, I didn't follow the thread ... what about > System.IO.hPutWord8 :: Handle -> Word8 -> IO () > that is, switched argument order and no result duh, sorry. Yes, that's the right type. > Will there be function names without 'h' prefix but qualification in > future, e.g. System.IO.Binary.Handle.putWord8 ? I think this is something for the next-generation I/O library. Cheers, Simon

1 0

RE: Cabal vs Haskell [sic]
by Simon Marlow 22 Apr '05

22 Apr '05

one minor correction: On 22 April 2005 11:08, Malcolm Wallace wrote: > "Needed to build" tells me which interfaces must be available at > compile-time, no more and no less. I found recently that ghc-pkg > (whose input format is essentially Cabal+extras) has a separate > "builds" field, you mean "depends", I think. And a comment on this: > As it happens, different compilers blur the boundary between interface > and implementation to a greater or lesser extent. To qualify this, I'd say that compilers differ in where they place the boundary between interface and implementation *of object code only*. The interface exposed by a source code module is perfectly well defined, by the language definition. I think it's important to make this distinction (I'm sure Malcolm is aware of it, but I just wanted to make sure it was expressed clearly). Build-depends uses the source-code notion of interface, not the object code. So, build-depends has the same meaning for all compilers. Again, this is an important point to bear in mind. Where the meaning differs, is when you want to replace the object code of an installed package with new object code. For example, upgrading a shared library. This might be a nice feature to have, but it requires us to think very carefully about the public interface of object code (and hence restrict some optimisations, in the case of GHC). Cheers, Simon

1 0