- Libraries - Haskell.org

RE: Environment variable does not work (especially DARCS_EDITOR)
by Simon Marlow 25 Apr '05

25 Apr '05

On 16 April 2005 20:41, Benedikt Schmidt wrote: > [ crossposted to libraries(a)haskell.org ] > Tommy Pettersson <ptp(a)lysator.liu.se> writes: > >> A likely source of the problem can be that pico aborts if its >> stdin or stdout isn't a tty. I have that problem with nvi. >> If you run darcs 1.0.2 you can try and set DARCS_EDITOR to >> "pico </dev/tty >/dev/tty". If you run a later build you >> may have to write a wrapper script like >> >> #!/bin/sh >> pico </dev/tty >/dev/tty >> >> since the to-be-next stable darcs not always perform the >> redirection on the system call any longer. >> >> Or, if this turns out to be the problem, and if you know how >> to make darcs do this redirection of stdin/out in a portable >> way, fix it in the darcs source. > > The problem seems to be related to rawSystem and the ghc RTS (at > least GHC 6.4). GHC sets stdin/stdout/stderr to nonblocking mode > on startup and nvi doesn't like that. There is a comment in > ghc/rts/RtsUtils.c that resetNonBlockingFd is called before > exiting because leaving standard file descriptors in nonblocking > mode confuses some shells. > > module Main where > import System.Cmd > main = rawSystem "nvi" [] > > The test program works after applying the following patch to GHC > that disables nonblocking mode after the fork and before the exec. > > diff -u -r1.7 runProcess.c > --- libraries/base/cbits/runProcess.c 27 Mar 2005 13:41:19 -0000 > 1.7 +++ libraries/base/cbits/runProcess.c 16 Apr 2005 19:08:52 -0000 > @@ -62,6 +62,10 @@ > dfl.sa_handler = (void *)quithandler; > (void)sigaction(SIGQUIT, &dfl, NULL); > } > + > + resetNonBlockingFd(fdStdInput); > + resetNonBlockingFd(fdStdOutput); > + resetNonBlockingFd(fdStdError); > > dup2 (fdStdInput, STDIN_FILENO); > dup2 (fdStdOutput, STDOUT_FILENO); > > I'm not sure how to handle that in darcs without > copying/reimplementing the parts of System.Cmd we need. If we did this, we would also need to re-enable non-blocking mode after the rawSystem completes. Also, this doesn't work well if runProcess is used and the Haskell program continues to run while the subprocess is executing, because the O_NONBLOCK flag is shared between parent and child. So I suggest doing this for System.Cmd.{system,rawSystem} only. Cheers, Simon

3 2

RE: Nonblocking IO (was: Binary IO)
by Simon Marlow 25 Apr '05

25 Apr '05

On 25 April 2005 10:51, Peter Simons wrote: > Simon Marlow writes: > > >> hPutBufNonBlocking :: Handle -> Ptr a -> Int -> IO Int > >> hGetBufNonBlocking :: Handle -> Ptr a -> Int -> IO Int > > > I believe they don't have proper non-blocking behaviour > > on Windows, though. > > What exactly would happen when I'd try to use those on > Windows? Would they block until they had read/written > exactly 'Int' bytes? Or is there some other, more subtle > difference? I think they behave like the blocking variants on Windows (but the code is quite intricate.. I'm not 100% sure). This should really be fixed. Cheers, Simon

1 0

RE: Binary IO
by Simon Marlow 25 Apr '05

25 Apr '05

On 22 April 2005 12:05, Peter Simons wrote: > Simon Marlow writes: > > > Does anyone have any *objections* to introducing > > > System.IO.hGetWord8 :: Handle -> IO Word8 > > System.IO.hPutWord8 :: Word8 -> Handle -> IO Word8 > > I don't mind having these functions, but to be honest I > doubt that they would be useful for real life applications. > Reading a byte at a time is a performance nightmare, > buffering or not. IMHO, the much better API is this one: > > hPutBufNonBlocking :: Handle -> Ptr a -> Int -> IO Int > hGetBufNonBlocking :: Handle -> Ptr a -> Int -> IO Int You have these in GHC.IO at the moment. I beieve they don't have proper non-blocking behaviour on Windows, though. These are also rather more difficult to implement than the single-byte versions, so I can't see nhc98 or Hugs implementing them any time soon. Cheers, Simon

6 9

Announce: Haskell Cryptographic Library 2.0.1
by Dominic Steinitz 24 Apr '05

24 Apr '05

I would like to announce the release of a new version of the Haskell Cryptographic Library. It now uses darcs and cabal and provides the modules listed below. The most significant change is in the way ASN.1 is handled. This has been made generic enough to handle PKCS#8 private keys and X.509 certificates. For further information see http://www.haskell.org/crypto. Dominic. Codec.ASN1 Codec.ASN1.BER Codec.ASN1.PKCS8 Codec.ASN1.TLV Codec.ASN1.X509 Codec.Binary.Base64 Codec.Encryption.AES Codec.Encryption.Blowfish Codec.Encryption.DES Codec.Encryption.Modes Codec.Encryption.Padding Codec.Encryption.RSA Codec.Encryption.RSA.EMEOAEP Codec.Encryption.RSA.MGF Codec.Encryption.RSA.NumberTheory Codec.Utils Data.Digest.MD5 Data.Digest.SHA1 Data.LargeWord

1 0

Re: Binary IO
by Dominic Steinitz 24 Apr '05

24 Apr '05

On Friday 22 Apr 2005 9:48 pm, libraries-request(a)haskell.org wrote: > Message: 1 > Date: Fri, 22 Apr 2005 08:30:27 -0700 > From: "David Brown" <haskell(a)davidb.org> > Subject: Re: Binary IO > To: "Peter Simons" <simons(a)cryp.to>, libraries(a)haskell.org > Message-ID: <op.spm301mpl6e4hk(a)a64.davidb.org> > Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-1 > > On Fri, 22 Apr 2005 08:09:27 -0700, Peter Simons <simons(a)cryp.to> wrote: > >> And hGet/PutWord8 are fast enough for most situations. > > I haven't been following the discussion closely but I am just about to make a big release of the crypto library and I have been using "Hal Daume III's NewBinary now ghc6ized, cabalized, debianize, darcsized." available at http://www.n-heptane.com/nhlab/repos/NewBinary. Specifically I use: import qualified NewBinary.Binary as B (BinHandle, getBits, getByte) I hadn't see anything about getBits but would make a plea for its inclusion in any Binary IO package. > > I seem to run into this kind of thing a lot with Haskell. Dominic's ASN > library is useful to me, however I'm finding I'll probably just have to > use it as a template for new code. It works fine for what designed for, > parsing keys and such, but I'm looking to use it to represent gigabytes of > data. Processing data like that one Word8 at a time isn't practical. For information, the most significant change in the crypto library (as well darcsizing, cabalizing and restructuring it to fit the agreement reached on this list - http://www.haskell.org//pipermail/libraries/2005-January/002856.html) is the ASN.1 handling. If that is going to cause anyone problems, please let me know I will see what can be done. Dominic.

1 0

RE: Cabal Issues
by Simon Marlow 23 Apr '05

23 Apr '05

On 23 April 2005 21:03, Benjamin Franksen wrote: > On Saturday 16 April 2005 03:22, Isaac Jones wrote: >> Ashley Yakeley <ashley(a)semantic.org> writes: >>> I've just started using Cabal, for my time library implementation >>> (<http://semantic.org/TimeLib/>). I have some issues: >>> >>> 1. If I do "runghc Setup.hs haddock" and then "runghc Setup.hs >>> install", the documentation doesn't appear to get installed >>> anywhere. >> >> Where do folks think that the docs should go? > > This is highly system dependent and even linux distributions don't > agree here. So: > >> Or should we add a >> haddock-prefix flag to configure? > > Yes. For unix like systems, a reasonable default (if no option is > specified) could be /usr/local/share/doc. Sure - this is only the default, we expect RPMs and other package systems to override it to match the local policy. A subdirectory of /usr/local/share/doc is the right thing on Unix systems. Cheers, Simon

1 0

RE: GC trouble [was Re: Binary IO]
by Simon Marlow 23 Apr '05

23 Apr '05

On 23 April 2005 20:12, Jan-Willem Maessen wrote: > Hmm; this sounds like a lot of full-heap collections, which is exactly > what generational GC is trying to avoid. A very large old generation > (like, say, 500+Mb) might help a lot in this instance; I have no idea > how GHC decides generation sizes. It might also help to set a very > large allocation area to reduce promotion rate to the second > generation, and give the gobs of transient data some time to die---or, > similarly, to increase the number of generations to increase the time > it takes things to get to the old generation. Fundamentally, though, > when you run really close to your memory limits GC tends to be > unhappy. For a program with this much data, GHC's default allocation area size of 256k is pretty small. In fact, it's pretty small full stop - I recently bumped it to 512k. I suggest trying -A1m at least. I'd be interested to know whether you get better results by tweaking any of the other GC settings: more generatsions and more steps in particular. Cheers, Simon

1 0

RE: Binary IO
by Simon Marlow 23 Apr '05

23 Apr '05

On 21 April 2005 18:36, Aaron Denney wrote: > On 2005-04-21, Simon Marlow <simonmar(a)microsoft.com> wrote: >> On 21 April 2005 11:16, John Meacham wrote: >> >>> but a useful stopgap, because their use would guarentee your code >>> wouldn't suddenly stop working when ghc changes its IO system. >> >> That isn't going to happen, but I take your point. > > You mean we're not going to fix it? Not at all - just that we don't plan to break existing code if/when a replacement for the current I/O library is introduced. Sorry for the misunderstanding. Does anyone have any *objections* to introducing System.IO.hGetWord8 :: Handle -> IO Word8 System.IO.hPutWord8 :: Word8 -> Handle -> IO Word8 (I prefer these names, please speak up if you prefer hGetByte/hPutByte instead, or something else like hGetOctet/hPutOctet). One constraint would be that the Handle has to be in Binary mode. Lifting this constraint in GHC is a bit tricky because we currenly use the OS's text<->binary translation to do I/O (doesn't matter on Unix, right now). I presume that hPutWord8 (fromIntegral (ord '\n')) should not flush a line-buffered Handle, which is another slight complication. Perhaps line-buffering should be the same as block-buffering on binary Handles. Cheers, Simon

9 10

Cabal Issues
by Ashley Yakeley 23 Apr '05

23 Apr '05

I've just started using Cabal, for my time library implementation (<http://semantic.org/TimeLib/>). I have some issues: 1. If I do "runghc Setup.hs haddock" and then "runghc Setup.hs install", the documentation doesn't appear to get installed anywhere. 2. Likewise, "runghc Setup.hs clean" doesn't delete the documentation. I had expected it to delete the whole dist directory. 3. There ought to be a mechanism for including tests. Something like "runghc Setup.hs check". Right now I have two tests that are executables that are run and the output compared to a reference output file (and another test that must be verified manually). But perhaps this is difficult. -- Ashley Yakeley, Seattle WA

3 3

RE: Binary IO
by Simon Marlow 23 Apr '05

23 Apr '05

On 22 April 2005 21:56, Glynn Clements wrote: > David Brown wrote: > >>>> And hGet/PutWord8 are fast enough for most situations. >>> >>> Are you certain? Which interesting applications do you know >>> that read and write one byte at a time? I can't speak for >>> "most" situations, but in those situations where I needed >>> binary I/O this API would have been impracticable. >> >> I'm going to chime in agreement with the disagreement. >> >> I have one application which uses single character binary I/O. >> Because it works on small files, it borders on usefuleness. As time >> goes on, the size of these inputs grows, and now the tool has become >> nearly worthless. >> >> I seem to run into this kind of thing a lot with Haskell. Dominic's >> ASN library is useful to me, however I'm finding I'll probably just >> have to use it as a template for new code. It works fine for what >> designed for, parsing keys and such, but I'm looking to use it to >> represent gigabytes of data. Processing data like that one Word8 at >> a time isn't practical. > > Personally, I doubt that Haskell will ever be practical for processing > very large amounts of data (e.g. larger than your system's RAM). I hope that's not true - whatever techniques you use in other languages for handling large amounts of data should translate straightforwardly into Haskell. Cheers, SImon

4 5