- Haskell-Cafe - Haskell.org

GHC 6.12 on OS X 10.5
by Conor McBride 08 May '10

08 May '10

Hi I thought I'd record my upgrade exerience (so far) in case anyone else finds it useful, and (more selfishly) in case anyone has some helpful advice. Summary of situation * I got 6.12 working. * It took a lot of fumbling around. * The eventual "fix" (renaming /opt/local/lib/libiconv.dylib) was a bit dodgy, but is holding for now. Long rambling narrative follows. Various features of GHC 6.12 made it very tempting to go for an earlier upgrade than I normally would. (SHE really needs GADT-style data families; everything I do needs deriving Traversable (deriving HalfZip would be nice too!).) Already there was a handy installer for Intel+Leopard macs (is anybody on the PPC+Leopard case? if not, I will be soon...). I thought "go for it". Installation, no problem. Compiling something: whoops, no mtl. OK, cabal install mtl: whoops, no acceptable cabal. Fair dos, if I'd read the warnings I'd have known that my cabal-install would be useless and tried to build the new one. QUESTION: Is the sensible upgrade path to build cabal-install 0.8 with your old GHC, before installing 6.12? Does this even work? At one point, I tried this and had some peculiar issues involving zlib... Anyhow, I've got 6.12 and I need cabal-install 0.8. Can I find it? Tricky, and http://haskell.org/cabal/ didn't help much. I'm very tolerant of busy people not quite getting round to it, and I can use google, so I find the darcs version and get cracking. This happens: ----------------------------------------------------------------------------- bash-3.2$ ./bootstrap.sh Checking installed packages for ghc-6.12.1... parsec-2.1.0.1 will be downloaded and installed. network-2.2.1.5 will be downloaded and installed. Cabal is already installed and the version is ok. mtl-1.1.0.2 will be downloaded and installed. HTTP-4000.0.8 will be downloaded and installed. zlib-0.5.2.0 will be downloaded and installed. Downloading parsec-2.1.0.1... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 15430 100 15430 0 0 12743 0 0:00:01 0:00:01 --:--:-- 19312 [1 of 1] Compiling Main ( Setup.hs, Setup.o ) Linking Setup ... Undefined symbols: "_iconv_close", referenced from: _hs_iconv_close in libHSbase-4.2.0.0.a(iconv.o) "_iconv", referenced from: _hs_iconv in libHSbase-4.2.0.0.a(iconv.o) "_iconv_open", referenced from: _hs_iconv_open in libHSbase-4.2.0.0.a(iconv.o) ld: symbol(s) not found collect2: ld returned 1 exit status Error during cabal-install bootstrap: Compiling the Setup script failed ----------------------------------------------------------------------------- At lthis point, I suspected a linker/dylib problem, but foolishly, I wanted the problem to be pretty much anything else, so I spent far too long looking the wrong way. I thought that if I could just get cabal-install working somehow, I'd be ok. I tried installing parsec with runhaskell Setup.hs {configure,build,install} but each of these commands appeared content to do precisely nothing: I found this perplexing. I tried reverting to 6.10.4 and compiling cabal-install. This made more progress, but died with some sort of zlib version snafu. (I'm sorry, I should have recorded the details but didn't.) The zlib-0.5.2.0 package did install successfully, but somehow the build for cabal-install itself went awry with an "even though this is the right version I can't find X" sort of a problem. I uninstalled 6.10.4, deleted my .cabal directory, nuked a few other relics showing up from what's probably a dumb choice of PATH setting. I had another go at 6.12, and this time I tried compiling a wee program of my own with no exciting package dependencies. Should have done that straight away, of course. Same iconv problem. This left no alternative. I found I had a /usr/lib/libiconv.dylib etc and an /opt/local/lib/libiconv.dylib etc, so I renamed the latter to opt/local/lib/moolibiconv.dylib, and suddenly, GHC became capable of linking stuff. The darcs version of cabal-install then built just fine; SHE rebuilt ok; Epigram needed a few extra LANGUAGE pragmas, but no trouble. I'm almost back where I was. I worry about this. (1) What have I broken by shifting the opt/local/lib copy of libiconv? (2) Why did that break things anyway? (3) How come I ended up with a broken library setup? (4) What else is broken? (What's worse than finding a maggot in your apple? Finding half a maggot in your apple.) I'm not at all a mac expert (I use a mac because I'm too stupid to maintain a linux installation -- the Nottingham grad students (all grown up now) told me they were fed up fixing my computer back in 04.) This dylib nonsense seems quite common. Is it MacPorts messing things up? Is there a more principled fix than the brutal thing I did? I wonder if it might be possible to build some sort of diagnostic tool to check paths and libraries for conflicts and omissions. Anyhow, sorry for the dreariness of this story, but it ends in an acceptable configuration, for the moment. I hope it can be of some small help. Happy hols Conor

10 19

Ideas for better network IO in Combinatorrent?
by Jesper Louis Andersen 08 May '10

08 May '10

Hi, Cafe! (This post got rather long, and in writing it, I probably answered some of my own questions. I'll still post it because it might be interesting to read for somebody). This post concerns the currently central problem in Combinatorrent. But the problem is interesting from another angle. If you want fast servers in Haskell, I am pretty sure you will need to crack this nut as well. Status is that my Erlang-client, which handles all IO by hand in Erlang uses much less CPU resources than the Combinatorrent client in Haskell. Especially under load. Productivity is usually around 70-75% so we spend some considerable time in the GC still. It is up from some 30%, so we are faring much better than earlier. Still, we use about 4 times as many CPU seconds per downloaded megabyte compared to Erlang. And it is not like the Erlang code is optimized: The parser loop is Erlang and not using the C drivers packeting facility. (Aside: For non-Erlangers -- In erlang you can set options on a TCP socket about how you want data to be delivered. There are several (C-hand-optimized protocol parsers for ASN.1, FastCGI, or even HTTP), but one of them designates that the TCP stream will receive data in the format of a 4 byte big endian integer header, the length, followed by the payload of that length. By setting the option {packet, 4} on the TCP link, you can "outsource" this packeting to the Erlang VM kernel in C and receive a message (in a process channel) whenever such a message arrives.) In Erlang I don't use it because it gives too erratic speed measurements. I need to know how fast I am communicating because it is used in the BitTorrent protocol to gauge which peers it is interesting to talk to. Rather, I run a handrolled parser loop, doing exactly the thing from above. Essentially the loop collects data from the socket in what is equivalent to a Lazy ByteString. When the LBS is full, it is converted to a Strict ByteString and handed on to the parser. While doing this measurements are done on the size of incoming data from the operating system. The parser is fairly naive, but it uses the Erlang-bit-syntax pattern matches to essentially decode the binary chunk with a pointer walking over it. Note that The largest packet type is 16 kilobytes + a_small_constant. Rather than reallocate the binary, the Erlang system just stores a Slice-triple (u, o, l), of the underlying byte array, u, the offset into that array, o and the length from the offset, l. When the slice disappears, the garbage collector can evict the binary. Binaries are ref-counted since they can not contain pointers and they are stored outside the process heap. This has to do with the Erlang VMs garbage collection strategy: Each Erlang process has its own GC and all message passing happens by copying. Since copying around large chunks of binary data is expensive, they are stored immutably and shared by all processes. It is fairly efficient. There is only one major copy going on when assembling the LBS into a Strict BS and that copy knows the exact size of the target allocation area, so it can just allocate the right size and then copy data into that area. Enough Erlang, what does Combinatorrent do? Combinatorrents receiver processes (There are one for each Peer we are communicating with) currently takes 46% of all allocation on their own. Why? Here is why. Our main loop in the receiver process looks like this (explanation below the source code, read it first): bs <- liftIO $ recv s 2048 when (B.length bs == 0) stopP loop s c (A.parse getMsg bs) where loop s c (A.Done r msg) = do liftIO . atomically $ writeTChan c (FromPeer (msg, fromIntegral $ msgSize msg)) loop s c (A.parse getMsg r) loop s c (prt@(A.Partial _)) = do bs <- liftIO $ recv s 4096 when (B.length bs == 0) stopP loop s c (A.feed prt bs) loop _ _ (A.Fail _ ctx err) = do warningP $ "Incorrect parse in receiver, context: " ++ show ctx ++ ", " ++ show err stopP [full code at: http://github.com/jlouis/combinatorrent/blob/master/src/Process/Peer/Receiv… ] The 'A' module you see is Bryan O'Sullivan's attoparsec package. This parser is incremental. Either it returns A.Done r msg, where msg is a decoded message and r is the rest of the input not parsed yet. Or, it returns a Partial parse where it demands more input to successfully parse. If you feed that continuation with more data, it happily continues on. The 'recv' you see is from Johan Tibell's network-bytestring package. It will block on the Socket s until data arrives. Then it will fill up to 4096 bytes into the strict bytestring bs. We use this to feed more data to attoparsec. One problem is that usually, we are so fast that the packet is only some 1448 bytes which make sense: the MTU of the ethernet wire is 1500 bytes and sizeable chunks are taken by TCP/IP and also the internet routers PPPoE/A. This means the bytestring is trimmed down most of the time, incurring a copy. BitTorrent connections are either dead-slow, only transferring a few bytes each second on average, to dead-fast, easily filling up the 4k window. I've toyed with the idea of an adapting receiver window size, but it is peanuts compared to the next problem it seems. The next problem is what attoparsec ends up doing. The relevant parser code is: getAPMsg :: Int -> Parser Message getAPMsg l = do c <- A.anyWord8 case c of 0 -> return Choke 1 -> return Unchoke ... 6 -> (Request <$> apW32be <*> (Block <$> apW32be <*> apW32be)) 7 -> (Piece <$> apW32be <*> apW32be <*> A.take (l - 9)) ... k -> fail $ "Illegal parse, code: " ++ show k [full code at: http://github.com/jlouis/combinatorrent/blob/master/src/Protocol/Wire.hs#L1… ] Where the attoparsec parser first reads of a character, c, designating the message type of the parse. Then it handles the message type by returning an ADT of the parse. The killer is A.take (l -9). If we look into the attoparsec source code, we see a call to its 'ensure' function. This function ensures we have enough data available for parsing. If not, it demands more input through A.Partial results. When it gets input it concatenates with '+++' which is really the 'append' function of Strict ByteStrings which are O(n) and destroys the allocation rate of the application :) My current plan for a solution is this: Read the socket and gather up chunks for a lazy bytestring. We know, due to the 32bit BE header how many bytes more to expect. When we have all of them, do the Strict BS conversion and hand that to attoparsec. It means no +++ call inside attoparsec. Furthermore I get a SBS.splitAt, yielding the slice-construction I am longing for. But are there any ideas out there for doing it even better than this? Conjure by David Himmelstrup removes the Lazy -> Strict conversion as well by preallocating the Strict ByteString and filling it directly, but I think I can live with the simpler solution for now. The only important thing currently is to get attoparsec happy, without feeding it sangria or beer :) In any event, there is food for thought on some possible extensions of network-bytestring I think. Another possibility is going back to Handle's as they have a buffer construction, but Handle's had even worse CPU resource usage if I remember correctly. I have not analyzed that problem to this depth however. I have also considered John Lato's iteratee's, but we are sorely lacking an iteratee 101 introduction. Can iteratees parse infinite streams for instance? -- J.

1 0

Shorthand method of enumerating a list a gotcha ... or is it just me?
by Gene A 08 May '10

08 May '10

The problem I see is that in both: Version: September 2006 of hugs, which is the one that is current for Ubuntu 9.10 release, and ghci 6.10.4, they both exhibit a {I think} strange behaviour, in regards to the shorthand way of calling out a list of enumerable values. I will explain the problem that I have run into with examples: Hugs> [3,7..22] [3,7,11,15,19] <- OK Hugs> map (* 1.0) [3,7,11,15,19] <- manual spec OK [3.0,7.0,11.0,15.0,19.0] Hugs> map (* 1.0) [3,7..22] <- same spec as first but !!! when mapped to with a (*1.0) to coerce them to reals: [3.0,7.0,11.0,15.0,19.0,23.0] <- went one outside of range spec. ======================================== Exactly the same behaviour from ghci 6.10.4 : Prelude> [3,7..22] [3,7,11,15,19] Prelude> map (* 1.0) [3,7..22] <- using a range [3.0,7.0,11.0,15.0,19.0,23.0] <- it screws up {at least it is not a "feature' to me} Prelude> map (* 1.0) [3,7,11,15,19] <- spelled out it acts "right". [3.0,7.0,11.0,15.0,19.0] This seems like a possible bug? or at least a sure fire trap waiting to be sprung ... one of those nasties that could really create havoc if someone is not aware of this behaviour and buries a function that include something that unwittingly coerces from an Integral to a Realfrac or Fractional. Is this a well known thing to watch out for.. or is it something that can be worked around, other then having to enumerate every value in a list rather then use the handiness of the range notation as shorthand? cheers, gene

5 4

ANN: test-framework 0.3.0
by Max Bolingbroke 08 May '10

08 May '10

I'm pleased to announce the release of version 0.3 of test-framework (http://hackage.haskell.org/package/test-framework-0.3.0) This package provides a nice test runner for HUnit, QuickCheck 1 and QuickCheck 2 tests. To see a detailed example of the runner in action, please check out our website at http://batterseapower.github.com/test-framework The highlights of this release are: * Thanks to the contributions of Rogan Creswick at Galois, you can now get JUnit XML output from the test runner using the --jxml=FILENAME option * There is a new command line option (--plain), which tells the test runner to avoid using any ANSI features - this can be handy if you are (for example) viewing test output in Emacs Cheers, Max

3 3

Re: [Haskell-cafe] Haskell and the Software design process
by Henning Thielemann 08 May '10

08 May '10

On Mon, 3 May 2010, Gregory Collins wrote: > Don Stewart <dons(a)galois.com> writes: > >> Some key points: >> >> * Avoid partial functions > > As an important corollary to this one I would add: "never throw > exceptions from pure code". They often leak out from "catch" blocks and > ruin your day. It's not possible to throw exceptions from pure code. You can only call 'error' and that's another name for 'undefined', i.e. you have a partial (non-total ?) function.

5 7

Haskell interface to Frama-C
by Tom Hawkins 08 May '10

08 May '10

I just started using Frama-C [1] for analyzing some of our embedded C programs. Pretty awesome suite of tools. Especially its ability to describe and verify function contracts with ACSL [2]. The tool suite is primarily build with OCaml. Has anyone considered building a Haskell interface to Frama-C? I lack the expertise and time for such a project. But it sure would be nice to be able to write verification plug-ins with Haskell. -Tom [1] http://frama-c.com/ [2] http://en.wikipedia.org/wiki/ANSI/ISO_C_Specification_Language

1 0

ANNOUNCE: network-protocol-xmpp 0.3
by John Millikin 07 May '10

07 May '10

XMPP[1] is a protocol for asynchronous communication via long-lived XML documents. It is most famous for underlying the Jabber and Wave protocols, but is also used for a variety of cases where security and extensibility are useful. My library, network-protocol-xmpp[2], is an implementation of most of RFC 3920 and a bit of RFC 3921. It supports opening client-to-server and component-to-server sessions, which is useful for implementing XMPP-based clients. This library's interface is very simple: clients may start a session with 'runClient' or 'runComponent', send and receive stanzas, and resume stored sessions. Later, I intend to add additional modules to support features such as MUC or file transmission, but for now I'd like to make sure the core library works. There's an example XMPP client, echo.hs[3], in the Darcs repository. Server-to-server sessions and accepting connection requests are not yet supported, so it's not usable as the foundation of an XMPP server yet. However, given that XMPP prefers to place features on the servers rather than clients, it's likely that writing an XMPP server in Haskell would take a *lot* of effort anyway. You're probably better off just using ejabberd or OpenFire. My thanks go to Stephan Maka, who let me know that this library actually has users, and who contributed most of the Component connection implementation. ---------- Additionally, I'm trying an experiment intended to break the gridlock of Haskell XML libraries. Currently, there are multiple options for XML handling (HXT, HaXml, libxml, tagsoup, xml, xml-basic) but each defines their own data types. This makes it very difficult to mix-and-match -- I can't easily use one library's parser with another's tree manipulation combinators. Simple use cases, such as using HXT for serializing, drag in dozens of dependencies on libraries like HTTP and QuickCheck. A library's poor data type definitions (such as HaXml's lack of support for namespaces) means that an entire set of potentially useful operations is unusable. And a library's choice propagates! In n-p-xmpp 0.2, which used HXT, clients were forced to figure out arrows before they could perform even rudimentary inspection of received stanzas. To fix this, I've created the xml-types[4] package. Its only dependency is the "text" library, and it defines full types for most XML structures[5]. My hope is that other libraries will replace their own types with these, so that (for example) I can parse events with libxml, manipulate them with HXT's arrow API, and write them back out using a custom serialiser without having to jump through type conversion hoops. It might take some time before any major XML libraries are transitioned to these types, but maybe in a year or two there'll be enough to be useful. [1] http://en.wikipedia.org/wiki/Extensible_Messaging_and_Presence_Protocol [2] http://hackage.haskell.org/package/network-protocol-xmpp [3] http://patch-tag.com/r/jmillikin/xmpp/snapshot/current/content/pretty/examp… [4] http://hackage.haskell.org/package/xml-types [5] I haven't included types for inner entity declarations in DTDs, out of laziness, because does anybody actually *use* those?

2 2

{-# LANGUAGE DeriveApplicative #-} ?
by Pavel Perikov 07 May '10

07 May '10

Hi, list!. Now in 6.12.1 we have DeriveFunctor, DeriveFoldable and DeriveTraversable. This greatly simplifies the "reuse structure" style of programming. Some structure (not just _data_ structure) got captured in ADT and can be reused for various purposes. Wouldn't it be nice to have the ability to derive Applicative as well? It shouldn't be more difficult than deriving Functor but will provide exciting possibilities. Just think about liftA2. Please note that deriving Monad is quite different story and seem generally unfeasible. Suggestions welcome. P.

3 3

accents
by Dupont Corentin 07 May '10

07 May '10

Hello, i have a list of french words with accents. How could i handle them? If i load them with ghci i get: > a <- readFile "list.txt" > head $ lines a "abn\233gation" putStrLn displays a strange character for the "é". Cheers, Corentin

3 4

Type system game (was: Nomic game in Haskell)
by Dupont Corentin 07 May '10

07 May '10

Hello, I think your type should be: type Board a b c d e f g h i = Either (Three a b c) (Either (Three d e f) (Either (Three g h i) (Either (Three a d g) (Either (Three b e h) (Either (Three c f i) (Either (Three a e i) (Three c e g))))))) as far as i can understand, it seems thats it's mandatory for Player 1 to play in a,b,c to win? So victory should go to player 2? Cheers, Corentin On 4/16/10, Dan Piponi <dpiponi(a)gmail.com> wrote: > On Thu, Apr 15, 2010 at 4:58 PM, Ashley Yakeley <ashley(a)semantic.org> wrote: > >> type Board a b c d e f g h i = >> Either (Three a b c) >> (Either (Three d e f) >> (Either (Three g h i) >> (Either (Three a d g) >> (Either (Three b e h) >> (Either (Three c f i) >> (Either (Three a e i) >> (Either (Three c e g) >> ))))))) > > In the service of readability we could also define: > > data X = X > data O > > Though the victory conditions aren't precisely the usual ones. > -- > Dan > _______________________________________________ > Haskell-Cafe mailing list > Haskell-Cafe(a)haskell.org > http://www.haskell.org/mailman/listinfo/haskell-cafe >

1 0