February 2009 - Haskell-Cafe

Re[2]: Ready for testing: Unicode support for Handle I/O
by Bulat Ziganshin 19 Feb '09

19 Feb '09

Hello Khudyakov, Thursday, February 19, 2009, 4:06:06 PM, you wrote: >> i think that FilePath=String is ok, we just need to use utf8 string >> encoding on unix/macs and utf-16 encoding with *W functions on >> Windows. i have implemented such support inside my own application, and >> would be happy to mentor appropriate gsoc project > It won't _always_ work. current solution works never :) > And there are still 8-bit locales lurking somewhere in bush. UTF8 isn't > appropriate for them at all. i think that SetFilenameEncoding whould solve this, among with access to lower-level openFileUsingCStringPath -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

1 0

Ready for testing: Unicode support for Handle I/O
by Simon Marlow 19 Feb '09

19 Feb '09

I've been working on adding proper Unicode support to Handle I/O in GHC, and I finally have something that's ready for testing. I've put a patchset here: http://www.haskell.org/~simonmar/base-unicode.tar.gz That is a set of patches against a GHC repo tree: unpack the tarball, and say 'sh apply /path/to/ghc/repo' to apply all the patches. Then clean your tree and build it from scratch (or if you're using the new GHC build system, just say 'make' ;-). It should validate, bar one or two minor failures. Oh, it doesn't work on Windows yet. That's the major thing left to do. If anyone else felt like tackling this I'd be delighted: all you have to do is implement a Win32 equivalent of the module GHC.IO.Encoding.Iconv (see below), everything else should work unchanged. Depending on whether any further changes are required, I may amend-record some of these patches, so treat them as temporary patches for testing only. Below is what will be the patch description in the patch for libraries/base. Comments/discussion please! Cheers, Simon Unicode-aware Handles ~~~~~~~~~~~~~~~~~~~~~ This is a significant restructuring of the Handle implementation with the primary goal of supporting Unicode character encodings. The only change to the existing behaviour is that by default, text IO is done in the prevailing encoding of the system. Handles created by openBinaryFile use the Latin-1 encoding, as do Handles placed in binary mode using hSetBinaryMode. We provide a way to change the encoding for an existing Handle: hSetEncoding :: Handle -> TextEncoding -> IO () and various encodings: latin1, utf8, utf16, utf16le, utf16be, utf32, utf32le, utf32be, localeEncoding, and a way to lookup other encodings: mkTextEncoding :: String -> IO TextEncoding (it's system-dependent whether the requested encoding will be available). Currently hSetEncoding is availble from GHC.IO.Handle, and the encodings are available from GHC.IO.Encoding. We may want to export these from somewhere more permanent; that's something for a library proposal. Thanks to suggestions from Duncan Coutts, it's possible to call hSetEncoding even on buffered read Handles, and the right thing happens. So we can read from text streams that include multiple encodings, such as an HTTP response or email message, without having to turn buffering off (though there is a penalty for switching encodings on a buffered Handle, as the IO system has to do some re-decoding to figure out where it should start reading from again). If there is a decoding error, it is reported when an attempt is made to read the offending character from the Handle, as you would expect. Performance is about 30% slower on "hGetContents >>= putStr" than before. I've profiled it, and about 25% of this is in doing the actual encoding/decoding, the rest is accounted for by the fact that we're shuffling around 32-bit chars rather than bytes in the Handle buffer, so there's not much we can do to improve this. IO library restructuring ~~~~~~~~~~~~~~~~~~~~~~~~ The major change here is that the implementation of the Handle operations is separated from the underlying IO device, using type classes. File descriptors are just one IO provider; I have also implemented memory-mapped files (good for random-access read/write) and a Handle that pipes output to a Chan (useful for testing code that writes to a Handle). New kinds of Handle can be implemented outside the base package, for instance someone could write bytestringToHandle. A Handle is made using mkFileHandle: -- | makes a new 'Handle' mkFileHandle :: (IODevice dev, BufferedIO dev, Typeable dev) => dev -- ^ the underlying IO device, which must support -- 'IODevice', 'BufferedIO' and 'Typeable' -> FilePath -- ^ a string describing the 'Handle', e.g. the file -- path for a file. Used in error messages. -> IOMode -- ^ The mode in which the 'Handle' is to be used -> Maybe TextEncoding -- ^ text encoding to use, if any -> IO Handle This also means that someone can write a completely new IO implementation on Windows based on native Win32 HANDLEs, and distribute it as a separate package (I really hope somebody does this!). This restructuring isn't as radical as previous designs. I haven't made any attempt to make a separate binary I/O layer, for example (although hGetBuf/hPutBuf do bypass the text encoding). The main goal here was to get Unicode support in, and to allow others to experiment with making new kinds of Handle. We could split up the layers further later. API changes and Module structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NB. GHC.IOBase and GHC.Handle are now DEPRECATED (they are still present, but are just re-exporting things from other modules now). For 6.12 we'll want to bump base to version 5 and add a base4-compat. For now I'm using #if __GLASGOW_HASKEL__ >= 611 to avoid deprecated warnings. I split modules into smaller parts in many places. For example, we now have GHC.IORef, GHC.MVar and GHC.IOArray containing the implementations of IORef, MVar and IOArray respectively. This was necessary for untangling dependencies, but it also makes things easier to follow. The new module structurue for the IO-relatied parts of the base package is: GHC.IO Implementation of the IO monad; unsafe*; throw/catch GHC.IO.IOMode The IOMode type GHC.IO.Buffer Buffers and operations on them GHC.IO.Device The IODevice and RawIO classes. GHC.IO.BufferedIO The BufferedIO class. GHC.IO.FD The FD type, with instances of IODevice, RawIO and BufferedIO. GHC.IO.Exception IO-related Exceptions GHC.IO.Encoding The TextEncoding type; built-in TextEncodings; mkTextEncoding GHC.IO.Encoding.Types GHC.IO.Encoding.Iconv Implementation internals for GHC.IO.Encoding GHC.IO.Handle The main API for GHC's Handle implementation, provides all the Handle operations + mkFileHandle + hSetEncoding. GHC.IO.Handle.Types GHC.IO.Handle.Internals GHC.IO.Handle.Text Implementation of Handles and operations. GHC.IO.Handle.FD Parts of the Handle API implemented by file-descriptors: openFile, stdin, stdout, stderr, fdToHandle etc.

7 15

Hoogle and Network.Socket
by Felix Martini 19 Feb '09

19 Feb '09

The Network.Socket module works fine on Windows. The original Winsock implementation was based on the Berkeley sockets api.

1 0

Re: [Haskell-cafe] How to create an online poll
by gregg reynolds 19 Feb '09

19 Feb '09

I.e. war, plague.famine. etc. gregg reynolds <dev(a)mobileink.com> wrote: >But what about the side effects? > >Rick R <rick.richardson(a)gmail.com> wrote: > >>I'm sure Premier Election Solutions (formerly Diebold) can provide us with >>an online voting solution. >>Their value-add services allows us to set the outcome beforehand, so, in >>effect, the the voting process will be determinate. Which is certainly of >>interest to Haskell coders. >> >> >>On Wed, Feb 18, 2009 at 4:05 PM, Max Rabkin <max.rabkin(a)gmail.com> wrote: >> >>> On Wed, Feb 18, 2009 at 10:40 PM, Anton van Straaten >>> <anton(a)appsolutions.com> wrote: >>> > There's also the Condorcet Internet Voting Service: >>> > >>> > http://www.cs.cornell.edu/andru/civs.html >>> >>> This looks like exactly what we need! Any objections? >>> >>> --Max >>> _______________________________________________ >>> Haskell-Cafe mailing list >>> Haskell-Cafe(a)haskell.org >>> http://www.haskell.org/mailman/listinfo/haskell-cafe >>> >> >> >> >>-- >>We can't solve problems by using the same kind of thinking we used when we >>created them. >> - A. Einstein >>_______________________________________________ >>Haskell-Cafe mailing list >>Haskell-Cafe(a)haskell.org >>http://www.haskell.org/mailman/listinfo/haskell-cafe

6 6

Re: [Haskell-cafe] Haskell.org GSoC
by Daniel Kraft 19 Feb '09

19 Feb '09

Hi, sylvain wrote: > In my humble opinion, Haskell presently fall short of its promises > because it does not embed theorem proving. Quickcheck-like testing is > truly great, but can not replace proofs to produce bug free software. > > With use of equational reasoning + induction, one can already prove a > huge amount of useful properties of an Haskell program [1]. this sounds like a really interesting project to me... Especially as I'm doing maths instead of CS as main subject (and it is simply inherently interesting :D). > theorem : ( xs :: [a], ys :: [a], f :: a -> b) => > length (map f (xs ++ ys )) = length xs + length yx > proof > length (map f (xs ++ ys )) = > length (xs ++ ys) = {- use length++ -} > length xs ++ length ys That's of course nice, but I'm at the moment not yet convinced something like this could be more or less easily implemented also for larger programs; and, I don't see from your example how the real implementation of the program should play in. Do you expect that Haskell figures out from the implementation that (map f) does not alter a lists length? Or should this be an already available theorem within the Prelude? I guess it should be the former, as otherwise the whole proofing seems to be just documentation, without real connection to the Haskell code. But in this case, I wonder whether something like that can be successfully done on more sophisticated code, and especially done as part of a GSoC project... But I guess with a competent mentor and clearly defined goals it should be possible. But all in all, as stated above, this could be really interesting :) Thanks for this suggestion and your ideas! Daniel -- Done: Arc-Bar-Cav-Ran-Rog-Sam-Tou-Val-Wiz To go: Hea-Kni-Mon-Pri

1 0

Memory
by Jeff Douglas 19 Feb '09

19 Feb '09

Hello All, The kind people at #haskell suggested I come to haskell-cafe for questions about haskell performance issues. I'm new to haskell, and I'm having a hard time understanding how to deal with memory leaks. I've been playing with some network server examples and I noticed with each new connection, the memory footprint increases by about 7k However, the leaks don't seem to have anything to do with the networking code. Actually I get a huge leak just from using using 'forever'. > import Control.Monad > import System.IO > > main = forever $ putStrLn "hi" When I run it for a few seconds with profiling... > total time = 0.36 secs (18 ticks @ 20 ms) > total alloc = 54,423,396 bytes (excludes profiling overheads) Can this be right?

4 5

Returned mail: see transcript for details
by malcolm.wallace＠cs.york.ac.uk 18 Feb '09

18 Feb '09

Message could not be delivered

1 0

Re: permuting a list
by Okasaki, C. DR EECS 18 Feb '09

18 Feb '09

The discussion of randomly permuting a list comes up every few years. Here's what I wrote last time (2005): "Clearly, you can do a perfect shuffle in O(N) time using mutable arrays, using the standard imperative algorithm. You can also do it in O(N) expected time using *immutable* arrays, using Haskell's bulk array operations. 1. Start with a list of length N. If N < 2, stop. 2. Pair each element with a random index in the range [1,2N] (or [0,2N-1]). 3. Use "accum" to create an array of size 2N, where slot I contains a list of all the elements paired with I. 4. Map the shuffle function recursively across the array to re-shuffle any slots that got more than one element. 5. Append all the slots together into the result list. If you leave out step 4, you get an algorithm that runs in O(N) worst-case time, rather than expected time, but you no longer have a perfect shuffle (although it might be good enough). Upping the size of the array from 2N to 3N or 4N will help reduce collisions, but will not entirely eliminate them. With step 4, you can imagine pathological cases where everything gets put in the same slot, but with a reasonable random number generator, it is vanishingly unlikely that you will have enough collisions to drive the cost higher than linear. Again, upping the size of the array from 2N to 3N or 4N makes this even more unlikely." Now, you can consider it cheating to say this is O(N) expected time because it uses O(N log N) random bits. Fair enough. It is O(N) to the same extent that the ordinary imperative algorithm is O(N), albeit expected rather than worst case. -- Chris

2 1

Re: [Haskell-cafe] How to create an online poll
by gregg reynolds 18 Feb '09

18 Feb '09

Perhaps you are a subversive functor, hmm? We have monadic ways of conrolling you, fiend! Tellus who your natural tansformations are, or else! Andrew Wagner <wagner.andrew(a)gmail.com> wrote: >Help! Help! I'm being suppressed!! > >On Feb 18, 2009, at 5:05 PM, Rick R <rick.richardson(a)gmail.com> wrote: > >> I can't comment on any quantitative side effects, but the some >> intangible side effects include Distrustful Populace and >> MaliciousDissenters. However, if ghc is run with the - >> XUnscrupulousPolitics flag, those can be suppressed. >> >> On Wed, Feb 18, 2009 at 4:33 PM, gregg reynolds <dev(a)mobileink.com> >> wrote: >> I.e. war, plague.famine. etc. >> >> gregg reynolds <dev(a)mobileink.com> wrote: >> >> >But what about the side effects? >> > >> >Rick R <rick.richardson(a)gmail.com> wrote: >> > >> >>I'm sure Premier Election Solutions (formerly Diebold) can provide >> us with >> >>an online voting solution. >> >>Their value-add services allows us to set the outcome beforehand, >> so, in >> >>effect, the the voting process will be determinate. Which is >> certainly of >> >>interest to Haskell coders. >> >> >> >> >> >>On Wed, Feb 18, 2009 at 4:05 PM, Max Rabkin <max.rabkin(a)gmail.com> >> wrote: >> >> >> >>> On Wed, Feb 18, 2009 at 10:40 PM, Anton van Straaten >> >>> <anton(a)appsolutions.com> wrote: >> >>> > There's also the Condorcet Internet Voting Service: >> >>> > >> >>> > http://www.cs.cornell.edu/andru/civs.html >> >>> >> >>> This looks like exactly what we need! Any objections? >> >>> >> >>> --Max >> >>> _______________________________________________ >> >>> Haskell-Cafe mailing list >> >>> Haskell-Cafe(a)haskell.org >> >>> http://www.haskell.org/mailman/listinfo/haskell-cafe >> >>> >> >> >> >> >> >> >> >>-- >> >>We can't solve problems by using the same kind of thinking we used >> when we >> >>created them. >> >> - A. Einstein >> >>_______________________________________________ >> >>Haskell-Cafe mailing list >> >>Haskell-Cafe(a)haskell.org >> >>http://www.haskell.org/mailman/listinfo/haskell-cafe >> >> >> >> -- >> We can't solve problems by using the same kind of thinking we used >> when we created them. >> - A. Einstein >> _______________________________________________ >> Haskell-Cafe mailing list >> Haskell-Cafe(a)haskell.org >> http://www.haskell.org/mailman/listinfo/haskell-cafe

1 0

Re: [Haskell-cafe] How to create an online poll
by gregg reynolds 18 Feb '09

18 Feb '09

Ok, that might protect us against the 7 plagues, so I'm not so worried about locusts. But what about Java, PHP etc. Is the monad sufficient protection against ultra-supernatural evil? Andrew Wagner <wagner.andrew(a)gmail.com> wrote: >Just wrap it in a ReallySafeWePromiseMonad > >2009/2/18 gregg reynolds <dev(a)mobileink.com> > >> I.e. war, plague.famine. etc. >> >> gregg reynolds <dev(a)mobileink.com> wrote: >> >> >But what about the side effects? >> > >> >Rick R <rick.richardson(a)gmail.com> wrote: >> > >> >>I'm sure Premier Election Solutions (formerly Diebold) can provide us >> with >> >>an online voting solution. >> >>Their value-add services allows us to set the outcome beforehand, so, in >> >>effect, the the voting process will be determinate. Which is certainly of >> >>interest to Haskell coders. >> >> >> >> >> >>On Wed, Feb 18, 2009 at 4:05 PM, Max Rabkin <max.rabkin(a)gmail.com> >> wrote: >> >> >> >>> On Wed, Feb 18, 2009 at 10:40 PM, Anton van Straaten >> >>> <anton(a)appsolutions.com> wrote: >> >>> > There's also the Condorcet Internet Voting Service: >> >>> > >> >>> > http://www.cs.cornell.edu/andru/civs.html >> >>> >> >>> This looks like exactly what we need! Any objections? >> >>> >> >>> --Max >> >>> _______________________________________________ >> >>> Haskell-Cafe mailing list >> >>> Haskell-Cafe(a)haskell.org >> >>> http://www.haskell.org/mailman/listinfo/haskell-cafe >> >>> >> >> >> >> >> >> >> >>-- >> >>We can't solve problems by using the same kind of thinking we used when >> we >> >>created them. >> >> - A. Einstein >> >>_______________________________________________ >> >>Haskell-Cafe mailing list >> >>Haskell-Cafe(a)haskell.org >> >>http://www.haskell.org/mailman/listinfo/haskell-cafe >> >> _______________________________________________ >> Haskell-Cafe mailing list >> Haskell-Cafe(a)haskell.org >> http://www.haskell.org/mailman/listinfo/haskell-cafe >> >>

2 1