July 2010 - Libraries

cryptohash and an incremental API
by Thomas DuBuisson 20 Jul '10

20 Jul '10

Sorry for the double Vincent, forgot to include libraries(a)h.o > Just a quick stab in the dark, what about in the class a "hashName :: String" ? > it would be easy to have in DRBG a function that match name to strength: I've considered something more like data HashName = Tiger | MD5 | SHA1 | SHA256 | ... | OtherHash String hashName :: HashName Forcing a standardized name so tools such as DRBG won't have to guess or try to normalize: (`elem` ["SHA2-256", "SHA-256", "SHA256" ]) . map toUpper . filter (not . isSpace)) > Regardless of either strength is implemented this way or not, would that be a > good idea to have a name field for ciphers and hashes ? Its been I thought but one I'm not convinced of enough yet because we either have a problem of name standardization or crypto-api enumerating hashes. [wrt IVs] > That's still sounds quite odd. When initializing my IVs usually, this is not > something randomly generated, but something known on both side. So for an example IPSec with AES CBC we have a ByteString of IV ++ Ciphertext: decryptRFC3602 :: (BlockCipher k) => k -> B.ByteString -> Maybe (B.ByteString) decryptRFC3602 ctInfo = let m = runGetState get ctInfo 0 in case m of Left err -> Nothing -- Serialize decode error getting IV Right (iv,ct) -> snd (unCbc k iv ct) In IPSec the IV is randomly generated by the encryptor, so the 'getIV' method might be of use (or just use "liftM decode $ readFile "/dev/urandom"). If both sides know an IV they wish to start with void of any randomness then the job is even simpler: encryptCBCknownIV :: (BlockCipher k) => k -> Bytestring -> ByteString encryptCBCknownIV k pt = let iv = decode $ Lazy.pack [0..] in snd (cbc k iv pt) decryptCBCknownIV k ct = let iv = decode $ Lazy.pack [0..] in snd (unCbc k iv ct) > To me an IV should just be a function of a input byteString, and that let the > application of the interface pull the IV bytes from any place it need to. crypto-api is giving the developer certainty that any "iv :: IV k" is "blockSize `for` k" bits long - which is a nice guarantee. The cost is that people wanting non-random IVs need to place their IV in a bytestring and decode it via cereal or binary - not a high cost. Cheers, Thomas

1 0

Re: cryptohash and an incremental API
by Thomas DuBuisson 20 Jul '10

20 Jul '10

On Thu, Jul 15, 2010 at 2:24 PM, Vincent Hanquez <tab(a)snarc.org> wrote: [Re: Semantics of Hash "updateCtx" and my desire for semantics option 1.2 (replacing my original preference for semantics 1.1) > > I'm not sure which document you are refering to here. http://www.haskell.org/pipermail/libraries/2010-June/013782.html > While thinking about it, i'm not sure if tracking the remainder should be left > at the higher level; The change will trickle through the finalize function, > which might not be very pratical. I'm not a fan of changing the API because some particular C bindings would be inconvenienced. I can sympathize but it doesn't seem like a strong enough argument when the argument on the other side is that we are removing duplicate code from all hash implementations. Anyone else reading? Could other people weigh in on this issue too? [Re: 'strength' field of the Hash class] > indeed, just that strenght looks like a warm-fuzzy-feeling value, that might be > wrongly used to choose a hash algorithm automatically or something. Otherwise i > really don't mind. I'm considering this, how its loss would impact DRBG, and how many (if any) other libraries can make any reasonable use of "strength" without being overly dumb about its use. Still tempted to keep it, bad users of hashes can always be bad without our help anyway. >> buildIV :: (BlockCipher k, MonadRandom m) => m (IV k) > > I'm not sure i understand this however. In particular the MonadRandom bits, > why is it there ? Is this pulling bits from a random generator to construct an > iv ? Right, this was my turn to be dumb. I just now implemented it as: getIV :: RandomGen g => g -> (IV k, g) So yes, I'm pulling bits from an RNG to construct an IV but we should avoid the dependency on MonadRandom. If someone doesn't like RandomGen then they can always build a random bytestring of sufficient length (using whatever method) and run Data.Serialize.decode. [re: the RC4 implementation and StreamCipher class] > No, it's not hackage yet, I'll clean it up and put somewhere accessible in the > next couple of days. I'm not sure I can be considered a stream cipher expert > yet though. I'll try to gather some different algorithms implementation to see > how it can be generalized to. Sounds good. Cheers, Thomas

2 1

Most popular libraries not in the HP
by Don Stewart 19 Jul '10

19 Jul '10

According to the 2010 download stats, the most popular libs *not* in the Haskell Platform, by category are : Libs binary (rank 5/2200) utf8-string (rank 7/2200) ghc-paths (rank 15/2200) -- needed by haddock! hslogger (rank 16/2200 -- LPGL transformers (rank 18) -- related to mtl uniplate (rank 21) -- conflict with syb? terminfo (rank 25) haskeline (rank 27) -- used internally by ghci? json (rank 28) xml (rank 29) text (rank 31) HaXml (rank 33) -- LGPL HDBC (rank 42) -- LGPL convertible (rank 43) dlist (rank 54) vector (rank 55) pureMD5 (rank 70) cereal (rank 82) Apps: cpphs (rank 26) hscolour (rank 48) pandoc (rank 51) -- GPL Things we might realistically propose to add for 2011.1 New capabilities: * Binary parsing: binary + cereal ? * JSON parsing: json * An xml lib: xml -- prob. needs polish * Unicode: text * Vectors: vector * Hashing: pureMD5 With license changes to avoid static linking problems: * Databases: HDBC * XML: HaXml Questions remain about what GUI lib to bring in, to augment the OpenGL suite. * gtk2hs * wxHaskell Both cabalized, but will hamper the ease of building from source (since they bring in big C lib deps). Personally, I'd like to see binary parsing, vectors and unicode text done. Maybe JSON. -- Don

19 37

ANNOUNCE: container-classes-0.0.0.0
by Ivan Lazar Miljenovic 18 Jul '10

18 Jul '10

I'm pleased to announce the fruits of my labours during AusHac [1]: version 0.0.0.0 of the container-classes library [2] I was talking about in my last blog post [3]. [1]: http://www.haskell.org/haskellwiki/AusHac2010 [2]: http://hackage.haskell.org/package/container-classes [3]: http://ivanmiljenovic.wordpress.com/2010/07/14/data-oriented-hierarchies/ The purposes of this library is two-fold: * As a library writer, let your users choose which data structure they want to use rather than whichever one you decided upon when writing the library (e.g. you use lists predominantly, but your users may want to use Sets; by outputting a generic Container value yoru users don't need to keep using Set.fromList and Set.toList). * When writing your own code and you realise that you're using the wrong data structure for your purposes, if you used this library (correctly) then the amount of work you need to do to swap data structures is minimised. This current version is only a first draft. As such, I would appreciate feedback on the API. Currently, it provides its own definitions of all list-oriented functions in the Prelude plus a few more (e.g. partition), and list instances of all the classes. In terms of definitions, most class methods have default definitions in terms of fold. Furthermore, as much as possible I've defined them using a build/fold setup outlined in the paper "A Short Cut to Deforestation" by Andrew Gill, John Launchbury and Simon Peyton Jones (however I haven't yet defined the rules to get this used yet). However, the current list instances use the pre-defined versions in the Prelude and Data.List; I'm not sure if I should keep it this way or - where equivalent - use the pre-defined ones so that hopefully conversions from lists to other data types would also be removed without needing intermediary data structures. I also plan on adding more instances and a Lookup class for Maps, etc. However, we have to start packing up the room now and so my hacking is at an end for this weekend :( -- Ivan Lazar Miljenovic Ivan.Miljenovic(a)gmail.com IvanMiljenovic.wordpress.com

2 2

Re: Haskell Platform 2010.2.0.0 RC
by Don Stewart 18 Jul '10

18 Jul '10

ashley: > On Sat, 2010-07-17 at 21:01 -0700, Don Stewart wrote: > > Diamond dependencies, surely, will occur? > > The other packages shouldn't depend on the old version, even if it does > happen to be the one shipped with GHC. > > The Haskell Platform should be the blessing of package versions, not > GHC. The current process for maintainers of bootstrap libraries is to work closely with Ian Lynagh prior to a GHC release to make sure the correct version of e.g. bytestring, is shipped. Looks like that didn't happen with the time lib. Now, something GHC boots from must depend on time, or it wouldn't be in the bootstrap set. Hunting around, it looks like: random And these boot packages depend on old-time: Cabal directory haskell98 So, if we clobbered the version of time installed with GHC, we'd end up with possible diamond dep failures constructed by users of the random package :/ I agree this is an unsatisfactory situation, that versions shipped with GHC's boot libs are set outside of the control of the HP release team. -- Don

1 0

Haskell Platform 2010.2.0.0 RC
by Don Stewart 18 Jul '10

18 Jul '10

Hey all, Release candidate specification for the 2010.2.0.0 release of the Haskell Platform is now available, the first encorporating GHC 6.12.3: http://code.haskell.org/haskell-platform/haskell-platform.cabal RC for the Unix installer is here: http://www.galois.com/~dons/tmp/hp/haskell-platform-2010.2.0.0.tar.gz Time to get the Mac and Windows installers ready, and to prep the distro packages!! Release due next Thursday. Changes to watch for: Core packages * array bumped. * base 4 bumped * bytestring bumped. * Cabal bumped * extensible-exceptions added * filepath bumped * hpc bumped * old-time bumped * process bumped * template-haskell bumped * unix bumped 3rd party * cgi bumped * fgl bumped * QuickCheck bumped * regex-base bumped * regex-compat bumped * regex-posix bumped * stm bumped Apps * cabal-install bumped * alex bumped * happy bumped

5 10

The new locale library?
by Ashley Yakeley 18 Jul '10

18 Jul '10

From http://hackage.haskell.org/package/old-locale : "This package provides the old locale library. For new code, the new locale library is recommended." Can someone tell me where to find the new locale library? -- Ashley Yakeley

1 0

Re: Haskell Platform 2010.2.0.0 RC
by Don Stewart 16 Jul '10

16 Jul '10

nomeata: > Hi, > > Am Mittwoch, den 14.07.2010, 15:41 -0700 schrieb Don Stewart: > > Release candidate specification for the 2010.2.0.0 release of the > > Haskell Platform is now available, the first encorporating GHC 6.12.3: > > > > http://code.haskell.org/haskell-platform/haskell-platform.cabal > > > > RC for the Unix installer is here: > > > > http://www.galois.com/~dons/tmp/hp/haskell-platform-2010.2.0.0.tar.gz > > > > Time to get the Mac and Windows installers ready, and to prep the distro packages!! > > > > Release due next Thursday. > > > this leads us to the question: What do we want in squeeze? Are we happy > with the current situation? Or should we try to get 6.12.3 and 2010.2. > ready? I think the GHC 6.12.3 compiler fixes a number of significant bugs of 6.12.1, so that alone is worth the upgrade: * A crash when Haskell calls C, which in turn calls back into Haskell, has been fixed. * A bug which caused forking large processes to not terminate has been fixed. * Calling hSetEncoding will now set the encoding for both the read and write side of a Handle, rather than just the read side. * Using hReady and hWaitForInput on Handles created from Sockets will no longer block on Windows. * Some improvements have been made in when library functions do, and do not, block asynchronous exceptions. * bytestring: Fixed bug that caused hGetContents to block when it shouldn't. * A new flag -no-rtsopts flag allows you to disable parsing of +RTS ... options by the run-time system. In some use-cases, when a program can be run as another user, this can cause security problems. For example, myprog -tsomefile will overwrite somefile if the other user has permissions to do so. * ghc-pkg now ensures that the conf files it writes are UTF8. * TH: A bug in the kind parser has been fixed. * directory: System.Directory.doesFileExist on Unix-like OSes now correctly identifies various types of special file as files. -- Don

2 2

Corrected unix patch
by Eric Mertens 14 Jul '10

14 Jul '10

Hello, It looks like my darcs-fu is a bit rusty. I've attached the patch with all of the changes to this message for adding the missing functions to unix package Sorry for confusion. -- Eric Mertens <emertens(a)galois.com> Galois, Inc.

4 4

Re: cryptohash and an incremental API
by Thomas DuBuisson 14 Jul '10

14 Jul '10

Vincent said: > couple of comments around the hashes interface: > > * updateCtx works on blockLength, instead of working on arbitrary size... So for performance reasons you seem to prefer Semantics 1.2? """ 1.2 Multiple of blockSize bytes Implementations are encouraged to consume data (continue updating, encrypting, or decrypting) until there is less than blockSize bits available. """ Also, I'll amend 1.2 and say the hashUpdate/encrypt/decrypt functions should only consume n * blockSize bytes, tracking the remainder will be done at the higher level. Also, the higher level default implementations should only pass n * blocksize inputs to these functions. I can see how that's reasonable and am strongly considering using these semantics instead of 1.1. > * hash is a generic operation based on the class Hash. In my case, it improve > performance by not running the pure init/update/finalize exposed, but use the hidden > impure function. I realized yesterday it's not as much as i though since i had > a bug in my benchmark, but it's still there (100ms for 500mb of data). Humm, 0.2 sections / GB is significant so again I can be swayed - it isn't like I can't have a default definition of hash (and others) when its part of the class instance. > * Why is the digest of a specific type ? I like representing different > things with different types, but i'm not sure what do you gain with digests > though. This I am less flexible on. My thought on how people will use this library is centered around the instantiation of classes on the keys used or resulting digests. Anyone wanting ByteString results can simply use Data.[Serialize,Binary].encode. Here is a user getting a sha256 hash: let h = hash contents :: SHA256 or the type could be implicit due to context (not shown): let h = hash contents > * is strength really useful in the Hash class ? it might be accurate when the > thing get implemented, but i'm not sure what would happens over time, and flaws > are discovered. would people actually updates it ? Will people actually update it? I hope so but if they don't are we really worse off than not having any strength numbers? People who care about strength will likely keep track of the algorithms on which they depend. I added strength largely because the Hash class came from DRBG (NIST SP 800-90) and that needed strength values. If we don't have strength then applications like DRBG need a way to know which algorithm each data type represents then to look up that algorithm their its own table of algorithm strength - very messy. I'd imaging crypto-api would have to look something like: \begin{code} data HashAlgorithm = MD5 | SHA1 | SHA256 | SHA512 | ... class Hash d c | d -> c, c -> d where ... algorithm :: Tagged d HashAlgorithm ... \end{code} I don't consider this a win - crypto-api now enumerating all hash algorithms wanting Hash instances. > The blockCipher should exposes the chaining modes as overridable typeclass > functions, with default generic implementations that use encryptBlocks. For > example the haskell AES package has different C implementations for each > chaining modes (e.g. cbc, ebc), and i suspect that using a generic chaining > implementation would slow things down. As with "hash" being part of the hash typeclass, I don't have a strong objection here. It allows particular implementations to be slightly higher performance and does not preclude default definitions. This is rather messier than I wanted, but the reasoning seems sound. WRT your specific examples: encryptBlocksCBC :: k -> ByteString -> (k, ByteString) decryptBlocksCBC :: k -> ByteString -> (k, ByteString) These I do object to. The key does not change as the CBC algorithm progresses, but contextual information does. My initial mode implementations have types like: cbc :: (BlockCipher k) => k -> IV k -> ByteString -> (ByteString, IV k) In other words, initialization vectors are explicit and separate from the key. The type parameter on IV allows us to build an IV of proper size, something like: buildIV :: (BlockCipher k, MonadRandom m) => m (IV k) and it is always true that iv :: IV k iv <- buildIV B.length (encode iv) == blockSize `for` (undefined :: k) > and my last comment, is that i don't understand the streamcipher interface > you're proposing. I've got a (inefficient) RC4 implementation that has this > interface: > > stream :: Ctx -> B.ByteString -> (Ctx, B.ByteString) > streamlazy :: Ctx -> L.ByteString -> (Ctx, L.ByteString) My interface was just a quick hack with me understanding it would likely change - I didn't know there was a Haskell RC4 binding or implementation and will happily follow your lead here. Is this implementation on hackage? Cheers, Thomas

1 0