Hi, Am Freitag, den 02.07.2010, 12:55 +1000 schrieb Ivan Miljenovic:

This can be resolved by putting the build-depends line in the if statement (and should maybe put up the top of the executable section to make it more obvious):

the latter was not possible last time I checked. Greetings, Joachim -- Joachim "nomeata" Breitner mail: mail@joachim-breitner.de | ICQ# 74513189 | GPG-Key: 4743206C JID: nomeata@joachim-breitner.de | http://www.joachim-breitner.de/ Debian Developer: nomeata@debian.org

Hi, Am Freitag, den 02.07.2010, 17:14 +1000 schrieb Ivan Miljenovic:

On 2 July 2010 16:57, Joachim Breitner wrote:

...

Am Freitag, den 02.07.2010, 12:55 +1000 schrieb Ivan Miljenovic:

...

This can be resolved by putting the build-depends line in the if statement (and should maybe put up the top of the executable section to make it more obvious):

the latter was not possible last time I checked.

Do you mean putting the if statement at the top of the executable section? Works for me: http://hackage.haskell.org/packages/archive/graphviz/2999.9.0.0/graphviz.cab... (also in other cases that I've looked at).

sorry, no, I meant wrapping the whole executable section in an if statement, which to seems the most natural way of making an executable conditional. Greetings, Joachim -- Joachim "nomeata" Breitner mail: mail@joachim-breitner.de | ICQ# 74513189 | GPG-Key: 4743206C JID: nomeata@joachim-breitner.de | http://www.joachim-breitner.de/ Debian Developer: nomeata@debian.org

Vincent Hanquez writes:

On Fri, Jul 02, 2010 at 12:55:03PM +1000, Ivan Miljenovic wrote:

...

On 1 July 2010 17:25, Vincent Hanquez wrote:

...

The main reason for this library is the lack of incremental api exposed by current digest libraries, and filling the void about some missing digest algorithms; Also the speed comes as a nice bonus.

Can you explain what you mean by "incremental API"?

The incremental API is the init/update/finalize functions where you can call update as many time as you need, instead of a single function "hash" where you need to hash all your data in one-go.

It's necessary in my case since i receive chunks of data to be hashed from the network, and I don't want to carry a buffer of data (with potential security issues), until i can hash everything.

The few existing packages that exposes the incremental API, usually do it in the IO monad; cryptohash do it purely, creating a new context as it get updated. (this has a cost but remains fast globally with the C implementation)

i.e. update : ctx -> bytestring -> IO () becomes: update : ctx -> bytestring -> ctx

So you're using explicit state parsing? Any particular reason for not using the State monad or something like that? -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/2/10 07:23 , Vincent Hanquez wrote:

I'm not sure exactly what the API would looks like, but I think basically you would enter/leave the state monad quite frequently in incremental mode, since the whole point of the incremental api is having this context built partially. for this reason, you end up with something really close to explicit state keeping, isn't it ?

You can do this if you need to (execState) but it's better to wrap the whole thing in a State monad. Think of it this way: (flip runState) can be read as "withContext":

flip runState initContext $ do -- do stuff, invoking update as needed to build the state

Although now that I think about it, if we're just appending to the state, this should possibly be a Writer instead of a State; the interface is simpler. Or to try to be a little more concrete about it:

type HashContext = String

-- build up a hash context incrementally, then hash it hashWithContext :: () -> Hash hashWithContext = hash . execWriter

-- add a chunk to the hash context addToHash :: String -> Writer HashContext () addToHash = tell

-- use it messageHash = hashWithContext $ do -- do something addToHash aLine -- doSomethingElse foldM addToHash [aListOfStuff] -- ...

- -- brandon s. allbery [linux,solaris,freebsd,perl] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwt/z0ACgkQIn7hlCsL25Uz5wCeO9db+eqE/azqoEgJ3L3vQJzF XKEAn0I8rEzf0tQfylc3pGq4+E6V4M72 =C/BN -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/2/10 17:24 , Gregory Crosswhite wrote:

The problem with this approach is that the hash context isn't a monoid; you can absorb data into the context, but you can't combine two hash contexts to form a new one. Thus, the Writer monad won't work for this purpose.

No? The context is simply a String, which is a monoid, and you can combine contexts quite meaningfully before you actually compute the final hash (in fact, that's the whole point!). - -- brandon s. allbery [linux,solaris,freebsd,perl] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwuiMcACgkQIn7hlCsL25XSbACfZ72nV39ozNV2yvoTO1gtpewK AMMAn2mKd0j0xQ7Ns728Jzoz7Xq8GqSo =ZKBD -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/2/10 20:51 , John Meacham wrote:

On Fri, Jul 02, 2010 at 08:48:07PM -0400, Brandon S Allbery KF8NH wrote:

...

On 7/2/10 17:24 , Gregory Crosswhite wrote:

...

The problem with this approach is that the hash context isn't a monoid; you can absorb data into the context, but you can't combine two hash contexts to form a new one. Thus, the Writer monad won't work for this purpose.

No? The context is simply a String, which is a monoid, and you can combine contexts quite meaningfully before you actually compute the final hash (in fact, that's the whole point!).

The context isn't a string. it is an intermediate state in the algorithm of the hash function. it is usually an opaque binary blob (represented by a ByteString) of a size that is on the order of the final hash (unrelated to the size of the input).

If you read the example code I posted, the point was how to turn the current monolithic hash function into a cumulative one by using Writer. As such, there's no opaque hash state inside the Writer. (see `hash . runWriter'). - -- brandon s. allbery [linux,solaris,freebsd,perl] allbery@kf8nh.com system administrator [openafs,heimdal,too many hats] allbery@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwujqcACgkQIn7hlCsL25W6UQCfXrwrxclyns5s6TODstkO4z0a 1DwAoLLHJpZ36iEGiydkFy+0GTQWFbFI =bTrg -----END PGP SIGNATURE-----

On Fri, Jul 02, 2010 at 11:01:17AM -0400, Brandon S Allbery KF8NH wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 7/2/10 07:23 , Vincent Hanquez wrote:

...

I'm not sure exactly what the API would looks like, but I think basically you would enter/leave the state monad quite frequently in incremental mode, since the whole point of the incremental api is having this context built partially. for this reason, you end up with something really close to explicit state keeping, isn't it ?

You can do this if you need to (execState) but it's better to wrap the whole thing in a State monad. Think of it this way: (flip runState) can be read as "withContext":

...

flip runState initContext $ do -- do stuff, invoking update as needed to build the state

I think that if the user wants to do that, he would have to build the state monad himself. Fortunately it's a trivial exercise in this case. In my case i'm having a state monad that is not only related to the hashing, but contains more state, so it would not be pratical having to "lift" into the digest context state monad just to update, then got back a context, that i would put in the bigger state monad.

Although now that I think about it, if we're just appending to the state, this should possibly be a Writer instead of a State; the interface is simpler.

Or to try to be a little more concrete about it:

I think that's exactly what i don't want to expose for the simple reason that it's just too easy to shoot self in foot with that. for example, you might be in a situation where the data that you need to hash is really big, and appending Gb of data is not going to fly. In the end, I'ld rather leave this policy to the user, so that a careful user, on controlled case, could implement a writer monad. -- Vincent

On Fri, Jul 02, 2010 at 08:10:19PM +1000, Ivan Lazar Miljenovic wrote:

...

The few existing packages that exposes the incremental API, usually do it in the IO monad; cryptohash do it purely, creating a new context as it get updated. (this has a cost but remains fast globally with the C implementation)

i.e. update : ctx -> bytestring -> IO () becomes: update : ctx -> bytestring -> ctx

So you're using explicit state parsing? Any particular reason for not using the State monad or something like that?

Not using the state monad allows explicit sharing/storing of the context, which would be quite handy if you arn't hashing your whole input in one go. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/

On 02/07/10 22:41, Gregory Crosswhite wrote:

On 7/2/10 5:16 AM, Vincent Hanquez wrote:

...

It's necessary in my case since i receive chunks of data to be hashed from the network, and I don't want to carry a buffer of data (with potential security issues), until i can hash everything.

As an aside, this kind of pattern where you are processing chunks of input and producing a summary output sounds like a good fit for the Iteratees style of I/O.

Indeed. I haven't played with iteratee yet, but it seems fairly easy for the user to add a iteratee interface on top of update and finalize. basically it seems that Chunk = update and EOF = finalize.

Also, if you want the intermediate chunks to go away, then don't forget to use strict evaluation when you update the context. :-) That is unfortunately one of the kinds of bugs that doesn't screw up the behaviour of the program in a way that is immediately obvious.

yes, you're absolutely right. I did actually forget in an early version of some code i've got on top of cryptohash, and it took me a good while to find ;) -- Vincent

On Fri, Jul 02, 2010 at 02:03:23PM +0200, Ertugrul Soeylemez wrote:

Hello Vincent,

Hello Ertugrul,

it's nice to have such a package. It's going to be very useful for me. Thank you a lot!

you're welcome.

BTW, as a third API type possible in Haskell I suggest making a lazy ByteString variant. In some places this could well replace the incremental API and be more natural to use.

I think it's already covered by the library. every module has a "hashlazy" (for the lack of better name) alongside "hash". hash takes a strict bytestring and hashlazy takes a lazy bytestring, so for example opening a huge file with Data.BS.Lazy.readFile that you pass to hashlazy is fully supported. Cheers, -- Vincent