Re: [Haskell-cafe] Splittable random numbers

Hi cafe,
I want to add the ability to use AES-NI instructions on Intel architectures
to GHC. Mainly I'd like to do splittable random number generators based on
AES as was suggested at the outset of this email. (I met Burton Smith last
week and this topic came up.)
I was just reading the below thread about the plugin architecture which got
me thinking about what the right way to add AES-NI is. (Disregarding for a
moment portability and the issue of where to test cpuid...)
http://www.haskell.org/pipermail/glasgow-haskell-users/2011-January/019874.h...
The FFI is always an option. But after reading the first N pages I could
come across from google I'm still not totally clear on whether "unsafe"
foreign calls can happen simultaneously from separate Haskell threads (and
with sufficiently low overhead for this purpose).
I also ran across the phrase "compiler primitive" somewhere wrt GHC:
http://hackage.haskell.org/trac/ghc/wiki/AddingNewPrimitiveOperations
Is that the right way to go? Or would the compiler plugin mechanism
possibly allowing doing this without modifying mainline GHC?
Thanks,
-Ryan
On Fri, Nov 12, 2010 at 6:26 PM, wren ng thornton
On 11/12/10 5:33 AM, Richard Senington wrote:
It does not give the results you would want. This may have something to do with picking "good" parameters for the mkLehmerTree function. For example, using a random setup, I just got these results result expected range 16.814 expected = 16.0 (1,31) 16.191 expected = 16.5 (1,32) 16.576 expected = 17.0 (1,33) 17.081 expected = 17.5 (1,34) 17.543 expected = 18.0 (1,35)
Have you run any significance tests? I wouldn't be surprised to see +/-0.5 as within the bounds of expected randomness. I'm more worried about it seeming to be consistently on the -0.5 end of things, than I am about it not matching expectation (how many samples did you take again?). For small ranges like this, a consistent -0.5 (or +0.5) tends to indicate off-by-one errors in the generator.
-- Live well, ~wren
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Ryan,
If you make an AES based RNG then consider making an instance for
CryptoRandomGen (see DRBG [1] for example instances). Such an
instance means you can use "splitGen" [2], which can split generators
in the manner described in this thread. If you make the RNG match
NIST SP 800-90 then feel free to send it to me for inclusion in the
DRBG package, I've been meaning to make the block cipher based DRBG
for a while now.
Finally, any implementation of AES (using NI or not) could probably go
in its own package or a cipher-specific package like CryptoCipher[3].
Its a shame we don't have an AES implementation on Hackage that 1)
exposes the fundamental block interface instead of some higher-level
wrapping and 2) isn't tied to a large library.
Cheers,
Thomas
[1] http://hackage.haskell.org/package/DRBG
and
http://hackage.haskell.org/packages/archive/DRBG/0.1.2/doc/html/src/Crypto-R...
[2] http://hackage.haskell.org/packages/archive/crypto-api/0.3.1/doc/html/Crypto...
[3] http://hackage.haskell.org/package/cryptocipher
On Fri, Jan 21, 2011 at 2:19 PM, Ryan Newton
Hi cafe,
I want to add the ability to use AES-NI instructions on Intel architectures to GHC. Mainly I'd like to do splittable random number generators based on AES as was suggested at the outset of this email. (I met Burton Smith last week and this topic came up.)
I was just reading the below thread about the plugin architecture which got me thinking about what the right way to add AES-NI is. (Disregarding for a moment portability and the issue of where to test cpuid...)
http://www.haskell.org/pipermail/glasgow-haskell-users/2011-January/019874.h...
The FFI is always an option. But after reading the first N pages I could come across from google I'm still not totally clear on whether "unsafe" foreign calls can happen simultaneously from separate Haskell threads (and with sufficiently low overhead for this purpose).
I also ran across the phrase "compiler primitive" somewhere wrt GHC: http://hackage.haskell.org/trac/ghc/wiki/AddingNewPrimitiveOperations
Is that the right way to go? Or would the compiler plugin mechanism possibly allowing doing this without modifying mainline GHC?
Thanks, -Ryan
On Fri, Nov 12, 2010 at 6:26 PM, wren ng thornton
wrote: On 11/12/10 5:33 AM, Richard Senington wrote:
It does not give the results you would want. This may have something to do with picking "good" parameters for the mkLehmerTree function. For example, using a random setup, I just got these results result expected range 16.814 expected = 16.0 (1,31) 16.191 expected = 16.5 (1,32) 16.576 expected = 17.0 (1,33) 17.081 expected = 17.5 (1,34) 17.543 expected = 18.0 (1,35)
Have you run any significance tests? I wouldn't be surprised to see +/-0.5 as within the bounds of expected randomness. I'm more worried about it seeming to be consistently on the -0.5 end of things, than I am about it not matching expectation (how many samples did you take again?). For small ranges like this, a consistent -0.5 (or +0.5) tends to indicate off-by-one errors in the generator.
-- Live well, ~wren _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users

I'm not too familiar with all the Haskell API's for RNGs. This is the first time I've looked at CryptoRandomGen, but I can see the benefit of having a bytestring interface rather than the System.Random Int based one. Is there a reason that the AES implementation in the "AES" or "crypto" packages can't be ripped out and repackage in the way you would like? Cheers, -Ryan On Fri, Jan 21, 2011 at 6:11 PM, Thomas DuBuisson < thomas.dubuisson@gmail.com> wrote:
Ryan, If you make an AES based RNG then consider making an instance for CryptoRandomGen (see DRBG [1] for example instances). Such an instance means you can use "splitGen" [2], which can split generators in the manner described in this thread. If you make the RNG match NIST SP 800-90 then feel free to send it to me for inclusion in the DRBG package, I've been meaning to make the block cipher based DRBG for a while now.
Finally, any implementation of AES (using NI or not) could probably go in its own package or a cipher-specific package like CryptoCipher[3]. Its a shame we don't have an AES implementation on Hackage that 1) exposes the fundamental block interface instead of some higher-level wrapping and 2) isn't tied to a large library.
Cheers, Thomas
[1] http://hackage.haskell.org/package/DRBG and
http://hackage.haskell.org/packages/archive/DRBG/0.1.2/doc/html/src/Crypto-R... [2] http://hackage.haskell.org/packages/archive/crypto-api/0.3.1/doc/html/Crypto... [3] http://hackage.haskell.org/package/cryptocipher
On Fri, Jan 21, 2011 at 2:19 PM, Ryan Newton
wrote: Hi cafe,
I want to add the ability to use AES-NI instructions on Intel architectures to GHC. Mainly I'd like to do splittable random number generators based on AES as was suggested at the outset of this email. (I met Burton Smith last week and this topic came up.)
I was just reading the below thread about the plugin architecture which got me thinking about what the right way to add AES-NI is. (Disregarding for a moment portability and the issue of where to test cpuid...)
http://www.haskell.org/pipermail/glasgow-haskell-users/2011-January/019874.h...
The FFI is always an option. But after reading the first N pages I could come across from google I'm still not totally clear on whether "unsafe" foreign calls can happen simultaneously from separate Haskell threads
(and
with sufficiently low overhead for this purpose).
I also ran across the phrase "compiler primitive" somewhere wrt GHC:
http://hackage.haskell.org/trac/ghc/wiki/AddingNewPrimitiveOperations
Is that the right way to go? Or would the compiler plugin mechanism possibly allowing doing this without modifying mainline GHC?
Thanks, -Ryan
On Fri, Nov 12, 2010 at 6:26 PM, wren ng thornton
wrote:
On 11/12/10 5:33 AM, Richard Senington wrote:
It does not give the results you would want. This may have something to do with picking "good" parameters for the mkLehmerTree function. For example, using a random setup, I just got these results result expected range 16.814 expected = 16.0 (1,31) 16.191 expected = 16.5 (1,32) 16.576 expected = 17.0 (1,33) 17.081 expected = 17.5 (1,34) 17.543 expected = 18.0 (1,35)
Have you run any significance tests? I wouldn't be surprised to see
+/-0.5
as within the bounds of expected randomness. I'm more worried about it seeming to be consistently on the -0.5 end of things, than I am about it not matching expectation (how many samples did you take again?). For small ranges like this, a consistent -0.5 (or +0.5) tends to indicate off-by-one errors in the generator.
-- Live well, ~wren _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
participants (2)
-
Ryan Newton
-
Thomas DuBuisson