On 6 July 2017 at 17:23, David Turner wrote:

Apologies, just seen the bit about wanting to pass the key in directly rather than using the GPG keyring because there are no email addresses attached to your various keys.

Maybe a silly question, but can you give them email addresses to identify them in a GPG-compatible manner? They don't have to have associated mailboxes so the addresses can be totally made-up.

Yes, that's my fallback, since it's possible to tell gpg to use a different directory so I can provide a key externally with my transmission request, load it into a temporary store, grab the identity out and use that. It would just be more convenient to have a "here's a ByteString with the public key" option (which I can always implement as a wrapper function).

If not, I'd probably start to look to something like openssl to do the symmetric encryption and manage the keys for that separately. It looks possible to build a streaming AES implementation using the nonstreaming functions in `cryptonite`, but the usual recommendation is that its far too risky to do any low-level crypto yourself so this seems like a bad idea.

On 6 Jul 2017 07:40, "David Turner" wrote:

...

Hi,

I do not know of a library to do this, sorry. Note that the way public-key crypto works in a streaming fashion is typically to use the public-key bit only to encrypt a key for a symmetric cipher and then use the (much-faster) symmetric encryption for the actual data. The symmetric bit could well be something like AES256-CBC or AES256-CTR.

This means that the format of the resulting data is a bit implementation-defined as it has to include the asymetrically-encrypted data first, followed by the stream of symmetrically-encrypted data. GnuPG includes quite a bit of metadata in its files that describes the algorithms used and delimits the pieces, so if you want the resulting files to be GnuPG-compatible you'll need to take this into account.

If it were me, I'd probably just shell out to `gpg`. It's fast and low-risk.

Hope that helps,

David

On 6 Jul 2017 05:59, "Ivan Lazar Miljenovic" wrote:

I have a use case for needing to use public key cryptography to encrypt a large amount of data in a streaming fashion (get it out of a DB, encrypt, put into an AWS S3 bucket).

The command-line gpg tool seems to be able to encrypt/decrypt data from stdin to stdout in a streaming fashion, but in my attempts to use it it seems very file-based for things like the keys to use (whereas I would prefer to be able to pass the public key as an actual value rather than a file; if nothing else because this is for tools that don't have email addresses to use and base their keys on for addressing).

Is there an existing library that can achieve this using conduit/pipes/whatever? cryptonite-conduit only covers hashing, hOpenPGP is poorly documented and I can't work out how to use it ("just follow the types" is difficult when Haddock docs don't link to the required types (seems to be because it uses the "import Module as X" trick for re-exporting everything but then everything from those modules isn't available).

Can anyone recommend a solution?

-- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

-- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com

On 6 July 2017 at 19:17, Patrick Chilton wrote:

If your data fits in RAM maybe it's best to forget about the streaming and use the saltine library's Box module. Then you can't really go wrong on the crypto front.

If that was the case, I wouldn't be asking ;-)

On Jul 6, 2017 09:41, "Ivan Lazar Miljenovic" wrote:

...

On 6 July 2017 at 17:23, David Turner wrote:

...

Apologies, just seen the bit about wanting to pass the key in directly rather than using the GPG keyring because there are no email addresses attached to your various keys.

Maybe a silly question, but can you give them email addresses to identify them in a GPG-compatible manner? They don't have to have associated mailboxes so the addresses can be totally made-up.

Yes, that's my fallback, since it's possible to tell gpg to use a different directory so I can provide a key externally with my transmission request, load it into a temporary store, grab the identity out and use that. It would just be more convenient to have a "here's a ByteString with the public key" option (which I can always implement as a wrapper function).

...

If not, I'd probably start to look to something like openssl to do the symmetric encryption and manage the keys for that separately. It looks possible to build a streaming AES implementation using the nonstreaming functions in `cryptonite`, but the usual recommendation is that its far too risky to do any low-level crypto yourself so this seems like a bad idea.

On 6 Jul 2017 07:40, "David Turner" wrote:

...

Hi,

I do not know of a library to do this, sorry. Note that the way public-key crypto works in a streaming fashion is typically to use the public-key bit only to encrypt a key for a symmetric cipher and then use the (much-faster) symmetric encryption for the actual data. The symmetric bit could well be something like AES256-CBC or AES256-CTR.

This means that the format of the resulting data is a bit implementation-defined as it has to include the asymetrically-encrypted data first, followed by the stream of symmetrically-encrypted data. GnuPG includes quite a bit of metadata in its files that describes the algorithms used and delimits the pieces, so if you want the resulting files to be GnuPG-compatible you'll need to take this into account.

If it were me, I'd probably just shell out to `gpg`. It's fast and low-risk.

Hope that helps,

David

On 6 Jul 2017 05:59, "Ivan Lazar Miljenovic" wrote:

I have a use case for needing to use public key cryptography to encrypt a large amount of data in a streaming fashion (get it out of a DB, encrypt, put into an AWS S3 bucket).

The command-line gpg tool seems to be able to encrypt/decrypt data from stdin to stdout in a streaming fashion, but in my attempts to use it it seems very file-based for things like the keys to use (whereas I would prefer to be able to pass the public key as an actual value rather than a file; if nothing else because this is for tools that don't have email addresses to use and base their keys on for addressing).

Is there an existing library that can achieve this using conduit/pipes/whatever? cryptonite-conduit only covers hashing, hOpenPGP is poorly documented and I can't work out how to use it ("just follow the types" is difficult when Haddock docs don't link to the required types (seems to be because it uses the "import Module as X" trick for re-exporting everything but then everything from those modules isn't available).

Can anyone recommend a solution?

-- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

-- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

-- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com

https://gist.github.com/snoyberg/20243aae347b38ad09daaf8b129e2efb

It's got some magic values in a few places (especially that 65!)

Off topic, but sometimes when I find myself using magic values I can't/won't get rid off, I'll just apply the good old habits learned in Java times. For example you might find a section at the top of a file with things like _DAYS_IN_A_WEEK_ , _KNOWN_SIZE_OF_POINT_ :: INT-- | Seems to be a good approximation for now _DAYS_IN_A_WEEK_ = 7 -- | When you ask "What's the point", this will not answer your question. _KNOWN_SIZE_OF_POINT_ = 65 Uppercase makes it easy to identify these as constants/magic values. The underscore in front works as an initial lowercase letter so they can be used as values despite uppercase. The goal is not necessarily to make these values easy to change, but to add documentation to usage sites. It's unnecessary In such a quick demonstration, but I wanted to take the opportunity to throw in my _VALUE_OF_CONTRIBUTION_IN_CENTS_ cents because I haven't seen others do something like this. Cheers, MarLinn PS: The underscore does introduce its own error message though. • Found hole: _VALUE_OF_CONTRIBUTION_IN_CENTS_ :: Double Or perhaps ‘_VALUE_OF_CONTRIBUTION_IN_CENTS_’ is mis-spelled, or not in scope

You can avoid the underscores by defining the constants as pattern synonyms so that they are syntactically more like constructors. This also, of course, allows you to use them as patterns, which can also be nice for constants.

Off topic, but sometimes when I find myself using magic values I can't/won't get rid off, I'll just apply the good old habits learned in Java times. For example you might find a section at the top of a file with things like

_DAYS_IN_A_WEEK_ , _KNOWN_SIZE_OF_POINT_ :: INT-- | Seems to be a good approximation for now _DAYS_IN_A_WEEK_ = 7 -- | When you ask "What's the point", this will not answer your question. _KNOWN_SIZE_OF_POINT_ = 65

Nice! This seems a bit evil, and not a full substitute. But it also helps hijack syntax highlighting, and it forces users to use a recent-ish version of GHC. I consider that A Good Thing™. Now… as everything is “constant” in Haskell, on to define everything in capital letter patterns and return to good old COBOL times! 🍺 Wohoo! Cheers, MarLinn

On 7 July 2017 at 01:44, Viktor Dukhovni wrote:

...

On Jul 6, 2017, at 12:58 AM, Ivan Lazar Miljenovic wrote:

I have a use case for needing to use public key cryptography to encrypt a large amount of data in a streaming fashion (get it out of a DB, encrypt, put into an AWS S3 bucket).

What are the data-format requirements? Do you need (binary) CMS output? GPG-compatible output? Or just roll your own?

The intent is to be able to transfer data between two parties such that only the recipient is able to view it (hence the usage of public key cryptography). GPG/PGP compatability is preferable as it's common, but anything that is sufficiently standardised (as this will potentially be used by others that aren't me doing so with Haskell and thus can't just use a library to do so) will suffice. (The other advantage of GPG/PGP is that the security testing team is more familiar with it and thus likely to sign off on it.)

Integrity protection can be tricky with large data streams. Most data formats for enveloped data have a single MAC at the end, which means that the decoder has to consume all the data before it is known to be valid!

So if you're in a position to avoid a standard all-in-one format, it makes sense to "packetize" the stream, with integrity protection for each "packet", and packet sequence numbers to preserve overall stream integrity. With vast amounts of data, you'll want to be careful with the symmetric cipher modes, AEAD (AES-GCM, for example) protects only a limited amount of data before you need to rekey. It may be simplest to just generate a new symmetric key for every N megabytes of data.

With a careful design of the "packet" format, you can use in-memory crypto for each packet. Don't forget to include an "end-of-stream" packet to defeat truncation attacks.

This sounds good in theory, but in practice I'm not versed enough in security to want to try and roll my own if I could avoid it, and trying to document such a format for others to use could be problematic.allowed to post. -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com

I'm also not sure what benefit you would get from rolling your own versus using AWS's server-side S3 bucket encryption[1], now that I think about it. [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.htm... On Thu, Jul 6, 2017 at 3:56 PM Rein Henrichs wrote:

If you are not a security expert, I would strongly recommend against rolling your own encryption scheme. If you are a security expert, you would probably not be asking us for advice on how to roll your own encryption scheme. I would suggest that you find something off the rack that might meet your needs (whether it's written in Haskell or not) and make sure you understand how it works and what your threat model is well enough to decide whether it does, in fact, meet your needs.

On Thu, Jul 6, 2017 at 3:29 PM Ivan Lazar Miljenovic < ivan.miljenovic@gmail.com> wrote:

...

On 7 July 2017 at 01:44, Viktor Dukhovni wrote:

...

...

On Jul 6, 2017, at 12:58 AM, Ivan Lazar Miljenovic <

ivan.miljenovic@gmail.com> wrote:

...

...

I have a use case for needing to use public key cryptography to encrypt a large amount of data in a streaming fashion (get it out of a DB, encrypt, put into an AWS S3 bucket).

What are the data-format requirements? Do you need (binary) CMS output? GPG-compatible output? Or just roll your own?

The intent is to be able to transfer data between two parties such that only the recipient is able to view it (hence the usage of public key cryptography). GPG/PGP compatability is preferable as it's common, but anything that is sufficiently standardised (as this will potentially be used by others that aren't me doing so with Haskell and thus can't just use a library to do so) will suffice.

(The other advantage of GPG/PGP is that the security testing team is more familiar with it and thus likely to sign off on it.)

...

Integrity protection can be tricky with large data streams. Most data formats for enveloped data have a single MAC at the end, which means that the decoder has to consume all the data before it is known to be valid!

So if you're in a position to avoid a standard all-in-one format, it makes sense to "packetize" the stream, with integrity protection for each "packet", and packet sequence numbers to preserve overall stream integrity. With vast amounts of data, you'll want to be careful with the symmetric cipher modes, AEAD (AES-GCM, for example) protects only a limited amount of data before you need to rekey. It may be simplest to just generate a new symmetric key for every N megabytes of data.

With a careful design of the "packet" format, you can use in-memory crypto for each packet. Don't forget to include an "end-of-stream" packet to defeat truncation attacks.

This sounds good in theory, but in practice I'm not versed enough in security to want to try and roll my own if I could avoid it, and trying to document such a format for others to use could be problematic.allowed to post.

-- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Ah, yes. That would do it. On Thu, Jul 6, 2017 at 4:25 PM Ivan Lazar Miljenovic < ivan.miljenovic@gmail.com> wrote:

On 7 July 2017 at 09:04, Rein Henrichs wrote:

...

I'm also not sure what benefit you would get from rolling your own versus using AWS's server-side S3 bucket encryption[1], now that I think about it.

[1]

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.htm...

Due to regulatory requirements, we can't rely upon that and need to ensure we never put any private data in AWS that even someone with login access to AWS can read.

...

On Thu, Jul 6, 2017 at 3:56 PM Rein Henrichs wrote:

...

If you are not a security expert, I would strongly recommend against rolling your own encryption scheme. If you are a security expert, you

would

...

...

probably not be asking us for advice on how to roll your own encryption scheme. I would suggest that you find something off the rack that might meet your needs (whether it's written in Haskell or not) and make sure you understand how it works and what your threat model is well enough to decide whether it does, in fact, meet your needs.

On Thu, Jul 6, 2017 at 3:29 PM Ivan Lazar Miljenovic wrote:

...

On 7 July 2017 at 01:44, Viktor Dukhovni

wrote:

...

...

...

On Jul 6, 2017, at 12:58 AM, Ivan Lazar Miljenovic wrote:

I have a use case for needing to use public key cryptography to encrypt a large amount of data in a streaming fashion (get it out

of a

...

...

DB, encrypt, put into an AWS S3 bucket).

What are the data-format requirements? Do you need (binary) CMS output? GPG-compatible output? Or just roll your own?

The intent is to be able to transfer data between two parties such that only the recipient is able to view it (hence the usage of public key cryptography). GPG/PGP compatability is preferable as it's common, but anything that is sufficiently standardised (as this will potentially be used by others that aren't me doing so with Haskell and thus can't just use a library to do so) will suffice.

(The other advantage of GPG/PGP is that the security testing team is more familiar with it and thus likely to sign off on it.)

...

Integrity protection can be tricky with large data streams. Most

data

...

formats for enveloped data have a single MAC at the end, which means that the decoder has to consume all the data before it is known to be valid!

So if you're in a position to avoid a standard all-in-one format, it makes sense to "packetize" the stream, with integrity protection for each "packet", and packet sequence numbers to preserve overall stream integrity. With vast amounts of data, you'll want to be careful with the symmetric cipher modes, AEAD (AES-GCM, for example) protects only a limited amount of data before you need to rekey. It may be simplest to just generate a new symmetric key for every N megabytes of data.

With a careful design of the "packet" format, you can use in-memory crypto for each packet. Don't forget to include an "end-of-stream" packet to defeat truncation attacks.

This sounds good in theory, but in practice I'm not versed enough in security to want to try and roll my own if I could avoid it, and trying to document such a format for others to use could be problematic.allowed to post.

-- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

-- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com http://IvanMiljenovic.wordpress.com

On Tue, Jul 11, 2017 at 03:35:56PM +0100, David Turner wrote:

I can't think of a terribly good way to achieve GPG/PGP-compatibility without simply using GPG/PGP, since the file format is quite involved.

That said, here is how to implement a CBC-mode block cipher encryption using Conduit, which is suitable for something like AES256 encryption. It is almost certainly vulnerable to side-channel attacks (timing, cache-poisoning, etc) but as a pure function from input to output it is equivalent to `openssl aes-256-cbc -e -K <KEY-IN-HEX> -iv <IV-IN-HEX> -in data/plain-text.txt` which I should hope would be standard enough for analysis.

Just straight CBC lacks integrity protection. A MAC is still required, and usually one wants asymmetric key exchange, rather than a shared symmetric key. So in practice one wants something like CMS (successor to S/MIME). The OpenSSL cms(1) command can do clear signing, encryption, or both (in either order) by piping the output of one to the other. In many applications it is safer to encrypt then sign, rather than sign and then encrypt. Packetizing large input streams is well worth it, but while each "packet" can use any of a number of standar formats, some standards (notably CMS AFAIK) may lack support for packetizing large input streams. The OpenPGP standard does support breaking streams into "packets", so for large streams that may be optimal, you just need an OpenPGP library implementation that supports sensible packet sizes, and perhaps an FFI interface for Haskell. Alternatively, just a pipe to a CLI will do, bug the "gpg" CLI does not appear to support creating streams with more than one packet (Unless this happens implicitly for "large-enough" streams). -- Viktor.