Re: [Haskell-cafe] Encrypting streamed data

On Jul 6, 2017, at 12:58 AM, Ivan Lazar Miljenovic wrote:

I have a use case for needing to use public key cryptography to encrypt a large amount of data in a streaming fashion (get it out of a DB, encrypt, put into an AWS S3 bucket).

What are the data-format requirements? Do you need (binary) CMS output? GPG-compatible output? Or just roll your own? Integrity protection can be tricky with large data streams. Most data formats for enveloped data have a single MAC at the end, which means that the decoder has to consume all the data before it is known to be valid! So if you're in a position to avoid a standard all-in-one format, it makes sense to "packetize" the stream, with integrity protection for each "packet", and packet sequence numbers to preserve overall stream integrity. With vast amounts of data, you'll want to be careful with the symmetric cipher modes, AEAD (AES-GCM, for example) protects only a limited amount of data before you need to rekey. It may be simplest to just generate a new symmetric key for every N megabytes of data. With a careful design of the "packet" format, you can use in-memory crypto for each packet. Don't forget to include an "end-of-stream" packet to defeat truncation attacks. -- Viktor.