Re: [Haskell-cafe] Amazon AWS storage best to use with Haskell?

On Tue, Nov 1, 2011 at 10:53 AM, Neil Davies wrote:

Word of caution

Understand the semantics (and cost profile) of the AWS services first - you can't just open a HTTP connection and dribble data out over several days and hope for things to work. It is not a system that has that sort of laziness at its heart.

AWS doesn't supply a traditional remote file store semantics - is queuing, simple database and object store have all been designed for large scale systems being offered as a service to a (potentially hostile) large set of users - you can see that in the way that things are designed. There are all sorts of (sensible from their point of view) performance related limits and retries.

The challenge in designing nice clean layers on top of AWS is how/when to hide the transient/load related failures.

As a straw-man approach I would go first to NData.Map backed by Data.Map with addition of "flush" function to write Data.Map to external key-value store / NoSQL DB. Another requirement for NData.Map is concurrent consistency, so different clients could modify its state preserving "happen-before" relationship. For this I would add to NData.Map a "reftresh" function, that updates local copy from external key-value store. As for hSimpleDB package, it looks like it doesn't build on ghc7: http://hackage.haskell.org/package/hSimpleDB

The hSimpleDB package

Interface to Amazon's SimpleDB service. PropertiesVersions0.1 http://hackage.haskell.org/package/hSimpleDB-0.1, 0.2 http://hackage.haskell.org/package/hSimpleDB-0.2, *0.3*Dependencies base http://hackage.haskell.org/package/base-3.0.3.2 (≥3 & ≤4), bytestring http://hackage.haskell.org/package/bytestring-0.9.2.0, Cryptohttp://hackage.haskell.org/package/Crypto-4.2.4, dataenc http://hackage.haskell.org/package/dataenc-0.14.0.2, HTTPhttp://hackage.haskell.org/package/HTTP-4000.1.2, hxt http://hackage.haskell.org/package/hxt-9.1.4, networkhttp://hackage.haskell.org/package/network-2.3.0.7, old-locale http://hackage.haskell.org/package/old-locale-1.0.0.3, old-time http://hackage.haskell.org/package/old-time-1.0.0.7, utf8-string http://hackage.haskell.org/package/utf8-string-0.3.7License BSD3AuthorDavid Himmelstrup 2009, Greg Heartsfield 2007MaintainerDavid Himmelstrup CategoryDatabasehttp://hackage.haskell.org/packages/archive/pkg-list.html#cat:database, Web http://hackage.haskell.org/packages/archive/pkg-list.html#cat:web, Networkhttp://hackage.haskell.org/packages/archive/pkg-list.html#cat:networkUpload dateThu Sep 17 17:09:26 UTC 2009Uploaded byDavidHimmelstrupBuilt onghc-6.10, ghc-6.12Build failureghc-7.0 (loghttp://hackage.haskell.org/packages/archive/hSimpleDB/0.3/logs/failure/ghc-7... )