We use AWS extensively. We use the aws package and have contributed to it, specifically SQS functionality. I will give you the rundown of what we do.<div><br></div><div>We moved off of SimpleDb and now use mondodb. The reason is that simple db seemed to have problems with write pressure and there are not good tools for profiling your queries. My main application is extremely write heavy with a single instance needing to do 100s or 1000s of writes a second. Mongodb has worked well for us. I am scared of things like cassandra having looked at the code, however some people have made it work.</div>
<div><br></div><div>We store data such as crawled web pages in S3. The files are lzma compressed and the data format is built on protocol buffers. We picked lzma for both storage costs of cold data and the fact that the pipe between S3 and EC2 is somewhat limited and we want to make the most effective use of it as possible.<br>
<br>In my experience AWS simulators are more trouble than they are worth since they don&#39;t accurately model the way AWS will respond to you under load. The free tier at AWS should allow you to experiment with building an app. The first couple of months of development cost us less than $1.</div>
<div><br></div><div>Steve<br><br><div class="gmail_quote">On Tue, Nov 1, 2011 at 1:27 AM, dokondr <span dir="ltr">&lt;<a href="mailto:dokondr@gmail.com">dokondr@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br><br><div class="gmail_quote"><div class="im">On Tue, Nov 1, 2011 at 10:53 AM, Neil Davies <span dir="ltr">&lt;<a href="mailto:semanticphilosopher@gmail.com" target="_blank">semanticphilosopher@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Word of caution<div><br></div><div>Understand the semantics (and cost profile) of the AWS services first - you can&#39;t just open a HTTP connection and dribble data out over several days and hope for things to work. It is not a system that has that sort of laziness at its heart.</div>

<div><br></div><div>AWS doesn&#39;t supply a traditional remote file store semantics - is queuing, simple database and object store have all been designed for large scale systems being offered as a service to a (potentially hostile) large set of users - you can see that in the way that things are designed. There are all sorts of (sensible from their point of view) performance related limits and retries.</div>

<div><br></div><div>The challenge in designing nice clean layers on top of AWS is how/when to hide the transient/load related failures.</div><div><br></div><font color="#888888"><div><br></div></font></div></blockquote></div>
<div>
<br>As a straw-man approach I would go first to NData.Map backed by Data.Map with addition of &quot;flush&quot; function  to write Data.Map to external key-value store / NoSQL DB. <br>Another requirement for NData.Map is concurrent consistency, so different clients could modify its state preserving &quot;happen-before&quot; relationship. For this I would add to NData.Map a &quot;reftresh&quot; function, that updates local copy from  external key-value store.<br>

<br>As for hSimpleDB package, it looks like it doesn&#39;t build on ghc7:<br><a href="http://hackage.haskell.org/package/hSimpleDB" target="_blank">http://hackage.haskell.org/package/hSimpleDB</a><br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">

<div style="word-wrap:break-word"><font color="#888888"><div></div></font><h1>The hSimpleDB package</h1><p>Interface to Amazon&#39;s SimpleDB service.
</p><h2>Properties</h2><table><tbody><tr><th align="left" valign="top">Versions</th><td><a href="http://hackage.haskell.org/package/hSimpleDB-0.1" target="_blank">0.1</a>, <a href="http://hackage.haskell.org/package/hSimpleDB-0.2" target="_blank">0.2</a>, <b>0.3</b></td>

</tr><tr><th align="left" valign="top">Dependencies</th><td><a href="http://hackage.haskell.org/package/base-3.0.3.2" target="_blank">base</a> (≥3 &amp; ≤4), <a href="http://hackage.haskell.org/package/bytestring-0.9.2.0" target="_blank">bytestring</a>, <a href="http://hackage.haskell.org/package/Crypto-4.2.4" target="_blank">Crypto</a>, <a href="http://hackage.haskell.org/package/dataenc-0.14.0.2" target="_blank">dataenc</a>, <a href="http://hackage.haskell.org/package/HTTP-4000.1.2" target="_blank">HTTP</a>, <a href="http://hackage.haskell.org/package/hxt-9.1.4" target="_blank">hxt</a>, <a href="http://hackage.haskell.org/package/network-2.3.0.7" target="_blank">network</a>, <a href="http://hackage.haskell.org/package/old-locale-1.0.0.3" target="_blank">old-locale</a>, <a href="http://hackage.haskell.org/package/old-time-1.0.0.7" target="_blank">old-time</a>, <a href="http://hackage.haskell.org/package/utf8-string-0.3.7" target="_blank">utf8-string</a></td>

</tr><tr><th align="left" valign="top">License</th><td>BSD3</td></tr><tr><th align="left" valign="top">Author</th><td>David Himmelstrup 2009, Greg Heartsfield 2007</td></tr><tr><th align="left" valign="top">Maintainer</th>

<td>David Himmelstrup &lt;<a href="mailto:lemmih@gmail.com" target="_blank">lemmih@gmail.com</a>&gt;</td></tr><tr><th align="left" valign="top">Category</th><td><a href="http://hackage.haskell.org/packages/archive/pkg-list.html#cat:database" target="_blank">Database</a>, <a href="http://hackage.haskell.org/packages/archive/pkg-list.html#cat:web" target="_blank">Web</a>, <a href="http://hackage.haskell.org/packages/archive/pkg-list.html#cat:network" target="_blank">Network</a></td>

</tr><tr><th align="left" valign="top">Upload date</th><td>Thu Sep 17 17:09:26 UTC 2009</td></tr><tr><th align="left" valign="top">Uploaded by</th><td>DavidHimmelstrup</td></tr><tr><th align="left" valign="top">Built on</th>

<td>ghc-6.10, ghc-6.12</td></tr><tr><th align="left" valign="top">Build failure</th><td>ghc-7.0 (<a href="http://hackage.haskell.org/packages/archive/hSimpleDB/0.3/logs/failure/ghc-7.0" target="_blank">log</a>)</td></tr>
</tbody></table></div>
</blockquote><div> </div></div>
<br>_______________________________________________<br>
Haskell-Cafe mailing list<br>
<a href="mailto:Haskell-Cafe@haskell.org">Haskell-Cafe@haskell.org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/haskell-cafe" target="_blank">http://www.haskell.org/mailman/listinfo/haskell-cafe</a><br>
<br></blockquote></div><br>
</div>