On Tue, Apr 27, 2004 at 02:05:06PM -0400, S. Alexander Jacobson wrote:
On Tue, 27 Apr 2004, Tomasz Zielonka wrote:
On Mon, Apr 26, 2004 at 08:43:44PM -0400, ajb@spamcop.net wrote:
It seems I qualify as almost nobody, cause I've had a database with more than 2^34 records on a single IA32 machine. However, I wouldn't ever think of loading the entire database to RAM.
Why not? Its only 17gb.
About 17000000000 records, yes, but not 17GB. It was more like 300 GB compressed data.
If you are doing any substantive volume of transactions on that database* you may be better off storing the database in memory than buying the multiple CPUs and reliable RAID arrays you need to serve it off disk.
I was not a typical relational database, so it's hard to compare. I managed on a 1 CPU P4 2.6 with 1GB mem and a RAID5 array. The database engine was optimised for bulk processing, reducing disk seeks and memory requirements.
A Dell PowerEdge 4600 w/ 24gb of memory and a write-caching controller w/ 6 36GB harddrives (RAID0 state,logging,status) costs around $34k. http://www1.us.dell.com/content/products/compare.aspx/tower?c=us&cs=555&l=en&s=biz Using e.g. a Prevayler style in memory logged data structures** you may be able to serve >100,000 queries/second. (see e.g. http://www.prevayler.org/wiki.jsp?topic=ScalabilityTestResults)
We were able to achieve a throughput of 200,000 records per seconds (not queries), which was about 3 times faster then necessary. The data you provided is interesting. However, if I had such a machine, my work could be much less challenging :)
* OTOH, if this is just your personal mail archive then nevermind :- )
No. Best regards, Tom -- .signature: Too many levels of symbolic links