ANNOUNCE: swapper: Transparently swapping data from in-memory structures to disk
The package swapper provides a wrapper for functors, which allows their data to be automatically and transparently swapped to disk and loaded back when necessary. The version 0.1 is the first public one. Hackage: http://hackage.haskell.org/package/swapper Github: http://github.com/roman-smrz/swapper/ The original motivation was the use with happstack-state, which is a system to maintain a global state whole kept in memory. If the data get too big, they can be swapped to disk by the operating system, yet sometimes a bit more control may be beneficial. That is provided by this package (a cache, with possible custom implementation, deciding which and how many items are kept in memory, is associated with the structure) and also the use of snapshots is optimized more than in the case of OS swap: only the data currently cached need to be added when creating snapshot and after loading one, the data are read only as needed. Regards, Roman Srmž
Epic! Lemmih and I have talked about doing something like this in theory. But having it in reality is much, much better :) Have you considered switching to acid-state and safecopy? Future versions of happstack will use those instead of happstack-data and happstack-state. (And, there is no reason you can not use them with Happstack 6 already). - jeremy On Mon, Aug 29, 2011 at 4:43 PM, Roman Smrž <roman.smrz@seznam.cz> wrote:
The package swapper provides a wrapper for functors, which allows their data to be automatically and transparently swapped to disk and loaded back when necessary. The version 0.1 is the first public one.
Hackage: http://hackage.haskell.org/package/swapper Github: http://github.com/roman-smrz/swapper/
The original motivation was the use with happstack-state, which is a system to maintain a global state whole kept in memory. If the data get too big, they can be swapped to disk by the operating system, yet sometimes a bit more control may be beneficial. That is provided by this package (a cache, with possible custom implementation, deciding which and how many items are kept in memory, is associated with the structure) and also the use of snapshots is optimized more than in the case of OS swap: only the data currently cached need to be added when creating snapshot and after loading one, the data are read only as needed.
Regards, Roman Srmž
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
Hi On Tue, Aug 30, 2011 at 12:24:44AM +0200, Jeremy Shaw wrote:
Lemmih and I have talked about doing something like this in theory.
But having it in reality is much, much better :)
It's nice to hear that someone's interested :)
Have you considered switching to acid-state and safecopy? Future versions of happstack will use those instead of happstack-data and happstack-state. (And, there is no reason you can not use them with Happstack 6 already).
Since the interface of safecopy and acid-state looks quite similar to the one of happstack-data + happstack-state, that should not be a problem. Since most of the code is independent of these libraries, it would be possible to support both, but that would entail either declaring dependencies on all of those or providing separate packages for the interoperability parts, and if happstack-state is going to be more or less deprecated, it is probably not worth the hassle. Although what I would ideally need as an interface for happstack/acid-state would be a variant Serialize/SafeCopy, which would allow IO actions to be performed during serialization/deserialization. This is currently achieved in a bit hackish way using a wrapper around createCheckpoint, which ensures that for each datafile prefix only one actual serialization/deserialization of Swapper takes place, regardless of how many times are those evaluated. But it also means that only one from a group of related Swappers (those descended from a single call to mkSwapper), may be present in the serialized state. Though maybe some cleverer approach can solve this problem. Roman
participants (2)
-
Jeremy Shaw -
Roman Smrž