----------------------------------------------------------------------------------------------------------------------------------------------------- Enumerators are an efficient, predictable, and safe alternative to lazy I/O. Discovered by Oleg Kiselyov, they allow large datasets to be processed in near–constant space by pure code. Although somewhat more complex to write, using enumerators instead of lazy I/O produces more correct programs. http://hackage.haskell.org/package/enumerator http://john-millikin.com/software/enumerator/ ----------------------------------------------------------------------------------------------------------------------------------------------------- Hello -cafe, It's been a while since the last point release of enumerator. This one is sufficiently large that I think folks might want to know about it, and since I try not to spam too many announcements, I'll give a quick rundown on major changes in other 0.4.x versions as well. First, most of what I call "list analogues" -- enumerator-based versions of 'head', 'take', 'map', etc -- have been separated into three modules (Data.Enumerator.List, .Binary, and .Text) depending on what sorts of data they operate on. This separation has been an ongoing process throughout 0.4.x releases, and I think it's now complete. The old names in Data.Enumerator will continue to exist in 0.4.x versions, but will be removed in 0.5. Second, Gregory Collins and Ertugrul Soeylemez found a space leak in Iteratee's (>>=), which could cause eventual space exhaustion in some circumstances. If you use enumerators to process very large or infinite streams, you probably want to upgrade to version 0.4.7 or higher. Third, the source code PDF has seen some substantial improvement -- if you're interested in how the library is implemented, or have insomnia, read it at < http://john-millikin.com/software/enumerator/enumerator_0.4.8.pdf
Finally, there is a known issue in the current encoding of iteratees -- if an iteratee yields extra data but never consumed anything, that iteratee will violate the monad law of associativity. Oleg has updated his implementations to fix this problem, but since it would break a *lot* of dependent libraries, I'm holding off until the vague future of version 0.5. Since iteratees that yield extra data they didn't consume are invalid anyway, I hope this problem will not cause too much inconvenience. New features ----------------- * Range-limited binary file enumeration (requested + initial patch by Bardur Arantsson). * splitWhen , based on the "split" package < http://hackage.haskell.org/package/split > * 0.4.6: Typeable instances for most types (requested by Michael Snoyman) * 0.4.5: joinE , which simplifies enumerator/enumeratee composition (requested by Michael Snoyman)
Great work as usual John. I'm actually very happy to see
enumHandleRange: the next version of WAI will support partial files,
and I just implemented my own version of enumHandleRange over there. I
will gladly switch to your (most likely more correct) version.
As far as the left-over data in a yield issue: does that require a
breaking API change, or a change to the definition of >>= which would
change semantics??
Michael
On Sat, Mar 26, 2011 at 7:39 AM, John Millikin
----------------------------------------------------------------------------------------------------------------------------------------------------- Enumerators are an efficient, predictable, and safe alternative to lazy I/O. Discovered by Oleg Kiselyov, they allow large datasets to be processed in near–constant space by pure code. Although somewhat more complex to write, using enumerators instead of lazy I/O produces more correct programs.
http://hackage.haskell.org/package/enumerator http://john-millikin.com/software/enumerator/ -----------------------------------------------------------------------------------------------------------------------------------------------------
Hello -cafe,
It's been a while since the last point release of enumerator. This one is sufficiently large that I think folks might want to know about it, and since I try not to spam too many announcements, I'll give a quick rundown on major changes in other 0.4.x versions as well.
First, most of what I call "list analogues" -- enumerator-based versions of 'head', 'take', 'map', etc -- have been separated into three modules (Data.Enumerator.List, .Binary, and .Text) depending on what sorts of data they operate on. This separation has been an ongoing process throughout 0.4.x releases, and I think it's now complete. The old names in Data.Enumerator will continue to exist in 0.4.x versions, but will be removed in 0.5.
Second, Gregory Collins and Ertugrul Soeylemez found a space leak in Iteratee's (>>=), which could cause eventual space exhaustion in some circumstances. If you use enumerators to process very large or infinite streams, you probably want to upgrade to version 0.4.7 or higher.
Third, the source code PDF has seen some substantial improvement -- if you're interested in how the library is implemented, or have insomnia, read it at < http://john-millikin.com/software/enumerator/enumerator_0.4.8.pdf
Finally, there is a known issue in the current encoding of iteratees -- if an iteratee yields extra data but never consumed anything, that iteratee will violate the monad law of associativity. Oleg has updated his implementations to fix this problem, but since it would break a *lot* of dependent libraries, I'm holding off until the vague future of version 0.5. Since iteratees that yield extra data they didn't consume are invalid anyway, I hope this problem will not cause too much inconvenience.
New features -----------------
* Range-limited binary file enumeration (requested + initial patch by Bardur Arantsson).
* splitWhen , based on the "split" package < http://hackage.haskell.org/package/split >
* 0.4.6: Typeable instances for most types (requested by Michael Snoyman)
* 0.4.5: joinE , which simplifies enumerator/enumeratee composition (requested by Michael Snoyman)
_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries
Since the release, a couple people have sent in feature requests, so I'm going to put out 0.4.9 in a day or so. New features will be: - tryIO: runs an IO computation, and converts any exceptions into ``throwError`` calls (requested by Kazu Yamamoto) - checkContinue: encapsulates a common pattern (loop (Continue k) = ...) when defining enumerators - mapAccum and mapAccum: sort of like map and mapM, except the step function is stateful (requested by Long Huynh Huu) Anyone else out there sitting on a request? Please send them in -- I am always happy to receive them, even if they must be declined. --- Also, I would like to do a quick poll regarding operators. 1. It has been requested that I add operator aliases for joinI and joinE. 2. There have been complaints that the library defines too many operators (currently, 5). Do any existing enumerator users, or anyone for that matter, have an opinion either way? The proposed operators are: ---------------------------------------------------------------------- infixr 0 =$ infixr 0 $= (=$) :: Monad m => Enumeratee ao ai m b -> Iteratee ai m b -> Iteratee ao m b enum =$ iter = joinI (enum $$ iter) ($=) :: Monad m => Enumerator ao m (Step ai m b) -> Enumeratee ao ai m b -> Enumerator ai m b ($=) = joinE ----------------------------------------------------------------------
On Tue, Mar 29, 2011 at 7:50 AM, John Millikin
Since the release, a couple people have sent in feature requests, so I'm going to put out 0.4.9 in a day or so.
New features will be:
- tryIO: runs an IO computation, and converts any exceptions into ``throwError`` calls (requested by Kazu Yamamoto)
- checkContinue: encapsulates a common pattern (loop (Continue k) = ...) when defining enumerators
- mapAccum and mapAccum: sort of like map and mapM, except the step function is stateful (requested by Long Huynh Huu)
Anyone else out there sitting on a request? Please send them in -- I am always happy to receive them, even if they must be declined.
---
Also, I would like to do a quick poll regarding operators.
1. It has been requested that I add operator aliases for joinI and joinE.
2. There have been complaints that the library defines too many operators (currently, 5).
Do any existing enumerator users, or anyone for that matter, have an opinion either way?
The proposed operators are:
---------------------------------------------------------------------- infixr 0 =$ infixr 0 $=
(=$) :: Monad m => Enumeratee ao ai m b -> Iteratee ai m b -> Iteratee ao m b enum =$ iter = joinI (enum $$ iter)
($=) :: Monad m => Enumerator ao m (Step ai m b) -> Enumeratee ao ai m b -> Enumerator ai m b ($=) = joinE ----------------------------------------------------------------------
The operators sound good to me. My only request would be to put in a usage example in the documentation. I'd be happy to write one if you'd like. Personally, I think that =$ will *greatly* clean up my code. Michael
Hello,
(=$) :: Monad m => Enumeratee ao ai m b -> Iteratee ai m b -> Iteratee ao m b enum =$ iter = joinI (enum $$ iter)
($=) :: Monad m => Enumerator ao m (Step ai m b) -> Enumeratee ao ai m b -> Enumerator ai m b ($=) = joinE ----------------------------------------------------------------------
The operators sound good to me. My only request would be to put in a usage example in the documentation. I'd be happy to write one if you'd like. Personally, I think that =$ will *greatly* clean up my code.
I have a tutorial to describe how to use the enumerator library in Japanese. Since it is popular among the Haskell community in Japan, I guess it's worth translating into English. So, I did. http://www.mew.org/~kazu/proj/enumerator/ This tutorial explains how to use (=$) and ($=) as well as other operators(($$), (<==<), (>>=)). Of course, my English is broken. If English native speakers will kindly correct broken grammar, it would be appreciated. I'm reachable by e-mail or twitter (@kazu_yamamoto). --Kazu
0.4.9 has been uploaded to cabal, with the new operators. Changes are in the replied-to post (and also quoted below), plus the new operators proposed by Kazu Yamamoto. Here's the corresponding docs (they have examples!) ------------------------------------------------------------------------------------------------------ -- | @enum =$ iter = 'joinI' (enum $$ iter)@ -- -- “Wraps” an iteratee /inner/ in an enumeratee /wrapper/. -- The resulting iteratee will consume /wrapper/’s input type and -- yield /inner/’s output type. -- -- Note: if the inner iteratee yields leftover input when it finishes, -- that extra will be discarded. -- -- As an example, consider an iteratee that converts a stream of UTF8-encoded -- bytes into a single 'TL.Text': -- -- > consumeUTF8 :: Monad m => Iteratee ByteString m Text -- -- It could be written with either 'joinI' or '(=$)': -- -- > import Data.Enumerator.Text as ET -- > -- > consumeUTF8 = joinI (decode utf8 $$ ET.consume) -- > consumeUTF8 = decode utf8 =$ ET.consume -- -- Since: 0.4.9 -- | @enum $= enee = 'joinE' enum enee@ -- -- “Wraps” an enumerator /inner/ in an enumeratee /wrapper/. -- The resulting enumerator will generate /wrapper/’s output type. -- -- As an example, consider an enumerator that yields line character counts -- for a text file (e.g. for source code readability checking): -- -- > enumFileCounts :: FilePath -> Enumerator Int IO b -- -- It could be written with either 'joinE' or '($=)': -- -- > import Data.Text as T -- > import Data.Enumerator.List as EL -- > import Data.Enumerator.Text as ET -- > -- > enumFileCounts path = joinE (enumFile path) (EL.map T.length) -- > enumFileCounts path = enumFile path $= EL.map T.length -- -- Since: 0.4.9 ------------------------------------------------------------------------------------------------------ Minor release note -- 0.4.9 and 0.4.9.1 are the exact same code; I just forgot a @ in one of the new docs and had to re-upload so Hackage would haddock properly. There is no difference in behavior. On Monday, March 28, 2011 10:50:45 PM UTC-7, John Millikin wrote:
Since the release, a couple people have sent in feature requests, so I'm going to put out 0.4.9 in a day or so.
New features will be:
- tryIO: runs an IO computation, and converts any exceptions into ``throwError`` calls (requested by Kazu Yamamoto)
- checkContinue: encapsulates a common pattern (loop (Continue k) = ...) when defining enumerators
- mapAccum and mapAccum: sort of like map and mapM, except the step function is stateful (requested by Long Huynh Huu)
Anyone else out there sitting on a request? Please send them in -- I am always happy to receive them, even if they must be declined.
---
Also, I would like to do a quick poll regarding operators.
1. It has been requested that I add operator aliases for joinI and joinE.
2. There have been complaints that the library defines too many operators (currently, 5).
Do any existing enumerator users, or anyone for that matter, have an opinion either way?
The proposed operators are:
---------------------------------------------------------------------- infixr 0 =$ infixr 0 $=
(=$) :: Monad m => Enumeratee ao ai m b -> Iteratee ai m b -> Iteratee ao m b enum =$ iter = joinI (enum $$ iter)
($=) :: Monad m => Enumerator ao m (Step ai m b) -> Enumeratee ao ai m b -> Enumerator ai m b ($=) = joinE ----------------------------------------------------------------------
Thanks, I look forward to being able to use these new operators!
On Tue, Mar 29, 2011 at 8:22 PM, John Millikin
0.4.9 has been uploaded to cabal, with the new operators. Changes are in the replied-to post (and also quoted below), plus the new operators proposed by Kazu Yamamoto.
Here's the corresponding docs (they have examples!)
------------------------------------------------------------------------------------------------------ -- | @enum =$ iter = 'joinI' (enum $$ iter)@ -- -- “Wraps” an iteratee /inner/ in an enumeratee /wrapper/. -- The resulting iteratee will consume /wrapper/’s input type and -- yield /inner/’s output type. -- -- Note: if the inner iteratee yields leftover input when it finishes, -- that extra will be discarded. -- -- As an example, consider an iteratee that converts a stream of UTF8-encoded -- bytes into a single 'TL.Text': -- -- > consumeUTF8 :: Monad m => Iteratee ByteString m Text -- -- It could be written with either 'joinI' or '(=$)': -- -- > import Data.Enumerator.Text as ET -- > -- > consumeUTF8 = joinI (decode utf8 $$ ET.consume) -- > consumeUTF8 = decode utf8 =$ ET.consume -- -- Since: 0.4.9
-- | @enum $= enee = 'joinE' enum enee@ -- -- “Wraps” an enumerator /inner/ in an enumeratee /wrapper/. -- The resulting enumerator will generate /wrapper/’s output type. -- -- As an example, consider an enumerator that yields line character counts -- for a text file (e.g. for source code readability checking): -- -- > enumFileCounts :: FilePath -> Enumerator Int IO b -- -- It could be written with either 'joinE' or '($=)': -- -- > import Data.Text as T -- > import Data.Enumerator.List as EL -- > import Data.Enumerator.Text as ET -- > -- > enumFileCounts path = joinE (enumFile path) (EL.map T.length) -- > enumFileCounts path = enumFile path $= EL.map T.length -- -- Since: 0.4.9 ------------------------------------------------------------------------------------------------------
Minor release note -- 0.4.9 and 0.4.9.1 are the exact same code; I just forgot a @ in one of the new docs and had to re-upload so Hackage would haddock properly. There is no difference in behavior.
On Monday, March 28, 2011 10:50:45 PM UTC-7, John Millikin wrote:
Since the release, a couple people have sent in feature requests, so I'm going to put out 0.4.9 in a day or so.
New features will be:
- tryIO: runs an IO computation, and converts any exceptions into ``throwError`` calls (requested by Kazu Yamamoto)
- checkContinue: encapsulates a common pattern (loop (Continue k) = ...) when defining enumerators
- mapAccum and mapAccum: sort of like map and mapM, except the step function is stateful (requested by Long Huynh Huu)
Anyone else out there sitting on a request? Please send them in -- I am always happy to receive them, even if they must be declined.
---
Also, I would like to do a quick poll regarding operators.
1. It has been requested that I add operator aliases for joinI and joinE.
2. There have been complaints that the library defines too many operators (currently, 5).
Do any existing enumerator users, or anyone for that matter, have an opinion either way?
The proposed operators are:
---------------------------------------------------------------------- infixr 0 =$ infixr 0 $=
(=$) :: Monad m => Enumeratee ao ai m b -> Iteratee ai m b -> Iteratee ao m b enum =$ iter = joinI (enum $$ iter)
($=) :: Monad m => Enumerator ao m (Step ai m b) -> Enumeratee ao ai m b -> Enumerator ai m b ($=) = joinE ----------------------------------------------------------------------
participants (3)
-
John Millikin -
Kazu Yamamoto -
Michael Snoyman