On Sat, 2011-10-29 at 12:05 +0200, Herbert Valerio Riedel wrote:
I propose to add optimized {to,from}Strict conversion functions between strict and lazy ByteStrings to the Data.ByteString.Lazy API.
Discussion deadline: 2 weeks from now (12 November)
I see we're still before the deadline, but it seems like unanimous support. I've added the functions. They'll be included in bytestring-0.10.x. Thanks Herbert and others who chimed in. While I was at it, I also exported foldrChunks and foldlChunks so we now match the Text API in this area. As Don pointed out, we deliberately didn't include {to,from}Strict functions to discourage people from converting back and forth, since it's expensive. Since that has not proved popular I've just documented it instead: -- |/O(n)/ Convert a lazy 'ByteString' into a strict 'ByteString'. -- -- Note that this is an /expensive/ operation that forces the whole lazy -- ByteString into memory and then copies all the data. If possible, try to -- avoid converting back and forth between strict and lazy bytestrings. -- toStrict :: ByteString -> S.ByteString BTW, I'm slightly sceptical of the benchmarks, it's using tiny chunk sizes, in practice I don't expect the performance of toStrict to be much different from B.concat . BL.toChunks. Duncan
= Current State =
The current Data.ByteString.Lazy API doesn't provide direct conversion functions to/from single strict ByteStrings
Currently, there are only `fromChunks` and `toChunks`, by which convert to/from a list of strict ByteStrings.
A possible reference implementation of the missing conversion functions is:
fromStrict = BL.fromChunks . (:[])
and
toStrict = B.concat . BL.toChunks
== The Issues ==
The lack of `fromStrict`/`toStrict` in the Data.ByteString.Lazy API has the following issues:
- Convenience: If the single-strict-bytestring conversion is often needed, one tends to define module- or package-local helper functions for convenience/readability to perform the desired conversion. This violates the DRY principle.
- Principle of least suprise: Might be confusing to users new to `Data.ByteString.Lazy` why there is no direct conversion.
- Symmetry with `Data.Text.Lazy` API which does provide such single-strict-text conversion functions (`fromStrict`/`toStrict`)
- Performance: The above provided "naive" `toStrict` definition has a roughly 2 to 4 times higher overhead than a manually fused version (which was kindly provided by Bas van Dijk -- whom I'd like to thank for providing me with the optimized versions of toStrict and fromStrict) -- see end of this mail for criterion benchmark code and results
= Proposed Enhancement =
Enhance the Data.ByteString.Lazy API by adding the following conversion functions (suggestions for improvements are highly welcome):
-- see benchmark code at end of mail for the qualified imports
-- |/O(n)/ Convert a strict ByteString into a lazy ByteString. fromStrict :: B.ByteString -> BL.ByteString fromStrict = flip BLI.chunk BLI.Empty
-- |/O(n)/ Convert a lazy ByteString into a strict ByteString. toStrict :: BL.ByteString -> B.ByteString toStrict lb = BI.unsafeCreate len $ go lb where len = BLI.foldlChunks (\l sb -> l + B.length sb) 0 lb
go BLI.Empty _ = return () go (BLI.Chunk (BI.PS fp s l) r) ptr = withForeignPtr fp $ \p -> do BI.memcpy ptr (p `plusPtr` s) (fromIntegral l) go r (ptr `plusPtr` l)
== Benchmark Code & Results ==
------------------------------------------------------------------------ {-# LANGUAGE OverloadedStrings #-}
import Criterion import Criterion.Main import qualified Data.ByteString as B import qualified Data.ByteString.Internal as BI import qualified Data.ByteString.Lazy as BL import qualified Data.ByteString.Lazy.Internal as BLI import Foreign.ForeignPtr import Foreign.Ptr
toStrict1 :: BL.ByteString -> B.ByteString toStrict1 = B.concat . BL.toChunks
toStrict2 :: BL.ByteString -> B.ByteString toStrict2 lb = BI.unsafeCreate len $ go lb where len = BLI.foldlChunks (\l sb -> l + B.length sb) 0 lb
go BLI.Empty _ = return () go (BLI.Chunk (BI.PS fp s l) r) ptr = withForeignPtr fp $ \p -> do BI.memcpy ptr (p `plusPtr` s) (fromIntegral l) go r (ptr `plusPtr` l)
main :: IO () main = do let lbs1 = "abcdefghij" lbs2 = BL.fromChunks (replicate 10 "abcdefghij") lbs3 = BL.fromChunks (replicate 1000 "abcdefghij")
-- force evaluation of lbs{1,2,3} and verify validity print $ toStrict1 lbs1 == toStrict2 lbs1 print $ toStrict1 lbs2 == toStrict2 lbs2 print $ toStrict1 lbs3 == toStrict2 lbs3
defaultMain [ bgroup "toStrict" [ bench "simple #1" $ whnf toStrict1 lbs1 , bench "simple #2" $ whnf toStrict1 lbs2 , bench "simple #3" $ whnf toStrict1 lbs3
, bench "optimized #1" $ whnf toStrict2 lbs1 , bench "optimized #2" $ whnf toStrict2 lbs2 , bench "optimized #3" $ whnf toStrict2 lbs3 ] ]
{-
True True True warming up estimating clock resolution... mean is 2.302557 us (320001 iterations) found 2039 outliers among 319999 samples (0.6%) 1658 (0.5%) high severe estimating cost of a clock call... mean is 54.99870 ns (14 iterations) found 1 outliers among 14 samples (7.1%) 1 (7.1%) low mild
benchmarking toStrict/simple #1 mean: 28.96077 ns, lb 28.89527 ns, ub 29.01562 ns, ci 0.950 std dev: 305.8466 ps, lb 262.1008 ps, ub 345.6136 ps, ci 0.950
benchmarking toStrict/simple #2 mean: 487.0739 ns, lb 486.7939 ns, ub 487.4713 ns, ci 0.950 std dev: 1.699232 ns, lb 1.262363 ns, ub 2.457099 ns, ci 0.950
benchmarking toStrict/simple #3 mean: 55.06322 us, lb 54.91370 us, ub 55.20236 us, ci 0.950 std dev: 741.6239 ns, lb 656.3273 ns, ub 846.6403 ns, ci 0.950
benchmarking toStrict/optimized #1 mean: 48.67522 ns, lb 48.65188 ns, ub 48.70237 ns, ci 0.950 std dev: 129.3192 ps, lb 111.3761 ps, ub 165.4819 ps, ci 0.950
benchmarking toStrict/optimized #2 mean: 178.6342 ns, lb 178.5480 ns, ub 178.7276 ns, ci 0.950 std dev: 457.4436 ps, lb 409.2746 ps, ub 519.8267 ps, ci 0.950
benchmarking toStrict/optimized #3 mean: 13.01866 us, lb 13.00734 us, ub 13.03549 us, ci 0.950 std dev: 70.09916 ns, lb 52.18012 ns, ub 97.77226 ns, ci 0.950
-}
_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries