UIO's also only a truly alpha idea as a proxy for bytestring/Binary support. Patches welcome. pumpkingod:

As far as I know, the reason for this is that the UIO instance for productions writes the two "rows" out sequentially to file, but doesn't include any means to determine the length of the two halves when it's loading up again. When you try to read the production back in, it tries to read in two arrays, but the first array read consumes all the input leaving the second with nothing.

Having said that, I'm not sure why it used to work. I remember testing this on the version in hackage and finding the same issue.

Cheers, Dan

On Fri, Mar 13, 2009 at 4:46 PM, Daniel Fischer wrote:

...

Am Freitag, 13. März 2009 21:18 schrieb Manlio Perillo:

...

Hi.

I'm sure this is a know bug, but I can't find details with Google.

The offending code is here: http://hpaste.org/fastcgi/hpaste.fcgi/view?id=2362

When I execute the program I get:    uio: readChunkBU: can't read

What's the problem?

I'm using uvector from: http://code.haskell.org/~dons/code/uvector/

Thanks   Manlio Perillo

Worked with uvector-0.1.0.1:

$ ghc --make readUArr.hs [1 of 1] Compiling Main             ( readUArr.hs, readUArr.o ) Linking readUArr ... $ ./readUArr 2

But not with uvector-0.2 $ ./readUArr readUArr: readChunkBU: can't read _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

...

As far as I know, the reason for this is that the UIO instance for productions writes the two "rows" out sequentially to file, but doesn't include any means to determine the length of the two halves when it's loading up again. When you try to read the production back in, it tries to read in two arrays, but the first array read consumes all the input leaving the second with nothing.

Ok, thanks.

For now, I think that the simple solution is to not use UArr (a:*:b), but (UArr a, UArr b).

Just newtype your own UIO instance, if you need a different instance... -- Don

manlio_perillo:

Daniel Fischer ha scritto:

...

[...] Worked with uvector-0.1.0.1:

[...] But not with uvector-0.2 [...]

The main difference is that in uvector 0.2, hPutBU does not write in the file the length of the array; hGetBU simply use the file size.

let elemSize = sizeBU 1 (undefined :: e) n <- fmap ((`div`elemSize) . fromInteger) $ hFileSize h

So, the problem seems to be here. This simply don't support having two arrays written in the same file, and sizeBU belongs to the UAE class, whose instances are only declared for builtin types.

So, the patch is: "just revert this change".

Or... use your own UIO instance. That's why it's a type class! Anyway, for the background on this: Tue Nov 18 08:44:46 PST 2008 Malcolm Wallace * Use hFileSize to determine arraysize, rather than encoding it in the file. "Here is a patch to the uvector library that fixes hGetBU and hPutBU to use the filesize to determine arraysize, rather than encoding it within the file. I guess the disadvantages are that now only one array can live in a file, and the given Handle must indeed be a file, not a socket Handle. But the advantage is that one can read packed raw datafiles obtained externally." Still, again, I'd point out that uvector is alpha, APIs can and will change. -- Don

manlio_perillo:

Don Stewart ha scritto:

...

[...]

...

So, the patch is: "just revert this change".

Or... use your own UIO instance. That's why it's a type class!

Why should I rewrite the UIO instance, if one already exists?

Oh, because you want different serialization semantics to the (arbitrary) defaults. -- Don

I'd like to do away with UIO altogether though. pumpkingod:

The main issue seems to be that although the semantics of UIO may be arbitrary, Wallace's patch actually broke deserialization for any production-based UArr, and I'm not sure the benefits are worthwhile (loading a file someone else sent you) given that endianness is already not taken into account when loading (so the chances of someone giving you a raw binary file that happens to contain values of the correct endianness is rather low, it seems).

On a related note, having a (more) complete testsuite might help prevent patches like this from breaking existing functionality.

Maybe a way to resolve the issue is to use the file size and break it according to the size of the two halves of the production? This avoids prefixing array length, but allows the productions to still work.

Either way, I'm in the process of writing a Binary instance, so maybe we can get the best of both worlds eventually.

Thanks, Dan

On Fri, Mar 13, 2009 at 7:21 PM, Don Stewart wrote:

...

manlio_perillo:

...

Don Stewart ha scritto:

...

[...]

...

So, the patch is: "just revert this change".

Or... use your own UIO instance. That's why it's a type class!

Why should I rewrite the UIO instance, if one already exists?

Oh, because you want different serialization semantics to the (arbitrary) defaults.

-- Don _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Daniel Peebles ha scritto:

I have added UAProd-based Binary instances to my copy of the uvector repo at http://patch-tag.com/publicrepos/pumpkin-uvector . I have some extensive tests (added to the existing tests directory) of things in there and they seem to be mostly sane.

Thanks for the work. I have a question, however. Isn't it possible to write binary serialization code for the generic Stream type? Manlio

The main issue seems to be that although the semantics of UIO may be arbitrary, Wallace's patch actually broke deserialization for any production-based UArr, and I'm not sure the benefits are worthwhile (loading a file someone else sent you) given that endianness is already not taken into account when loading (so the chances of someone giving you a raw binary file that happens to contain values of the correct endianness is rather low, it seems).

In my experience, having written several libraries in Haskell for serialisation and deserialisation, it is highly problematic when a library writer decides that all data to be stored began its life in Haskell, and is only being serialised in order to be read back in again by the same Haskell library. I have already made that mistake myself in two different libraries now, eventually regretting it (and fixing it). The real utility of serialisation is when it is possible to read data from any arbitrary external source, and to write data according to external standards. A library that can only read and write data in its own idiosyncratic format is not production-ready at all. This is why I submitted the patch that enables the uvector library to read raw binary data that was not produced by itself. I had 300Gb of data from an external source that I needed to deal with efficiently, and uvector was the ideal candidate apart from this small design flaw. And yes, my code also had to deal with endianness conversion on this data. Regards, Malcolm

Am Freitag, 13. März 2009 23:53 schrieb Don Stewart:

manlio_perillo:

...

Daniel Fischer ha scritto:

...

[...] Worked with uvector-0.1.0.1:

[...] But not with uvector-0.2

[...]

The main difference is that in uvector 0.2, hPutBU does not write in the file the length of the array; hGetBU simply use the file size.

let elemSize = sizeBU 1 (undefined :: e) n <- fmap ((`div`elemSize) . fromInteger) $ hFileSize h

So, the problem seems to be here. This simply don't support having two arrays written in the same file, and sizeBU belongs to the UAE class, whose instances are only declared for builtin types.

So, the patch is: "just revert this change".

Or... use your own UIO instance. That's why it's a type class!

Anyway, for the background on this:

Tue Nov 18 08:44:46 PST 2008 Malcolm Wallace * Use hFileSize to determine arraysize, rather than encoding it in the file.

"Here is a patch to the uvector library that fixes hGetBU and hPutBU to use the filesize to determine arraysize, rather than encoding it within the file. I guess the disadvantages are that now only one array can live in a file, and the given Handle must indeed be a file, not a socket Handle. But the advantage is that one can read packed raw datafiles obtained externally."

Still, again, I'd point out that uvector is alpha, APIs can and will change.

-- Don

Though I don't really know whether what I did is sane, I can offer a few patches which seem to work. Check for sanity before applying :) hunk ./Data/Array/Vector/Prim/BUArr.hs 85 - hPutBU, hGetBU + hPutBU, hGetBU, hGetLengthBU hunk ./Data/Array/Vector/Prim/BUArr.hs 864 +hGetLengthBU :: forall e. UAE e => Int -> Handle -> IO (BUArr e) +hGetLengthBU numEntries h = + do + marr@(MBUArr _ marr#) <- stToIO (newMBU numEntries) + let bytes = sizeBU numEntries (undefined :: e) + wantReadableHandle "hGetBU" h $ + \handle@Handle__{ haFD=fd, haBuffer=ref, haIsStream=is_stream } -> do + buf@Buffer { bufBuf = raw, bufWPtr = w, bufRPtr = r } <- readIORef ref + let copied = bytes `min` (w - r) + remaining = bytes - copied + newr = r + copied + newbuf | newr == w = buf{ bufRPtr = 0, bufWPtr = 0 } + | otherwise = buf{ bufRPtr = newr } + --memcpy_ba_baoff marr# raw (fromIntegral r) (fromIntegral copied) + memcpy_ba_baoff marr# raw (fromIntegral r) (fromIntegral copied) + writeIORef ref newbuf + readChunkBU fd is_stream marr# copied remaining + stToIO (unsafeFreezeAllMBU marr) + hunk ./Data/Array/Vector/UArr.hs 59 - BUArr, MBUArr, UAE, - lengthBU, indexBU, sliceBU, hGetBU, hPutBU, + BUArr, MBUArr, UAE(..), + lengthBU, indexBU, sliceBU, hGetBU, hGetLengthBU, hPutBU, hunk ./Data/Array/Vector/UArr.hs 867 + hGetLengthU :: Int -> Handle -> IO (UArr a) hunk ./Data/Array/Vector/UArr.hs 875 +primGetLengthU :: UPrim a => Int -> Handle -> IO (UArr a) +primGetLengthU n = liftM mkUAPrim . hGetLengthBU n + hunk ./Data/Array/Vector/UArr.hs 880 -instance UIO Bool where hPutU = primPutU; hGetU = primGetU -instance UIO Char where hPutU = primPutU; hGetU = primGetU -instance UIO Int where hPutU = primPutU; hGetU = primGetU -instance UIO Word where hPutU = primPutU; hGetU = primGetU -instance UIO Float where hPutU = primPutU; hGetU = primGetU -instance UIO Double where hPutU = primPutU; hGetU = primGetU +instance UIO Bool where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Char where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Int where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Word where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Float where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Double where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU hunk ./Data/Array/Vector/UArr.hs 887 -instance UIO Word8 where hPutU = primPutU; hGetU = primGetU -instance UIO Word16 where hPutU = primPutU; hGetU = primGetU -instance UIO Word32 where hPutU = primPutU; hGetU = primGetU -instance UIO Word64 where hPutU = primPutU; hGetU = primGetU +instance UIO Word8 where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Word16 where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Word32 where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Word64 where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU hunk ./Data/Array/Vector/UArr.hs 892 -instance UIO Int8 where hPutU = primPutU; hGetU = primGetU -instance UIO Int16 where hPutU = primPutU; hGetU = primGetU -instance UIO Int32 where hPutU = primPutU; hGetU = primGetU -instance UIO Int64 where hPutU = primPutU; hGetU = primGetU +instance UIO Int8 where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Int16 where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Int32 where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU +instance UIO Int64 where hPutU = primPutU; hGetU = primGetU; hGetLengthU = primGetLengthU hunk ./Data/Array/Vector/UArr.hs 899 -instance (UIO a, UIO b) => UIO (a :*: b) where +instance (UAE a, UAE b, UIO a, UIO b) => UIO (a :*: b) where hunk ./Data/Array/Vector/UArr.hs 902 - hGetU h = do xs <- hGetU h - ys <- hGetU h + hGetU h = do let elemSize = sizeBU 1 (undefined :: a) + sizeBU 1 (undefined :: b) + n <- fmap ((`div` elemSize) . fromInteger) $ hFileSize h + xs <- hGetLengthU n h + ys <- hGetLengthU n h + return (UAProd xs ys) + hGetLengthU n h = do xs <- hGetLengthU n h + ys <- hGetLengthU n h hunk ./Data/Array/Vector/UArr.hs 914 -instance (RealFloat a, UIO a) => UIO (Complex a) where +instance (RealFloat a, UAE a, UIO a) => UIO (Complex a) where hunk ./Data/Array/Vector/UArr.hs 918 + hGetLengthU n h = do arr <- hGetLengthU n h + return (UAComplex arr) hunk ./Data/Array/Vector/UArr.hs 921 -instance (Integral a, UIO a) => UIO (Ratio a) where +instance (Integral a, UAE a, UIO a) => UIO (Ratio a) where hunk ./Data/Array/Vector/UArr.hs 925 + hGetLengthU n h = do arr <- hGetLengthU n h + return (UARatio arr)