hi all, two things: 1/ i do something and i'd like to know if it is correct 2/ i propose something about 1/ 1/ i want to read some binary file (e.g. targa file format : *.tga). i do this : -- first way : via IOUArray showInfoHeader1 handle = do a <- newArray_ (1,8) :: IO (IOUArray Int Word8) hGetArray handle a 8 idLength <- readArray a 1 -- or getElems... putStrLn ("id length : " ++ show idLength) return () -- second way : via c-like array showInfoHeader2 handle = do b <- mallocArray 8 :: IO (Ptr Word8) hGetBuf handle b 8 [idLength] <- peekArray 1 b -- or peakArray 8 b putStrLn ("id length : " ++ show idLength) free b return () so, briefly, i have to read some content into some kind of buffer (IOUArray Int Word8 or Ptr Word8), then get one (or more) elements from the buffor into a standard haskell variable (is it the correct word ?) (or list). in the second case, i also have to free the buffer. in some case, when the data is more than one Word8 long, i have to 'reconstruct' it, i.e.: [x1,x2] <- getElems a let x = fromIntegral x1 + fromIntegral x2 * 256 :: Int is it the correct way to read binary files ? ------- 2/ haskell is (i heard that once ... :-) a high level language, so it has (must have) good support for abstraction... but in 1/, i have to choose between different kind of array representation (and i dont know which one is better) and it seems to me that the resulting code (compiled) would have to be the same. the thing i want to say here is : what i want to do is pretty obvious (in both code) but could be expressed more succintly and with only one possible syntax: for example, the couples (hGet*,peek/readArray) could be written in one line; also, one line for the reading/reconstructing more-than-one-Word8 value. is it already possible ? would it be interesting to add such capabilities to haskell ? (i think so) i can try to add it but i need some pointers about how to do it. thx a lot, vo minh thu
Hello minh, Wednesday, April 5, 2006, 10:41:02 PM, you wrote:
but in 1/, i have to choose between different kind of array representation (and i dont know which one is better) and it seems to me that the resulting code (compiled) would have to be the same.
no, the code will be slightly different. IOUArray will allocate space in the GHC's heap, while malloc - in the C heap (ghc's heap is additional storey on the C heap) btw, `getElems` is VERY INEEFECIENT way - it will convert entire array to the list before return
for example, the couples (hGet*,peek/readArray) could be written in one line; also, one line for the reading/reconstructing more-than-one-Word8 value.
is it already possible ? would it be interesting to add such capabilities to haskell ? (i think so) i can try to add it but i need some pointers about how to do it.
i don't see much problems here, just add peek16LE and other procedures like it and you can use trivial code: idLength <- peek8 a 1 x <- peek16LE a 8 peek8 a i = do (x::Word8) <- peekByteOff a i return (fromIntegral x) peek16LE a i = do (x::Word8) <- peekByteOff a i (y::Word8) <- peekByteOff a (i+1) return (fromIntegral x + fromIntegral y * 256 ) there are a couple of binary I/O libs (including my own one :) ), i just don't think you need such power here. of course, if you want to read data sequentially, binary i/o lib will be preferable. with my lib you can write smth like this: -- Create new MemBuf filled with data from file h <- readFromFile "test" -- Read header fields sequentially idLength <- getWord8 h x <- getWord16le h .... i attached here a part of my library docs where this described in much more details :) the lib itself is at http://freearc.narod.ru/Streams.tar.gz -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
thanks a lot ! the code you give in the mail is +/- what i thought about ... is-it the fastest way ? thank you also for your lib, i will read it later in the day. when i said "would have to be the same" in previous mail, it's because i like to see haskell as a really high-level abstraction where you just say what you want, but not in which way to do it. in the examples (mines or yours), the programmer has a lot of choices ... another problem is that you have to allocate a buffer before reading the file, but (i m not a os expert) i think there is already some kind of plumbing (maybe another buffer) to read file. so maybe that in all cases, we're losing efficiency when reading a file ? also, if our buffer allocation is mandatory, the compiler could put the right code for us (i.e. choose the best buffer length). again thx, minh thu 2006/4/6, Bulat Ziganshin <bulat.ziganshin@gmail.com>:
Hello minh,
Wednesday, April 5, 2006, 10:41:02 PM, you wrote:
but in 1/, i have to choose between different kind of array representation (and i dont know which one is better) and it seems to me that the resulting code (compiled) would have to be the same.
no, the code will be slightly different. IOUArray will allocate space in the GHC's heap, while malloc - in the C heap (ghc's heap is additional storey on the C heap)
btw, `getElems` is VERY INEEFECIENT way - it will convert entire array to the list before return
for example, the couples (hGet*,peek/readArray) could be written in one line; also, one line for the reading/reconstructing more-than-one-Word8 value.
is it already possible ? would it be interesting to add such capabilities to haskell ? (i think so) i can try to add it but i need some pointers about how to do it.
i don't see much problems here, just add peek16LE and other procedures like it and you can use trivial code:
idLength <- peek8 a 1 x <- peek16LE a 8
peek8 a i = do (x::Word8) <- peekByteOff a i return (fromIntegral x)
peek16LE a i = do (x::Word8) <- peekByteOff a i (y::Word8) <- peekByteOff a (i+1) return (fromIntegral x + fromIntegral y * 256 )
there are a couple of binary I/O libs (including my own one :) ), i just don't think you need such power here. of course, if you want to read data sequentially, binary i/o lib will be preferable. with my lib you can write smth like this:
-- Create new MemBuf filled with data from file h <- readFromFile "test" -- Read header fields sequentially idLength <- getWord8 h x <- getWord16le h ....
i attached here a part of my library docs where this described in much more details :)
the lib itself is at http://freearc.narod.ru/Streams.tar.gz
-- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
Hello minh, Thursday, April 6, 2006, 12:41:32 PM, you wrote:
the code you give in the mail is +/- what i thought about ... is-it the fastest way ?
if you will say about your task and speed requirements, i will say how you can do it. the fastest way is to use asm :)
thank you also for your lib, i will read it later in the day.
when i said "would have to be the same" in previous mail, it's because i like to see haskell as a really high-level abstraction where you just say what you want, but not in which way to do it. in the examples (mines or yours), the programmer has a lot of choices ...
it was on advertising :) in Real World (tm) the things are more complex ;) on the other side, you can write something like this (using my library): data TiffHeader = Tiff { id : Word8, height : Word16, width : Word16, .... } main = do h <- openBinaryFD "filename" ReadMode header <- get h print (height header, width header) and got what you want. here, all fields will be read as big-endian, to read them as little-endian you should use: main = do h <- openBinaryFD "filename" ReadMode >>= openByteAlignedLE
another problem is that you have to allocate a buffer before reading the file, but (i m not a os expert) i think there is already some kind of plumbing (maybe another buffer) to read file. so maybe that in all cases, we're losing efficiency when reading a file ? also, if our buffer allocation is mandatory, the compiler could put the right code for us (i.e. choose the best buffer length).
with my lib, you can either read data directly from file (through implicit 512-byte buffer) or read whole file into the automatically allocated buffer with `readFromFile`. what is better - depends on what you plan to do with rest of file
-- Create new MemBuf filled with data from file h <- readFromFile "test"
-- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
thanks a lot Bulat ! i ve seen the lib use System.Stream which i don't know... the fact you can discribe a layout then read a file according to the layout is just a feature i thought about. i'll definitely learn your library. minh thu 2006/4/6, Bulat Ziganshin <bulat.ziganshin@gmail.com>:
Hello minh,
Thursday, April 6, 2006, 12:41:32 PM, you wrote:
the code you give in the mail is +/- what i thought about ... is-it the fastest way ?
if you will say about your task and speed requirements, i will say how you can do it. the fastest way is to use asm :)
thank you also for your lib, i will read it later in the day.
when i said "would have to be the same" in previous mail, it's because i like to see haskell as a really high-level abstraction where you just say what you want, but not in which way to do it. in the examples (mines or yours), the programmer has a lot of choices ...
it was on advertising :) in Real World (tm) the things are more complex ;) on the other side, you can write something like this (using my library):
data TiffHeader = Tiff { id : Word8, height : Word16, width : Word16, .... }
main = do h <- openBinaryFD "filename" ReadMode header <- get h print (height header, width header)
and got what you want. here, all fields will be read as big-endian, to read them as little-endian you should use:
main = do h <- openBinaryFD "filename" ReadMode >>= openByteAlignedLE
another problem is that you have to allocate a buffer before reading the file, but (i m not a os expert) i think there is already some kind of plumbing (maybe another buffer) to read file. so maybe that in all cases, we're losing efficiency when reading a file ? also, if our buffer allocation is mandatory, the compiler could put the right code for us (i.e. choose the best buffer length).
with my lib, you can either read data directly from file (through implicit 512-byte buffer) or read whole file into the automatically allocated buffer with `readFromFile`. what is better - depends on what you plan to do with rest of file
-- Create new MemBuf filled with data from file h <- readFromFile "test"
-- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
participants (2)
-
Bulat Ziganshin -
minh thu