On Sun, May 21, 2006 at 01:20:54PM -0700, Jeremy Shaw wrote:

Cabal Question:

The streams library uses cpphs to do some preprocessing. There is a hugs specific Makefile that invokes cpphs with the command-line:

cpphs --noline -D__HUGS__ -D__HUGS_VERSION__=2005 -DSIZEOF_HSINT=4 -DSIZEOF_HSWORD=4

I tried adding the -D stuff to the cc-includes section of the .cabal file, but that caused the ghc6 build to start failing. Is there an easy way to specify flags that should only be used with cpphs and only when building hugs? Or is this one of those cases when I need to use some of the fancy hook features of cabal?

Cabal already adds -D__HUGS__ when building for Hugs, __HUGS_VERSION__ isn't used, and the SIZEOF's aren't universally valid. For GHC, the package gets them from MachDeps.h (an undocumented interface). Doing it portably probably requires autoconfery. Apart from that, the main in Setup.lhs could just be defaultMain.

Hello Jeremy, Monday, May 22, 2006, 12:20:54 AM, you wrote:

For my own needs, I cabalized and debianized the Streams library. It generates binary debs for ghc6 and hugs -- but I think the hugs version is broken. In any case, it is a start, you can download the packaging at:

http://www.n-heptane.com/nhlab/tmp/Streams_packaging.tar.gz

can i include your work in the library itself? is it better to include 'debian' directory to my archive or left this to the debian packagers? can you say how you are use my library? it's both interesting for me and can be helpful in deciding how it should be further developed -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

...

can i include your work in the library itself? Absolutely.

Hello Jeremy, Sunday, May 28, 2006, 1:29:02 AM, you wrote: thanks

...

is it better to include 'debian' directory to my archive or left this to the debian packagers?

If someone volunteers to maintain the package -- then it is probably

afaiu, you say about maintaining their debian ports? it seems that i should include these files now and omit them when someone will start to maintain the package?

would be useful to provide the debian directory (with a disclaimer) so

what disclaimer?

To get them into BDB I need to convert the haskell data structure into a C structure that looks like this:

struct __db_dbt { void *data; /* Key/data */ u_int32_t size; /* key/data length */ };

Currently I am doing it like this -- but this will clearly fail if the serialized data structure is longer than 512 bytes...

withDBT :: (Binary a) => a -> (Ptr DBT -> IO b) -> IO b withDBT thedata f = allocaBytes #{size DBT} $ \dbtPtr -> allocaBytes 512 $ \dataPtr -> do h <- openMemBuf dataPtr 512 withByteAlignedLE h $ flip put_ thedata wrote <- vTell h vClose h #{poke DBT, data} dbtPtr (castPtr dataPtr) #{poke DBT, size} dbtPtr ((fromIntegral wrote) :: Int) f dbtPtr

i will prefer to split it into two parts. and, DBD-interfacing part can be also implemented using binary i/o: withDBT00 :: Ptr a -> Int -> (Ptr DBT -> IO b) -> IO b withDBT00 f buf size = do h <- createMemBuf 20 >>= openByteAlignedLE put_ h buf putWord32 h size vRewind h (dbt,_) <- vReceiveBuf h result <- f dbt vClose h return result withDBT f thedata = encodeMemBufLE (withDBT00 f) thedata

I don't really need the file-system interface for this project -- what would be nice is something like 'withCStringLen' and 'peekCString' for the encode/decode functions:

type PtrLen a = (Ptr a, Int) encodePtrLen :: (Binary a) => a -> (PtrLen a -> IO b) -> IO b decodePtr :: (Binary a) => Ptr a -> IO a

encodeMemBufLE f thedata = do h <- createMemBuf 512 >>= openByteAlignedLE put_ h thedata vRewind h (buf,size) <- vReceiveBuf h result <- f buf size vClose h return result decodeMemBufLE buf size = do h <- openMemBuf buf size >>= openByteAlignedLE result <- get h vClose h return result but it will work only with Streams 0.1. you have spotted the problem that there is no official way to get access to the whole buffer contents if buffer was created with createMemBuf.

I could simulate this by using 'encode' to convert the data structure to a String and then use 'withCStringLen' to get the pointer and length -- but having the intermediate String seems like it could be a big performance hit.

Strings are slow by itself and moreover 'encode' has O(n^2) complexity

Two alternative ideas are:

(1) accurately pre-calculate the size of the serialized structure and allocate the correct amount of memory from the start

it's good idea to have 'binarySize :: Binary a => a->Int' function, although using it will halve the speed, so for you it's not the best solution

(2) start with a 'guess' and realloc the memory if the initial guess is too small.

createMemBuf does exactly this :) it's for why the whole Streams part exist. actually i have started with trivial instance ByteStream Handle where vPutByte h n = do hPutChar h (chr (fromEnum n)) vGetByte h = do c <- hGetChar h return $! (toEnum (ord c)) and only after Binary part was enough matured, i goes to adding all those fancy Stream features

I have not looked at the library exhaustively, so if there is already a good way to do this, let me know.

the way exist but it's not guaranteed by library interface and uses my knowledge of library internals. i will add interface which guarantees access to full buffer's contents. after that, 'encodeMemBufLE' can be written using official library capabilities. i will also add encodeMemBuf*/decodeMemBuf* to the lib, although i'm not sure that these functions are universal enough -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Duncan, Sunday, May 28, 2006, 3:05:53 PM, you wrote:

...

createMemBuf does exactly this :)

One of the areas where we found that Data.ByteString.Lazy was performing better than the ordinary Data.ByteString is cases like this where we do not know beforehand how big the buffer will be.

i like your idea of using ByteString.Lazy to implement fast and easy-to-use i/o, although i don't think that speed will be in 10% of C :) ghc by itself generates code that is several times slower than gcc-generated and you can't do anything agaist this, except for implementing everything in C. i've wriiten about this in ghc-users list in February. for example, simple "a[i]=b[i]+c[i]" floating-point loop works 20 times slower but, nevertheless, i think that this is a great idea - much faster than String-based hGetContents. it should help in numerous programs that need fast-and-dirty text processing, although it needs further development of library in order to implement for LazyByteString full String-like interface

If you have to use a single contiguous buffer then it involves guessing and possible reallocation. With a 'chunked' representation like ByteString.Lazy it's not a problem as we just allocate another chunk and start to fill that.

Obvious example include concat and getContents.

Would the same make sense for a MemBuf stream? Why does it need to be a single large buffer? Couldn't it be a list of buffers?

i also had this idea and it can be implemented in 1 day, i think (when someone will need this). but this is not for Jeremy, he need a contiguous buffer for interfacing with DBD. btw, it's better to use UArray instead of list -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Bulat, Sunday, May 28, 2006, 2:44:37 PM, you wrote:

...

type PtrLen a = (Ptr a, Int) encodePtrLen :: (Binary a) => a -> (PtrLen a -> IO b) -> IO b decodePtr :: (Binary a) => Ptr a -> IO a

Finally i've implemented the following (you then would use 'withForeignPtr' to work with contents of ForeignPtr): -- ----------------------------------------------------------------------------- -- Encode/decode contents of memory buffer encodePtr :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrLE :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrBitAligned :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrBitAlignedLE :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtr = encodePtr' openByteAligned encodePtrLE = encodePtr' openByteAlignedLE encodePtrBitAligned = encodePtr' openBitAligned encodePtrBitAlignedLE = encodePtr' openBitAlignedLE decodePtr :: (Binary a, Integral size) => Ptr x -> size -> IO a decodePtrLE :: (Binary a, Integral size) => Ptr x -> size -> IO a decodePtrBitAligned :: (Binary a, Integral size) => Ptr x -> size -> IO a decodePtrBitAlignedLE :: (Binary a, Integral size) => Ptr x -> size -> IO a decodePtr = decodePtr' openByteAligned decodePtrLE = decodePtr' openByteAlignedLE decodePtrBitAligned = decodePtr' openBitAligned decodePtrBitAlignedLE = decodePtr' openBitAlignedLE -- Universal function what encodes data with any alignment encodePtr' open thedata = do h <- createMemBuf 512 >>= open put_ h thedata vFlush h vRewind h (buf,size) <- vReceiveBuf h READING -- FIXME: MemBuf-implementation specific fptr <- newForeignPtr finalizerFree (castPtr buf) -- FIXME: also MemBuf-implementation specific return (fptr,size) -- Universal function what decodes data written with any alignment decodePtr' open ptr size = do h <- openMemBuf ptr size >>= open result <- get h vClose h return result -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Jacques, Wednesday, May 31, 2006, 5:33:39 PM, you wrote:

...

decodePtrBitAlignedLE :: (Binary a, Integral size) => Ptr x -> size -> IO a

Am I the only one who finds this encoding-of-types in the _name_ of a function quite distateful? There is no type safety being enforced here,

can you please write code you suggested? i'm not sure that type "a" should be encoded only to area pointed by "Ptr a" - binary encoding of value and it's memory representation are different concepts, although they are similar at the look -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Jacques, Wednesday, May 31, 2006, 5:33:39 PM, you wrote:

...

encodePtr :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrLE :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrBitAligned :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size) encodePtrBitAlignedLE :: (Binary a, Integral size) => a -> IO (ForeignPtr x, size)

Am I the only one who finds this encoding-of-types in the _name_ of a function quite distateful? There is no type safety being enforced here, no ensuring one will not be encoding a Ptr one way and decoding it another. Why not use Haskell's type system to help you there?

i misunderatood you when i wrote previous message. now that i can say: you are right. but on practice this means more text typing and coercion. especially when we go to ForeignPtrs. moreover, in most cases, imho, data encoded by 'encodePtr*', will go to the FFI libraries, so we can't use typechecking anyway i'm not against your idea, you absolutely right that this will be more Haskell way, but can this be implemented without additional complications for library users? -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Jacques, Wednesday, May 31, 2006, 8:07:29 PM, you wrote:

I am completely biased in this regard: I have spent several years maintaining ~800Kloc of legacy dynamically typed [commercial] code. A lot of this code has a life-span of roughly 20 years [ie once written, it takes an average of 20 years before it gets re-written]. That experience has converted me to a static-type fan, as well as a fan of designs that are for the "long term"; short-term convenience is something that is great for short-lived code (< 5 years is short-lived to me ;-) ).

my own programming experience say the same - strict typing significantly simplify program writing by ensuring it's correctness. and Haskell catch many problems as early as i compile code. but in this case we will add more complexity for standard use of functions (when just Ptr required) without any improvements in reliability just to catch potential problems with unusual usage. moreover, there are also encode/encodeLE/... functions that produce String - they also don't need any special String types why i include encoding type in function name? just to simplify usage, all the 'encodePtr*' functions can be expressed via one encodePtrWith, but i don't think that many peoples want to write this himself: encodePtr = encodePtrWith put_ encodePtrLE = encodePtrLEWith put_ encodePtrBitAligned = encodePtrBitAlignedWith put_ encodePtrBitAlignedLE = encodePtrBitAlignedLEWith put_ encodePtrLEWith write = encodePtrWith (\s a-> withByteAlignedLE s (`write` a)) encodePtrBitAlignedWith write = encodePtrWith (\s a-> withBitAligned s (`write` a)) encodePtrBitAlignedLEWith write = encodePtrWith (\s a-> withBitAlignedLE s (`write` a)) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

I have no problems with marshalling/unmarshalling (and even with the implicit casting going on). What I dislike is having a bunch of functions which are "the same" but with different names, where the difference boils down to enumerated types that end up being encoded in the function name. Regardless of type safety. Jacques Tim Newsham wrote:

...

Am I the only one who finds this encoding-of-types in the _name_ of a function quite distateful? There is no type safety being enforced here, no ensuring one will not be encoding a Ptr one way and decoding it another. Why not use Haskell's type system to help you there?

When marshalling data you often don't want any type safety. You often want to explicitely linearize data from one type and then unlinearize it into another type. The net result is that of casting. In fact, you can write a marshalling library with an interface based entirely on this concept:

http://www.lava.net/~newsham/x/Pkts5.lhs

The interface is, in essense, a glorified casting mechanism. To marshall data you convert it to an array of bytes and to unmarshall data you unconvert it.

...

Jacques

Tim Newsham http://www.lava.net/~newsham/

Hello Donald, Monday, May 22, 2006, 4:19:59 AM, you wrote:

...

i foresee that Streams + Fast Packed Strings together will yield a breakthrough in GHC I/O speed, and this can be implemented even without waiting for GHC 6.6

Before reading this I had thought it might be an XOR situation, but now it seems a happy coexistence may be possible. Are there any preliminary results on how these might work together, and the potential speedups?

I imagine that Bulat could easily write some IO operations that construct ByteStrings as results. So that should be very possible.

yes, it is no problem. i tried this with other packed string libraries, but at those attempts results was bad because of inefficiency in my library organization. now i've changed these internals according to one Simon's idea, and i think that ByteString I/O will be done with the same speed as low-level character i/o, i.e. about 100-200 mb/sec on modern cpus (faster than disk I/O by itself :) ) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com