Thomas Schilling wrote:

On 15 jun 2007, at 18.13, Jim Burton wrote:

...

import qualified Data.ByteString.Char8 as B

Have you tried

import qualified Data.ByteString.Lazy.Char8 as B

?

No -- I'll give it a try and compare them. Is laziness preferable here? Thanks,

On 15 jun 2007, at 21.14, Jim Burton wrote:

Thomas Schilling wrote:

...

On 15 jun 2007, at 18.13, Jim Burton wrote:

...

import qualified Data.ByteString.Char8 as B

Have you tried import qualified Data.ByteString.Lazy.Char8 as B ?

No -- I'll give it a try and compare them. Is laziness preferable here?

yes, since you were talking of big files. if you don't have to keep the data around lazy bytestrings will keep the memory footprint low.

Sebastian Sylvan wrote:

On 15/06/07, Jim Burton wrote: [snip] Hi, Hi Sebastian, I haven't compiled this, but you get the general idea:

import qualified Data.ByteString.Lazy.Char8 as B -- takes a bytestring representing the file, concats the lines -- then splits it up into "real" lines using the delimiter clean :: Char -> B.ByteString -> [B.ByteString] clean' d = B.split d . B.concat . B.lines

I think that would only work if there was one column per line...I didn't make it clear that as well as being comma separated, the delimiter is around each column, of which there are several on a line so if the delimiter is ~ a file might look like: ~sdlkfj~, ~dsdkjf~ #eo row1 ~sdf dfkj~, ~dfsd~ #eo row 2

Jason Dagit wrote: [snip]

I love to see people using Haskell, especially professionally, but I have to wonder if the real tool for this job is sed? :-)

Jason

Maybe it is -- I've never used sed. (cue oohs and ahhs from the gallery?) But from the (unquantified) gains so far haskell may certainly be enough of an improvement to fit the bill even though I'd be interested in anything that improved on it further still.

"Brandon S. Allbery KF8NH" writes:

On Jun 15, 2007, at 18:37 , Jason Dagit wrote:

...

I love to see people using Haskell, especially professionally, but I have to wonder if the real tool for this job is sed? :-)

Actually, while sed could do that, it'd be a nightmare. You really want a parser to deal with general CSV like this, and while you can write parsers in sed, you *really* don't want to. :)

sed ':a /,$/!{N;s/\n//;ba}' somefile.csv

On 15/06/07, Jim Burton wrote:

Sebastian Sylvan wrote:

...

On 15/06/07, Jim Burton wrote: [snip] Hi, Hi Sebastian, I haven't compiled this, but you get the general idea:

import qualified Data.ByteString.Lazy.Char8 as B -- takes a bytestring representing the file, concats the lines -- then splits it up into "real" lines using the delimiter clean :: Char -> B.ByteString -> [B.ByteString] clean' d = B.split d . B.concat . B.lines

I think that would only work if there was one column per line...I didn't make it clear that as well as being comma separated, the delimiter is around each column, of which there are several on a line so if the delimiter is ~ a file might look like:

~sdlkfj~, ~dsdkjf~ #eo row1 ~sdf dfkj~, ~dfsd~ #eo row 2

A sorry, I thought the delimiter was a line delimiter. I'm trying to get to that fusion goodness by using built-in functions as much as possible... How about this one: clean del = B.map ( B.filter (/='\n') ) . B.groupBy (\x y -> (x,y) /= (del,'\n')) That groupBy will group it into groups which don't have the delimiter followed by a newline in them (which is the sequence your rows end with), then it filters out newlines in each row. You might want to filter out spaces first (if there are any) so that you don't get a space between the delimiter and newline at the end... -- Sebastian Sylvan +44(0)7857-300802 UIN: 44640862

On 16/06/07, Jim Burton wrote:

Sebastian Sylvan wrote:

...

A sorry, I thought the delimiter was a line delimiter. I'm trying to get

...

that fusion goodness by using built-in functions as much as possible...

How about this one:

clean del = B.map ( B.filter (/='\n') ) . B.groupBy (\x y -> (x,y) /= (del,'\n'))

That groupBy will group it into groups which don't have the delimiter followed by a newline in them (which is the sequence your rows end with), then it filters out newlines in each row. You might want to filter out spaces first (if there are any) so that you don't get a space between

to the

...

delimiter and newline at the end...

I think you still need unlines after that so is the time complexity different to the

unlines . foldr (function including `last') . lines

in my first post? Or is it better for another reason, such as "fusion goodness"?

Benchmark it I guess :-) Both versions use a non-bytestring recursive functions (the outer B.mapshould just be a straight map, and yours use a foldr), which may mess fusion up... Not sure what would happe here... I don't have a Haskell compiler at this computer so I can't try anything out... -- Sebastian Sylvan +44(0)7857-300802 UIN: 44640862

On Fri, Jun 15, 2007 at 11:31:36PM +0100, Jim Burton wrote:

I think that would only work if there was one column per line...I didn't make it clear that as well as being comma separated, the delimiter is around each column, of which there are several on a line so if the delimiter is ~ a file might look like:

~sdlkfj~, ~dsdkjf~ #eo row1 ~sdf dfkj~, ~dfsd~ #eo row 2

It would be easier to experiment if you could provide us with an example input file. If you are worried about revealing sensitive information, you can change all characters other then newline, ~ and , to "A"s, for example. An accompanying output file, for checking correctness, would be even nicer. Best regards Tomek

On Sat, Jun 16, 2007 at 12:08:22PM +0100, Jim Burton wrote:

Tomasz Zielonka wrote:

...

It would be easier to experiment if you could provide us with an example input file. If you are worried about revealing sensitive information, you can change all characters other then newline, ~ and , to "A"s, for example. An accompanying output file, for checking correctness, would be even nicer.

Hi Tomasz, I can do that but they do essentially look like the example above, except with 10 - 30 columns, more data in each column, and more rows, maybe this side of a million. They are produced by an Oracle export which escapes the delimiter (often a tilde) from within the cols. The output file should have exactly one row per line, with extra newlines replaced by a string given as a param (it might be a space or a html tag -- I only just remembered this and my initial effort doesn't do it).

I guess you've tried to convince Oracle to produce the right format in the first place, so there would be no need for post-processing...? I wonder what would you get if you set the delimiter to be a newline ;-) Best regards Tomek