ByteString.getContents fails for files >2GB on OS X - Haskell-Cafe - Haskell.org

newer
The use of continuation monad in...

ByteString.getContents fails for files >2GB on OS X

older
Cannot System,Timeout.timeout...

Shaun Jackman

7 Jun 2012 7 Jun '12

8:25 p.m.

Hi, Data.ByteString.Char8.getContents fails for files >2GB on OS X. Is there a fix for this? $ cat getContents.hs main = getContents $ ./getContents

Reply

Sign in to reply online Use email software

Show replies by date

Erik Hesselink

8 Jun 8 Jun

4:29 a.m.

New subject: ByteString.getContents fails for files >2GB on OS X

Do you have a 32bit or 64bit GHC build? That might have something to do with it, if you're nearing 2^32 (or 2^31) bytes. Erik On Fri, Jun 8, 2012 at 2:25 AM, Shaun Jackman wrote:

Hi,

Data.ByteString.Char8.getContents fails for files >2GB on OS X. Is there a fix for this?

$ cat getContents.hs main = getContents $ ./getContents
Mac OS X 10.7.4 64-bit

As a workaround, I used ByteString.Lazy instead of the strict ByteString, which worked, but found it was ~4 times slower for my program, so I'd like to get the strict ByteString working with large files.

Cheers, Shaun

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Reply

Sign in to reply online Use email software

Serge Le Huitouze

8:08 a.m.

New subject: ByteString.getContents fails for files >2GB on OS X

Isn't it more likely to be due to the garbage collector's strategy (copying) ? --Serge On Fri, Jun 8, 2012 at 10:29 AM, Erik Hesselink wrote:

Do you have a 32bit or 64bit GHC build? That might have something to do with it, if you're nearing 2^32 (or 2^31) bytes.

Erik

On Fri, Jun 8, 2012 at 2:25 AM, Shaun Jackman wrote:

...
Hi,

Data.ByteString.Char8.getContents fails for files >2GB on OS X. Is there a fix for this?

$ cat getContents.hs main = getContents $ ./getContents
Mac OS X 10.7.4 64-bit

As a workaround, I used ByteString.Lazy instead of the strict ByteString, which worked, but found it was ~4 times slower for my program, so I'd like to get the strict ByteString working with large files.

Cheers, Shaun

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Reply

Sign in to reply online Use email software

Shaun Jackman

2:23 p.m.

New subject: ByteString.getContents fails for files >2GB on OS X

Hi Erik, Serge, I have a 64-bit build of GHC: http://www.haskell.org/ghc/dist/7.4.1/ghc-7.4.1-x86_64-apple-darwin.tar.bz2 I think it's fundamentally an OS X issue. The system call read(2) fails for reads >2 GB with EINVAL, even though I have a 64-bit OS X kernel. GHC would need to hack around this issue. Cheers, Shaun On 8 June 2012 05:08, Serge Le Huitouze wrote:

Isn't it more likely to be due to the garbage collector's strategy (copying) ?

--Serge

On Fri, Jun 8, 2012 at 10:29 AM, Erik Hesselink wrote:

...
Do you have a 32bit or 64bit GHC build? That might have something to do with it, if you're nearing 2^32 (or 2^31) bytes.

Erik

On Fri, Jun 8, 2012 at 2:25 AM, Shaun Jackman wrote:

...
Hi,

Data.ByteString.Char8.getContents fails for files >2GB on OS X. Is there a fix for this?

$ cat getContents.hs main = getContents $ ./getContents
Mac OS X 10.7.4 64-bit

As a workaround, I used ByteString.Lazy instead of the strict ByteString, which worked, but found it was ~4 times slower for my program, so I'd like to get the strict ByteString working with large files.

Cheers, Shaun

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Reply

Sign in to reply online Use email software

Gregory Collins

2:56 p.m.

New subject: ByteString.getContents fails for files >2GB on OS X

Try http://hackage.haskell.org/package/bytestring-mmap ? G On Fri, Jun 8, 2012 at 8:23 PM, Shaun Jackman wrote:

Hi Erik, Serge,

I have a 64-bit build of GHC: http://www.haskell.org/ghc/dist/7.4.1/ghc-7.4.1-x86_64-apple-darwin.tar.bz2

I think it's fundamentally an OS X issue. The system call read(2) fails for reads >2 GB with EINVAL, even though I have a 64-bit OS X kernel. GHC would need to hack around this issue.

Cheers, Shaun

On 8 June 2012 05:08, Serge Le Huitouze wrote:

...
Isn't it more likely to be due to the garbage collector's strategy (copying) ?

--Serge

On Fri, Jun 8, 2012 at 10:29 AM, Erik Hesselink wrote:

...
Do you have a 32bit or 64bit GHC build? That might have something to do with it, if you're nearing 2^32 (or 2^31) bytes.

Erik

On Fri, Jun 8, 2012 at 2:25 AM, Shaun Jackman wrote:

...
Hi,

Data.ByteString.Char8.getContents fails for files >2GB on OS X. Is there a fix for this?

$ cat getContents.hs main = getContents $ ./getContents
Mac OS X 10.7.4 64-bit

As a workaround, I used ByteString.Lazy instead of the strict ByteString, which worked, but found it was ~4 times slower for my program, so I'd like to get the strict ByteString working with large files.

Cheers, Shaun

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Gregory Collins

Reply

Sign in to reply online Use email software

Gracjan Polak

11 Jun 11 Jun

10:08 a.m.

New subject: [Haskell-cafe] ByteString.getContents fails for files >2GB on OS X

Gregory Collins writes:

Try http://hackage.haskell.org/package/bytestring-mmap ?

Or: http://hackage.haskell.org/package/mmap -- Gracjan

Reply

Sign in to reply online Use email software

Shaun Jackman

21 Jun 21 Jun

7:55 p.m.

New subject: ByteString.getContents fails for files >2GB on OS X

System.IO.MMap (mmapFileByteString) worked like a charm in loading files larger than 2 GB on OS X. Using mmapFileByteString and strict ByteString is roughly seven times faster for my program than using getContents and ByteString.Lazy. Cheers, Shaun On 11 June 2012 07:08, Gracjan Polak wrote:

Gregory Collins writes:

...
Try http://hackage.haskell.org/package/bytestring-mmap ?

Or:

http://hackage.haskell.org/package/mmap

-- Gracjan

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Reply

Sign in to reply online Use email software

4719

Age (days ago)

4732

Last active (days ago)

Download

6 comments

5 participants

tags

participants (5)

Erik Hesselink
Gracjan Polak
Gregory Collins
Serge Le Huitouze
Shaun Jackman