
There seems to be a 2Gb file limit when using GHC, I don't suppose there's any way to get around it, apart from manually splitting the contents over several smaller files? (I'm even unable to use writeFile on a file that's larger than 2Gb)
That's probably a bug, as long as the underlying OS uses 64-bit file offsets. How does the 2Gb limit manifest itself? Cheers, Simon

"Simon Marlow"
That's probably a bug, as long as the underlying OS uses 64-bit file offsets. How does the 2Gb limit manifest itself?
My program dumps a bunch of data to a file. File grows, I leave, next morning, I'm greeted with: % time ./xsact -k 25 -n 64 -L -p 3 -x ~/data/ug-250.seq > ug-250.L zsh: file size limit exceeded ./xsact -k 25 -n 64 -L -p 3 -x ~/data/ug-250.seq > ug-250.L ./xsact -k 25 -n 64 -L -p 3 -x ~/data/ug-250.seq > ug-250.L 2556.40s user 39.50s system 96% cpu 44:58.50 total I can still do % echo "foo" >> ug-250.L and the file grows a bit, without error. Here's a quick (except for the dd) case: % dd if=/dev/zero of=big_file bs=1M count=3000 % du -sh big_file 3.0G big_file % ghci Prelude> x <- readFile "big_file" *** Exception: permission denied Action: openFile Reason: File too large File: big_file Prelude> writeFile "big_file" "foo!" *** Exception: permission denied Action: openFile Reason: File too large File: big_file All standard Unix tools seem to work. RedHat 8.0, GHC 5.04.2 from RPMs. -kzm -- If I haven't seen further, it is by standing in the footprints of giants

"Simon Marlow"
That's probably a bug, as long as the underlying OS uses 64-bit file offsets. How does the 2Gb limit manifest itself?
On Fri, May 09, 2003 at 12:05:19PM +0200, Ketil Z. Malde wrote:
All standard Unix tools seem to work. RedHat 8.0, GHC 5.04.2 from RPMs.
/* * Called when an inode is about to be open. * We use this to disallow opening RW large files on 32bit systems if * the caller didn't specify O_LARGEFILE. On 64bit systems we force * on this flag in sys_open. */ int generic_file_open(struct inode * inode, struct file * filp) { if (!(filp->f_flags & O_LARGEFILE) && inode->i_size > MAX_NON_LFS) return -EFBIG; return 0; } i.e. the right open() etc. flags need to be passed. What does strace(1) say it's doing? -- wli

William Lee Irwin III
if (!(filp->f_flags & O_LARGEFILE) && inode->i_size > MAX_NON_LFS) return -EFBIG;
What does strace(1) say it's doing?
It says open("big_file", O_RDONLY|O_NONBLOCK|O_NOCTTY) = -1 EFBIG (File too large) So it sounds like this is right on the money. (Even if the comment you included is wrong, this applies with O_RDONLY, too) -kzm -- If I haven't seen further, it is by standing in the footprints of giants

William Lee Irwin III
if (!(filp->f_flags & O_LARGEFILE) && inode->i_size > MAX_NON_LFS) return -EFBIG; What does strace(1) say it's doing?
On Fri, May 09, 2003 at 03:26:25PM +0200, Ketil Z. Malde wrote:
It says open("big_file", O_RDONLY|O_NONBLOCK|O_NOCTTY) = -1 EFBIG (File too large) So it sounds like this is right on the money. (Even if the comment you included is wrong, this applies with O_RDONLY, too)
Oh, the comment is bogus. It's not checking in that function for O_RDONLY. I'll send in a patch. -- wli

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 man 2 open describes O_LARGEFILE as follows: O_LARGEFILE: On 32-bit systems that support the Large Files System, allow files whose sizes cannot be represented in 31 bits to be opened. I'm slightly puzzled about why you would need to specify this flag. That is, why would it not just go ahead and open the file anyway? My best guess is that you are promising that all indexes into the file will be 64 bit (unsigned?) ints and not a 32 bit unsigned int. In other words, it's a backward compatability feature with a little typechecking thrown in. - -- Alastair Reid -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQE+u7CebLKSuD/RJBMRAlD4AJ9nTVio+CoXquE9UtUAUKO5iAEbUQCeK9Ls j1W7OYAmlnMi8p+EdxFDTAo= =yjva -----END PGP SIGNATURE-----

On Fri, May 09, 2003 at 02:43:58PM +0100, Alastair Reid wrote:
man 2 open describes O_LARGEFILE as follows: O_LARGEFILE: On 32-bit systems that support the Large Files System, allow files whose sizes cannot be represented in 31 bits to be opened. I'm slightly puzzled about why you would need to specify this flag. That is, why would it not just go ahead and open the file anyway? My best guess is that you are promising that all indexes into the file will be 64 bit (unsigned?) ints and not a 32 bit unsigned int. In other words, it's a backward compatability feature with a little typechecking thrown in.
SuSv3 doesn't specify a difference between RW and RO. My first guess is unexpected nontermination could occur if something loops until getting some kind of sign it's passed EOF since offsets would wrap so they wanted to avoid that compatibility issue. Anyway, will the following kernel update help clarify things? I'll send it in to akpm et al if so. I've checked with a couple of people and it appears the code is what was intended, not the comment. -- wli ===== fs/open.c 1.38 vs edited ===== --- 1.38/fs/open.c Wed Apr 9 06:42:36 2003 +++ edited/fs/open.c Fri May 9 07:19:25 2003 @@ -902,7 +902,7 @@ /* * Called when an inode is about to be open. - * We use this to disallow opening RW large files on 32bit systems if + * We use this to disallow opening large files on 32bit systems if * the caller didn't specify O_LARGEFILE. On 64bit systems we force * on this flag in sys_open. */

ketil@ii.uib.no (Ketil Z. Malde) writes:
"Simon Marlow"
writes:
That's probably a bug, as long as the underlying OS uses 64-bit file offsets. How does the 2Gb limit manifest itself?
On Solaris, the file grows until 2Gb, then: Fail: permission denied Action: commitAndReleaseBuffer Handle: {loc=/Home/stip/ketil/data/ug-250.seq.tmp_M,type=writable,binary=False,buffering=block (1024)} Reason: File too large File: /Home/stip/ketil/data/ug-250.seq.tmp_M This file can be read from GHCI, but after I append "foo" to it and try it agin, I get: Prelude> x <- readFile "/Home/stip/ketil/data/ug-250.seq.tmp_M" *** Exception: failed Action: openFile Reason: Value too large for defined data type File: /Home/stip/ketil/data/ug-250.seq.tmp_M
From the manual pages to open(2):
O_LARGEFILE If set, the offset maximum in the open file descrip- tion is the largest value that can be represented correctly in an object of type off64_t. This is with SunOS 5.8. Seems like it behaves quite similar to Linux in this respect. -kzm -- If I haven't seen further, it is by standing in the footprints of giants
participants (4)
-
Alastair Reid
-
ketil@ii.uib.no
-
Simon Marlow
-
William Lee Irwin III