Issues with the large address space allocator and HPC systems with resource limits
Dear GHC devs, I hope this is the right forum to bring this up. I am the lead developer of NGLess (https://github.com/ngless-toolkit/ngless a bioinformatics tool, written in Haskell). Several users have complained about not being able to easily use NGLess in an academic cluster environment due to the fact that it allocates 1TB of address space (e.g., https://groups.google.com/forum/#!topic/ngless/9su2E0EdeCc and I have also gotten several private emails on this issue). In particular, many systems are set up with a limit on the address space so that if the job allocates more than the given limit, it is immediately killed. This appears to be the default way to set up SGE, the most widely used batch system. Users are dependent on their sysadmins and lack the permissions to change these settings easily (and may not always be cognizant of the difference between "allocating address space" and "allocating memory"). Using ulimit seem to make the issue disappear on most, but not all, user setups. I have now built NGLess with a version of GHC that was compiled without the large address allocator (using ./configure --disable-large-address-space). At least locally, this seems to run correctly and solve the issue. I assume that there are performance or other reasons to use the large address space allocator as the default, but, right now, for the problem space I am working in, disabling it seems to be a better trade-off. In principle, the RTS that is used for GHC and the one that is used for the programme being linked do not need to be the same. Is there any possibility of making this choice when a programme is linked and not when GHC is compiled? Thank you for all your effort! Luis -- Luis Pedro Coelho | Fudan University | http://luispedro.org PI of Big Data Biology Lab at Fudan University (start mid-2018) http://big-data-biology.org
Pre-allocation of large memory also causes trouble on Windows Subsystem for Linux. In that case also, compiling with --disable-large-address-space solves the problem. If your OS happens to be Ubuntu, there is an unofficial PPA for GHC which includes versions of GHC and cabal-install that were compiled without the pre-allocation, and which compile executables that do not do the pre-allocation. The PPA is here: https://launchpad.net/~hvr/+archive/ubuntu/ghc-wsl Yitz On Tue, Jul 3, 2018 at 11:29 PM, Luis Pedro Coelho <luispedro@big-data-biology.org> wrote:
Dear GHC devs,
I hope this is the right forum to bring this up.
I am the lead developer of NGLess (https://github.com/ngless-toolkit/ngless a bioinformatics tool, written in Haskell). Several users have complained about not being able to easily use NGLess in an academic cluster environment due to the fact that it allocates 1TB of address space (e.g., https://groups.google.com/forum/#!topic/ngless/9su2E0EdeCc and I have also gotten several private emails on this issue).
In particular, many systems are set up with a limit on the address space so that if the job allocates more than the given limit, it is immediately killed.
This appears to be the default way to set up SGE, the most widely used batch system. Users are dependent on their sysadmins and lack the permissions to change these settings easily (and may not always be cognizant of the difference between "allocating address space" and "allocating memory"). Using ulimit seem to make the issue disappear on most, but not all, user setups.
I have now built NGLess with a version of GHC that was compiled without the large address allocator (using ./configure --disable-large-address-space). At least locally, this seems to run correctly and solve the issue.
I assume that there are performance or other reasons to use the large address space allocator as the default, but, right now, for the problem space I am working in, disabling it seems to be a better trade-off. In principle, the RTS that is used for GHC and the one that is used for the programme being linked do not need to be the same. Is there any possibility of making this choice when a programme is linked and not when GHC is compiled?
Thank you for all your effort!
Luis
-- Luis Pedro Coelho | Fudan University | http://luispedro.org
PI of Big Data Biology Lab at Fudan University (start mid-2018) http://big-data-biology.org _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Luis Pedro Coelho <luispedro@big-data-biology.org> writes:
Dear GHC devs,
I hope this is the right forum to bring this up.
Absolutely, thanks for bringing this up.
I am the lead developer of NGLess (https://github.com/ngless-toolkit/ngless a bioinformatics tool, written in Haskell). Several users have complained about not being able to easily use NGLess in an academic cluster environment due to the fact that it allocates 1TB of address space (e.g., https://groups.google.com/forum/#!topic/ngless/9su2E0EdeCc and I have also gotten several private emails on this issue).
In particular, many systems are set up with a limit on the address space so that if the job allocates more than the given limit, it is immediately killed.
This appears to be the default way to set up SGE, the most widely used batch system. Users are dependent on their sysadmins and lack the permissions to change these settings easily (and may not always be cognizant of the difference between "allocating address space" and "allocating memory"). Using ulimit seem to make the issue disappear on most, but not all, user setups.
Indeed as someone who has had to run on a shared cluster in the past, I can sympathize. These sorts of restrictions are common and quite annoying. Are these address space limits advertised via getrlimit(2)? If so, have you tried GHC 8.6.1-alpha1? While fixing #14492 I taught GHC to respect rlimits when allocating its heap, so this might work now.
I have now built NGLess with a version of GHC that was compiled without the large address allocator (using ./configure --disable-large-address-space). At least locally, this seems to run correctly and solve the issue.
I assume that there are performance or other reasons to use the large address space allocator as the default, but, right now, for the problem space I am working in, disabling it seems to be a better trade-off. In principle, the RTS that is used for GHC and the one that is used for the programme being linked do not need to be the same. Is there any possibility of making this choice when a programme is linked and not when GHC is compiled?
Indeed you will take a bit of performance hit by using the one-step allocator since the check of whether an object resides in the heap (which is very hot during GC) is a fair bit more complex. As far as I know the choice of allocator has no effect on code generation so in principle it should be possible to link the same code against either RTS. However, the build system looks to be built around the assumption that the choice is made at configure-time. I'm sure this could be fixed, but it's not immediately obvious how. I suppose you could make the two allocators different RTS ways (e.g. like the distinction between event-logged, debugging, profiled and vanilla RTSs), but that would double the already large number of ways. Cheers, - Ben
Thanks for your feedback.
I am the lead developer of NGLess (https://github.com/ngless-toolkit/ngless a bioinformatics tool, written in Haskell). Several users have complained about not being able to easily use NGLess in an academic cluster environment due to the fact that it allocates 1TB of address space (e.g., https://groups.google.com/forum/#!topic/ngless/9su2E0EdeCc and I have also gotten several private emails on this issue).
In particular, many systems are set up with a limit on the address space so that if the job allocates more than the given limit, it is immediately killed. [snip] Are these address space limits advertised via getrlimit(2)? If so, have you tried GHC 8.6.1-alpha1? While fixing #14492 I taught GHC to respect rlimits when allocating its heap, so this might work now.
Thanks for fixing #14492 (that was us reporting it too, btw)! Some well configured systems do advertise the limits correctly and things work. For unrelated reasons, I have access to an AWS virtual private cloud, which was setup using SGE with default settings (and I think it should be fairly up-to-date). I took the opportunity to test and indeed NGLess runs without a glitch even pre-#14492 fix (just that memory is somewhat wasted). However, it seems that some users are running on misconfigured systems so that NGLess (and Haskell) end up getting blamed for the situation (furthermore, not all sysadmins are as helpful as they could be).
Indeed you will take a bit of performance hit by using the one-step allocator since the check of whether an object resides in the heap (which is very hot during GC) is a fair bit more complex.
I expected that. As I said, for now, I prefer that trade-off: we are already orders of magnitude faster than our competition, so I can afford a slowdown to support all these old-school HPC systems (which, for better or for worse, are still used by many of our target users).
As far as I know the choice of allocator has no effect on code generation so in principle it should be possible to link the same code against either RTS. However, the build system looks to be built around the assumption that the choice is made at configure-time.
I'm sure this could be fixed, but it's not immediately obvious how. I suppose you could make the two allocators different RTS ways (e.g. like the distinction between event-logged, debugging, profiled and vanilla RTSs), but that would double the already large number of ways.
I see. This becomes a distribution issue then. I currently distribute the binaries through bioconda, and stack is used for compilation. I will try to see if I can convince conda/stack to recompile ghc with the right flags for me. Thank you, Luis
participants (3)
-
Ben Gamari -
Luis Pedro Coelho -
Yitzchak Gale