Compilation of large data
Dear Café, How can I compile large data in the source code? The data structure contains a few numerical and String constants and a long list of structures (containing two short lists of Strings). If I use ghci, it's fine, no problem (until some size). When I want to compile then I run out of memory (32GB, plus 32GB swap) on arch linux. Yes, machine with 256GB solved temporarily the problem, but only temporarily. The data is generated (about 10.000 elements for the list and more). Should it be read from the file during the runtime? Should the structure be somehow decoupled (to small lists, each of them in a separate file)? Or there is no way, how to make it work... :-( If I use different input, a bit smaller, which I can compile, the resulting binary is rather small, less than 7MB. Regards Dušan
Hi Dušan, Could you open a GHC ticket with a reproducing example? It looks similar to https://gitlab.haskell.org/ghc/ghc/-/issues/16577 so you may have more chance with `-O0`. Thanks, Sylvain On 29/01/2021 10:57, Dušan Kolář wrote:
Dear Café,
How can I compile large data in the source code? The data structure contains a few numerical and String constants and a long list of structures (containing two short lists of Strings).
If I use ghci, it's fine, no problem (until some size). When I want to compile then I run out of memory (32GB, plus 32GB swap) on arch linux. Yes, machine with 256GB solved temporarily the problem, but only temporarily.
The data is generated (about 10.000 elements for the list and more). Should it be read from the file during the runtime? Should the structure be somehow decoupled (to small lists, each of them in a separate file)? Or there is no way, how to make it work... :-(
If I use different input, a bit smaller, which I can compile, the resulting binary is rather small, less than 7MB.
Regards
Dušan
_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.
On Fri, 29 Jan 2021, Sylvain Henry wrote:
Could you open a GHC ticket with a reproducing example?
It looks similar to https://gitlab.haskell.org/ghc/ghc/-/issues/16577 so you may have more chance with `-O0`.
If disabling optimization helps, then it might be useful to disable it only for this module by adding a pragma to the top of the module: {-# OPTIONS_GHC -O0 #-}
So -O0 didn't help. Nevertheless, ghc suggests to read it from file :-) "... ghc: sorry! (unimplemented feature or known bug) (GHC version 8.6.3 for x86_64-unknown-linux): Trying to allocate more than 129024 bytes. This is currently not possible due to a limitation of GHC's code generator. See http://hackage.haskell.org/trac/ghc/ticket/4505 for details. Suggestion: read data from a file instead of having large static data structures in code. ..." It seems that splitting the list into several modules (several thousands of elements per list are OK) works. Moreover, as it is generated, several modules can be generated too. And even more, efficiency is not affected (the same execution time and -O2 used for all modules). Regards (and over :-)) Dušan On pátek 29. ledna 2021 13:22:24 CET Henning Thielemann wrote:
On Fri, 29 Jan 2021, Sylvain Henry wrote:
Could you open a GHC ticket with a reproducing example?
It looks similar to https://gitlab.haskell.org/ghc/ghc/-/issues/16577 so you may have more chance with `-O0`. If disabling optimization helps, then it might be useful to disable it only for this module by adding a pragma to the top of the module:
{-# OPTIONS_GHC -O0 #-} _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.
-- Dusan Kolar tel: +420 54 114 1238 UIFS FIT VUT Brno fax: +420 54 114 1270 Bozetechova 2 e-mail: kolar@fit.vut.cz Brno 612 66 Czech Republic --
For truly monster static data, I’ve had success with putting into a c
file. But for interesting data structures this can be not ideal.
On Fri, Jan 29, 2021 at 8:26 AM Dušan Kolář
So -O0 didn't help.
Nevertheless, ghc suggests to read it from file :-)
"...
ghc: sorry! (unimplemented feature or known bug)
(GHC version 8.6.3 for x86_64-unknown-linux):
Trying to allocate more than 129024 bytes.
This is currently not possible due to a limitation of GHC's code generator.
See http://hackage.haskell.org/trac/ghc/ticket/4505 for details.
Suggestion: read data from a file instead of having large static data
structures in code.
..."
It seems that splitting the list into several modules (several thousands of elements per list are OK) works. Moreover, as it is generated, several modules can be generated too. And even more, efficiency is not affected (the same execution time and -O2 used for all modules).
Regards (and over :-))
Dušan
On pátek 29. ledna 2021 13:22:24 CET Henning Thielemann wrote:
On Fri, 29 Jan 2021, Sylvain Henry wrote:
Could you open a GHC ticket with a reproducing example?
It looks similar to https://gitlab.haskell.org/ghc/ghc/-/issues/16577 so
you may have more chance with `-O0`.
If disabling optimization helps, then it might be useful to disable it
only for this module by adding a pragma to the top of the module:
{-# OPTIONS_GHC -O0 #-}
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
--
Dusan Kolar tel: +420 54 114 1238
UIFS FIT VUT Brno fax: +420 54 114 1270
Bozetechova 2 https://www.google.com/maps/search/Bozetechova+2+e?entry=gmail&source=g e-mail: kolar@fit.vut.cz
Brno 612 66
Czech Republic
--
_______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are https://www.google.com/maps/search/Bozetechova+2+e?entry=gmail&source=g allowed to post.
participants (4)
-
Carter Schonwald -
Dušan Kolář -
Henning Thielemann -
Sylvain Henry