Don Stewart wrote:

You can build large constant bytestrings, fwiw. They turn into an Addr#, and GHC will leave them alone.

Well, for my particular problem -- I guess I could align all the elements of the lists, and then build the trie and maps from the ByteStrings at startup. -- _jsn

Don Stewart dons@galois.com:

jay:

...

I also have constants that are too large to compile. I am resigned to loading them from data files--other solutions seem even worse. ... Data.Binary eases the irritation somewhat.

Did you try bytestring literals (and maybe parsing them in-memory with Data.Binary)?

That didn't occur to me, since neither of my large constants includes strings.... I think you're suggesting that each constant could appear in the source as a long bytestring and be deserialized into the data structure. If that works, it should improve the startup time, but it's still not as nice as simply compiling it straight up. I'll try it. Jay

Bryan O'Sullivan wrote:

The trick I usually use in cases like this is to compile the data as C code and link against it, then access it from Haskell via a Ptr.

For my particular application, I really need to ship a single static binary that has it all -- data as well as algorithms -- so I'm going with the FFI. It's too bad that I end up working in the IO monad much of the time. I hope we'll have massive static constants someday soon! -- _jsn

On Sat, Mar 1, 2008 at 12:24 PM, Jay Scott wrote:

Jason Dusek jason.dusek@gmail.com: unsafePerformIO should be safe on constants, right? It has worked for me, at least.

Unfortunately, reading in the list does not allow me to ship a single binary. It's not file reading, though, that forces me in to the IO Monad -- it's pointer dereferencing. -- _jsn

Don Stewart dons@galois.com:

jay:

...

Don Stewart dons@galois.com:

...

jay:

...

I also have constants that are too large to compile. I am resigned to loading them from data files--other solutions seem even worse. ... Data.Binary eases the irritation somewhat.

Did you try bytestring literals (and maybe parsing them in-memory with Data.Binary)?

That didn't occur to me, since neither of my large constants includes strings.... I think you're suggesting that each constant could appear in the source as a long bytestring and be deserialized into the data structure. If that works, it should improve the startup time, but it's still not as nice as simply compiling it straight up.

I'll try it.

Here's an example, which stores a Data.Map in a gzip-compressed bytestring literal (a C string literal in the compiled code). The Map is reconstructed on startup.

{-# LANGUAGE OverloadedStrings #-}

import Data.Binary import qualified Data.Map as M import qualified Data.ByteString.Char8 as S import Data.ByteString.Lazy import Codec.Compression.GZip

-- -- this is a gzip compressed literal bytestring, storing a binary- encoded Data.Map -- mytable = "\US\139\b\NUL\NUL\NUL\NUL\NUL\NUL\ETXEN\219\SO\194 \f\197\224 \188\196\CAN\227\US\224\171~\NAKc\GS4ce\161`\178\191\215(\176\190\180\167 \231\210\n\241\171\203\191\ti\157\217\149\249< \ENQ\214\&9>\202\162\179a \132X\233\ESC=\231\215\164\SYN\157\DC2D\226*\146\174o\t\167\DLE\209\"i_ \240\193\129\199

main = print =<< M.lookup "ghc" m where -- build the table from the bytestring: m :: M.Map String (Maybe String) m = decode . decompress . fromChunks . return $ mytable

Running it:

$ ./A Just "dinosaur!"

:)

Important to use a bytestring, since that gets compiled to a C string literal (and not messed with by the simplifier).

-- Don

jay:

Don Stewart dons@galois.com:

...

jay:

...

Don Stewart dons@galois.com:

...

jay:

...

I also have constants that are too large to compile. I am resigned to loading them from data files--other solutions seem even worse. ... Data.Binary eases the irritation somewhat.

Did you try bytestring literals (and maybe parsing them in-memory with Data.Binary)?

I finally squeezed enough time to try it, and it didn't work for me.

-- ghc Overflow.hs [1 of 1] Compiling Overflow ( Overflow.hs, Overflow.o )

Enable optimisations! Compile with ghc -O2. You need this to avoid having a very slow pack call at runtime.

Overflow.hs:8:10:stack overflow: use +RTS -K<size> to increase it --

where Overflow.hs is in the vicinity of 40M and looks like

-- {-# LANGUAGE OverloadedStrings #-}

module Overflow where

import qualified Data.ByteString.Lazy as S

bigData :: S.ByteString bigData = "\0\0\0\0\0\5\67\195\0\0\0\0... --

I didn't compress it, because Codec.Compression.GZip didn't compile for me. It looked like a library change since 6.6 broke it.

Probably you don't have the zlib.h header? Or make sure you have the latest version of zlib from hackage -- it does work.

Is there a handy string escaping function in the libraries somewhere? It only took a minute to write one, and I spent longer than that looking, so maybe it's the wrong question.... Surely it's in there somewhere, and I'm just 2 dum 2 c.

The show function?

jay:

Don Stewart dons@galois.com:

...

jay:

...

Don Stewart dons@galois.com:

...

jay:

...

Don Stewart dons@galois.com:

...

jay: > I also have constants that are too large to compile. I am resigned to > loading them from data files--other solutions seem even worse. ... > Data.Binary eases the irritation somewhat.

Did you try bytestring literals (and maybe parsing them in-memory with Data.Binary)?

I finally squeezed enough time to try it, and it didn't work for me.

...

-- ghc Overflow.hs [1 of 1] Compiling Overflow ( Overflow.hs, Overflow.o )

Enable optimisations! Compile with ghc -O2. You need this to avoid having a very slow pack call at runtime.

Yes, I tried basic variations like that. The result is the same with -O1 or with -O2, and with Data.ByteString or Data.ByteString.Lazy .

Ok, hmm, that really shouldn't be the case. Do you have the example available somewhere? It's just a 40M inline bytestring? -- Don