Using GHC API to compile Haskell file

Hi, Is this the right place for GHC API queries? If not, is there anywhere better? I want to compile a Haskell module, much like `ghc --make` or `ghc -c` does. The sample code on the Haskell wiki (https://wiki.haskell.org/GHC/As_a_library#A_Simple_Example), StackOverflow (http://stackoverflow.com/a/5631338/160673) and in GHC API slides (http://sneezy.cs.nott.ac.uk/fplunch/weblog/wp-content/uploads/2008/12/ghc-ap...) says: import GHC import GHC.Paths ( libdir ) import DynFlags main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do runGhc (Just libdir) $ do dflags <- getSessionDynFlags setSessionDynFlags dflags target <- guessTarget "Test.hs" Nothing setTargets [target] load LoadAllTargets However, given a `Test.hs` file with the contents `main = print 1`, I get the error: C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSbase-4.7.0.1-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSinteger-gmp-0.5.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSghc-prim-0.3.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSrts-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lffi-6 collect2: ld returned 1 exit status Has the recipe changed? By turning up the verbosity, I was able to compare the command line passed to the linker. The failing GHC API call contains: "-lHSbase-4.7.0.1-ghc7.8.3" "-lHSinteger-gmp-0.5.1.0-ghc7.8.3" "-lHSghc-prim-0.3.1.0-ghc7.8.3" "-lHSrts-ghc7.8.3" "-lffi-6" While the succeeding ghc --make contains: "-lHSbase-4.7.0.1" "-lHSinteger-gmp-0.5.1.0" "-lHSghc-prim-0.3.1.0" "-lHSrts" "-lCffi-6" Should I be getting DynFlags differently to influence those link variables? Thanks, Neil

The problem is that the default code is trying to build a dynamically linked executable, but the Windows distributions don't come with dlls by default. Why doesn't the GHC API code pick this up? Based on snooping ghc/Main.hs, it's probably because you need to call parseDynamicFlags* which will call updateWays which will turn off -dynamic-too if the platform doesn't support it. GHC bug? Absolutely! Please file a ticket. Edward Excerpts from Neil Mitchell's message of 2015-08-23 05:43:28 -0700:
Hi,
Is this the right place for GHC API queries? If not, is there anywhere better?
I want to compile a Haskell module, much like `ghc --make` or `ghc -c` does. The sample code on the Haskell wiki (https://wiki.haskell.org/GHC/As_a_library#A_Simple_Example), StackOverflow (http://stackoverflow.com/a/5631338/160673) and in GHC API slides (http://sneezy.cs.nott.ac.uk/fplunch/weblog/wp-content/uploads/2008/12/ghc-ap...) says:
import GHC import GHC.Paths ( libdir ) import DynFlags
main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do runGhc (Just libdir) $ do dflags <- getSessionDynFlags setSessionDynFlags dflags target <- guessTarget "Test.hs" Nothing setTargets [target] load LoadAllTargets
However, given a `Test.hs` file with the contents `main = print 1`, I get the error:
C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSbase-4.7.0.1-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSinteger-gmp-0.5.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSghc-prim-0.3.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSrts-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lffi-6 collect2: ld returned 1 exit status
Has the recipe changed?
By turning up the verbosity, I was able to compare the command line passed to the linker. The failing GHC API call contains:
"-lHSbase-4.7.0.1-ghc7.8.3" "-lHSinteger-gmp-0.5.1.0-ghc7.8.3" "-lHSghc-prim-0.3.1.0-ghc7.8.3" "-lHSrts-ghc7.8.3" "-lffi-6"
While the succeeding ghc --make contains:
"-lHSbase-4.7.0.1" "-lHSinteger-gmp-0.5.1.0" "-lHSghc-prim-0.3.1.0" "-lHSrts" "-lCffi-6"
Should I be getting DynFlags differently to influence those link variables?
Thanks, Neil

Thanks Edward, that fixed the issue with GHC 7.8.3. While trying to
replicate with 7.10.2 to submit a bug report, I got a different error,
even with your fix included:
C:\Users\NDMIT_~1\AppData\Local\Temp\ghc2428_1\ghc_4.o:ghc_3.c:(.text+0x55):
undefined reference to `ZCMain_main_closure'
Doing another diff of the command lines, I see ghc --make includes
"Test.o" on the Link line, but the API doesn't.
Thanks, Neil
On Mon, Aug 24, 2015 at 12:00 AM, Edward Z. Yang
The problem is that the default code is trying to build a dynamically linked executable, but the Windows distributions don't come with dlls by default.
Why doesn't the GHC API code pick this up? Based on snooping ghc/Main.hs, it's probably because you need to call parseDynamicFlags* which will call updateWays which will turn off -dynamic-too if the platform doesn't support it.
GHC bug? Absolutely! Please file a ticket.
Edward
Excerpts from Neil Mitchell's message of 2015-08-23 05:43:28 -0700:
Hi,
Is this the right place for GHC API queries? If not, is there anywhere better?
I want to compile a Haskell module, much like `ghc --make` or `ghc -c` does. The sample code on the Haskell wiki (https://wiki.haskell.org/GHC/As_a_library#A_Simple_Example), StackOverflow (http://stackoverflow.com/a/5631338/160673) and in GHC API slides (http://sneezy.cs.nott.ac.uk/fplunch/weblog/wp-content/uploads/2008/12/ghc-ap...) says:
import GHC import GHC.Paths ( libdir ) import DynFlags
main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do runGhc (Just libdir) $ do dflags <- getSessionDynFlags setSessionDynFlags dflags target <- guessTarget "Test.hs" Nothing setTargets [target] load LoadAllTargets
However, given a `Test.hs` file with the contents `main = print 1`, I get the error:
C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSbase-4.7.0.1-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSinteger-gmp-0.5.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSghc-prim-0.3.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSrts-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lffi-6 collect2: ld returned 1 exit status
Has the recipe changed?
By turning up the verbosity, I was able to compare the command line passed to the linker. The failing GHC API call contains:
"-lHSbase-4.7.0.1-ghc7.8.3" "-lHSinteger-gmp-0.5.1.0-ghc7.8.3" "-lHSghc-prim-0.3.1.0-ghc7.8.3" "-lHSrts-ghc7.8.3" "-lffi-6"
While the succeeding ghc --make contains:
"-lHSbase-4.7.0.1" "-lHSinteger-gmp-0.5.1.0" "-lHSghc-prim-0.3.1.0" "-lHSrts" "-lCffi-6"
Should I be getting DynFlags differently to influence those link variables?
Thanks, Neil

So after all the linking bugs, I decided to give up on the linking
step (for now), as it's not essential to my use case. I can compile
two files with:
forM_ ["Test1.hs","Test.hs"] $ \file -> do
runGhc (Just libdir) $ do
liftIO $ print file
dflags <- getSessionDynFlags
(dflags, _, _) <- parseDynamicFlags dflags []
setSessionDynFlags dflags{ghcMode=OneShot, hscTarget =
HscAsm, ghcLink=NoLink}
setTargets [Target (TargetFile file Nothing) True Nothing]
load LoadAllTargets
And assuming Test.hs imports Test1.hs then everything is fine.
However, if I switch the runGhc and forM lines, so I compile both in
the same Ghc session, then I get an error about "Can't find
interface-file declaration for variable test1" when compiling Test.hs,
and when compiling Test.hs it does a fresh dependency check on Test1
and would compile it if I hadn't done so already.
The motivation for using a single Ghc session is that I'd like to
share things like loading the Prelude and not have to repeat that
work. Is there any way of doing that with two compiles which other
than sharing a cache I'd like to be separate?
Thanks, Neil
On Mon, Aug 24, 2015 at 8:42 AM, Neil Mitchell
Thanks Edward, that fixed the issue with GHC 7.8.3. While trying to replicate with 7.10.2 to submit a bug report, I got a different error, even with your fix included:
C:\Users\NDMIT_~1\AppData\Local\Temp\ghc2428_1\ghc_4.o:ghc_3.c:(.text+0x55): undefined reference to `ZCMain_main_closure'
Doing another diff of the command lines, I see ghc --make includes "Test.o" on the Link line, but the API doesn't.
Thanks, Neil
On Mon, Aug 24, 2015 at 12:00 AM, Edward Z. Yang
wrote: The problem is that the default code is trying to build a dynamically linked executable, but the Windows distributions don't come with dlls by default.
Why doesn't the GHC API code pick this up? Based on snooping ghc/Main.hs, it's probably because you need to call parseDynamicFlags* which will call updateWays which will turn off -dynamic-too if the platform doesn't support it.
GHC bug? Absolutely! Please file a ticket.
Edward
Excerpts from Neil Mitchell's message of 2015-08-23 05:43:28 -0700:
Hi,
Is this the right place for GHC API queries? If not, is there anywhere better?
I want to compile a Haskell module, much like `ghc --make` or `ghc -c` does. The sample code on the Haskell wiki (https://wiki.haskell.org/GHC/As_a_library#A_Simple_Example), StackOverflow (http://stackoverflow.com/a/5631338/160673) and in GHC API slides (http://sneezy.cs.nott.ac.uk/fplunch/weblog/wp-content/uploads/2008/12/ghc-ap...) says:
import GHC import GHC.Paths ( libdir ) import DynFlags
main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do runGhc (Just libdir) $ do dflags <- getSessionDynFlags setSessionDynFlags dflags target <- guessTarget "Test.hs" Nothing setTargets [target] load LoadAllTargets
However, given a `Test.hs` file with the contents `main = print 1`, I get the error:
C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSbase-4.7.0.1-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSinteger-gmp-0.5.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSghc-prim-0.3.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSrts-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lffi-6 collect2: ld returned 1 exit status
Has the recipe changed?
By turning up the verbosity, I was able to compare the command line passed to the linker. The failing GHC API call contains:
"-lHSbase-4.7.0.1-ghc7.8.3" "-lHSinteger-gmp-0.5.1.0-ghc7.8.3" "-lHSghc-prim-0.3.1.0-ghc7.8.3" "-lHSrts-ghc7.8.3" "-lffi-6"
While the succeeding ghc --make contains:
"-lHSbase-4.7.0.1" "-lHSinteger-gmp-0.5.1.0" "-lHSghc-prim-0.3.1.0" "-lHSrts" "-lCffi-6"
Should I be getting DynFlags differently to influence those link variables?
Thanks, Neil

Hello Neil,
Sorry about the delay; I hadn't gotten around to seeing if I could reproduce it. Here is a working copy of the program which appears to work with GHC 7.10.2 on 64-bit Windows:
module Main where
import GHC
import GHC.Paths ( libdir )
import DynFlags
import SysTools
main = do
defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
runGhc (Just libdir) $ do
dflags <- getSessionDynFlags
setSessionDynFlags (gopt_set dflags Opt_Static)
target <- guessTarget "Test.hs" Nothing
setTargets [target]
load LoadAllTargets
Here is how I tested it:
stack ghc -- -package ghc -package ghc-paths --make Main.hs
(after stack installing ghc-paths)
Did you mean the error occurred when you did set Opt_Static? I can’t reproduce your specific error in that case either.
Cheers,
Edward
Sent from Windows Mail
From: Neil Mitchellmailto:ndmitchell@gmail.com
Sent: Monday, August 24, 2015 12:42 AM
To: Edward Z Yangmailto:ezyang@mit.edu
Cc: ghc-devs@haskell.orgmailto:ghc-devs@haskell.org
Thanks Edward, that fixed the issue with GHC 7.8.3. While trying to
replicate with 7.10.2 to submit a bug report, I got a different error,
even with your fix included:
C:\Users\NDMIT_~1\AppData\Local\Temp\ghc2428_1\ghc_4.o:ghc_3.c:(.text+0x55):
undefined reference to `ZCMain_main_closure'
Doing another diff of the command lines, I see ghc --make includes
"Test.o" on the Link line, but the API doesn't.
Thanks, Neil
On Mon, Aug 24, 2015 at 12:00 AM, Edward Z. Yang
The problem is that the default code is trying to build a dynamically linked executable, but the Windows distributions don't come with dlls by default.
Why doesn't the GHC API code pick this up? Based on snooping ghc/Main.hs, it's probably because you need to call parseDynamicFlags* which will call updateWays which will turn off -dynamic-too if the platform doesn't support it.
GHC bug? Absolutely! Please file a ticket.
Edward
Excerpts from Neil Mitchell's message of 2015-08-23 05:43:28 -0700:
Hi,
Is this the right place for GHC API queries? If not, is there anywhere better?
I want to compile a Haskell module, much like `ghc --make` or `ghc -c` does. The sample code on the Haskell wiki (https://wiki.haskell.org/GHC/As_a_library#A_Simple_Example), StackOverflow (http://stackoverflow.com/a/5631338/160673) and in GHC API slides (http://sneezy.cs.nott.ac.uk/fplunch/weblog/wp-content/uploads/2008/12/ghc-ap...) says:
import GHC import GHC.Paths ( libdir ) import DynFlags
main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do runGhc (Just libdir) $ do dflags <- getSessionDynFlags setSessionDynFlags dflags target <- guessTarget "Test.hs" Nothing setTargets [target] load LoadAllTargets
However, given a `Test.hs` file with the contents `main = print 1`, I get the error:
C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSbase-4.7.0.1-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSinteger-gmp-0.5.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSghc-prim-0.3.1.0-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lHSrts-ghc7.8.3 C:/Program Files (x86)/MinGHC-7.8.3/ghc-7.8.3/mingw/bin/ld.exe: cannot find -lffi-6 collect2: ld returned 1 exit status
Has the recipe changed?
By turning up the verbosity, I was able to compare the command line passed to the linker. The failing GHC API call contains:
"-lHSbase-4.7.0.1-ghc7.8.3" "-lHSinteger-gmp-0.5.1.0-ghc7.8.3" "-lHSghc-prim-0.3.1.0-ghc7.8.3" "-lHSrts-ghc7.8.3" "-lffi-6"
While the succeeding ghc --make contains:
"-lHSbase-4.7.0.1" "-lHSinteger-gmp-0.5.1.0" "-lHSghc-prim-0.3.1.0" "-lHSrts" "-lCffi-6"
Should I be getting DynFlags differently to influence those link variables?
Thanks, Neil

Sorry about the delay; I hadn't gotten around to seeing if I could reproduce it. Here is a working copy of the program which appears to work with GHC 7.10.2 on 64-bit Windows:
Thanks, that does indeed solve it the first bit. To try and make it a bit clearer what I'm after, I've put the stuff in a git repo: https://github.com/ndmitchell/ghc-process/blob/master/Main.hs Looking at Main.hs, there are three modes, Process (run ghc.exe 3 times), APIMake (the code you sent me), and APISingle (attempt to replicate the 3 ghc.exe invokations through the GHC API). The first two work perfectly, following Edward's tweaks. The final one fails at linking. So I have two questions: 1) Is there any way to do the two compilations sharing some cached state, e.g. loaded packages/.hi files, so each compilation goes faster. 2) Is there any way to do the link alone through the GHC API. Thanks, Neil

Hello Neil, It looks like my second message got eaten. Let's try again.
1) Is there any way to do the two compilations sharing some cached state, e.g. loaded packages/.hi files, so each compilation goes faster.
You can, using withTempSession in the GhcMonad. The external package state will be preserved across calls here, but things put in the HPT will get thrown out.
2) Is there any way to do the link alone through the GHC API.
I am confused by your code. There are two ways you can do linking: 1. Explicitly specify all of the objects to link together. This works even if the source files aren't available. 2. Run ghc --make. This does dependency analysis to figure out what objects to link together, but since everything is already compiled, it just links. Your code seems to be trying to do (1) and (2) simultaneously (you set the mode to OneShot, but then you call load which calls into GhcMake). If you want to use (1), stop calling load and call 'oneShot' instead. If you want to use (2), just reuse your working --make code. (BTW, how did I figure this all out? By looking at ghc/Main.hs). Cheers, Edward

1) Is there any way to do the two compilations sharing some cached state, e.g. loaded packages/.hi files, so each compilation goes faster.
You can, using withTempSession in the GhcMonad. The external package state will be preserved across calls here, but things put in the HPT will get thrown out.
So as far as I can tell, you are suggesting I basically do getSession in one session, grab the cache bits of the HscEnv, and inject them into the start of the next session with setSession? (withTempSession and all the other session functions just seem to be some variation on that pattern). I tried that, but even storing just hsc_EPS between sessions (which seemed like it should both be something that never changes), causes weird compile failures. Is moving things like hsc_EPS between sessions supported? Or were you suggesting I do something else?
2) Is there any way to do the link alone through the GHC API.
I am confused by your code. There are two ways you can do linking:
1. Explicitly specify all of the objects to link together. This works even if the source files aren't available.
2. Run ghc --make. This does dependency analysis to figure out what objects to link together, but since everything is already compiled, it just links.
Your code seems to be trying to do (1) and (2) simultaneously (you set the mode to OneShot, but then you call load which calls into GhcMake).
If you want to use (1), stop calling load and call 'oneShot' instead. If you want to use (2), just reuse your working --make code.
(BTW, how did I figure this all out? By looking at ghc/Main.hs).
Thanks. I have now started down that rabbit hole, and am currently peering around. I haven't yet succeeded, but given what I can see, it just seems like a matter of time. Thanks, Neil

Excerpts from Neil Mitchell's message of 2015-09-14 14:07:14 -0700:
1) Is there any way to do the two compilations sharing some cached state, e.g. loaded packages/.hi files, so each compilation goes faster.
You can, using withTempSession in the GhcMonad. The external package state will be preserved across calls here, but things put in the HPT will get thrown out.
So as far as I can tell, you are suggesting I basically do getSession in one session, grab the cache bits of the HscEnv, and inject them into the start of the next session with setSession? (withTempSession and all the other session functions just seem to be some variation on that pattern). I tried that, but even storing just hsc_EPS between sessions (which seemed like it should both be something that never changes), causes weird compile failures. Is moving things like hsc_EPS between sessions supported? Or were you suggesting I do something else?
No, something a bit different: I'm suggesting that you use this functionality to "fork" a session: so you do some work, getting a session, snapshot the session, do some more work, and then use the snapshot to rollback before the work. I haven't actually tested this, however. Edward
participants (3)
-
Edward Z Yang
-
Edward Z. Yang
-
Neil Mitchell