Hi: Here is what Cabal does: $ ./Setup.lhs -v build C:\DOCUME~1\vivian\LOCALS~1\Temp\ghc139100_0\ghc139100_0.hc:8:37: ../src/Matlab/C/CMatrix.h: No such file or directory Reading parameters from D:\Apps\MSYS\1.0\local\src\MatlabCompiling\matlab.buildinfo Preprocessing library Matlab-0.0.1... D:\Apps\MSYS\1.0\local\bin\c2hs.exe -C -D__GLASGOW_HASKELL__=606 -C -Ic:\Apps\MATLAB704\bin\win32\..\..\extern\include -o src\Matlab\C\CMatrix.hs src\Matlab\C\CMatrix.chs D:\Apps\MSYS\1.0\local\bin\c2hs.exe -C -D__GLASGOW_HASKELL__=606 -C -Ic:\Apps\MATLAB704\bin\win32\..\..\extern\include -o src\Matlab\C\CMat.hs src\Matlab\C\CMat.chs c2hs.exe: Matlab/C/CMatrix.chi not found in: . Setup.lhs: got error code while preprocessing: Matlab.C.CMat Here is my command that works (from MatlabCompiling/test) $ c2hs --cppopts=-I/c/Apps/MATLAB704/extern/include --include=../src ../src/Matlab/C/CMatrix.chs Here is the addition to matlab.buildinfo which makes Cabal work: c2hs-options: --cppopts=-I/c/Apps/MATLAB704/bin/win32/../../extern/include --include=src I'm also attaching a file which should recreate all the errors that I've found on MinGW/WinXP. Note that there are files outside the cabal hierarchy to simulate headers/library which are already on the system and outside the packager's control. Cheers, Vivian

-----Original Message----- From: Duncan Coutts [mailto:duncan.coutts@worc.ox.ac.uk] Sent: Friday, 20 April 2007 6:10 p.m. To: Vivian McPhail Cc: cabal-devel@haskell.org Subject: RE: darcs patch: added cppOptions and c2hsOptions

On Fri, 2007-04-20 at 16:32 +1200, Vivian McPhail wrote:

...

Hi Duncan,

I added cppOptions because in the Cabal.cabal file there was a comment to add cppOptions and so I did it.

Ok.

...

With respect to c2hsOptions, the problem is that c2hs passes options to the CPP nonstandardly (with --cppopts=), so even if I have an -I* option in my ccOptions that gets passed to c2hs, this works for c2hs, but doesn't get passed to the cpp that c2hs uses.

Sorry, I still don't see it fully. Could you say exactly what the c2hs command line is that you use when it works and what command line cabal uses that does not work (and that your patch solves).

...

I've sent a message to the c2hs/cabal mailing list with a dummy package that recreates all the problems I had on mingw/WinXP.

I'm not sure that we got this. Could you send it again, perhaps ccing it to me directly.

Duncan

On Fri, 2007-04-20 at 20:17 +1000, Duncan Coutts wrote:

Hia Vivian, [...] I'm looking into it.

I have a patch, I'll send it soonish (probably Monday) for other people to review and test. It's a general change to the way Cabal does pre-processing. Instead of generating the .hs files in place in the source tree we put them all in a special build tree in the dist dir (like where we put the .o and .hi files, but not actually the same directory). So the patch does that for all pre-processors, and in addition gives the pre-processors a bit more information so that they can distinguish the base name of the output file and the output dir prefix. This allows us to tell c2hs this relative path so that it will use it when it generates links to .h files in the .hs files it generates. This also depends on a separate patch to c2hs. Duncan

On Sat, 2007-04-21 at 22:35 +0100, Ross Paterson wrote:

On Sat, Apr 21, 2007 at 10:32:08PM +1000, Duncan Coutts wrote:

...

I have a patch, I'll send it soonish (probably Monday) for other people to review and test.

It's a general change to the way Cabal does pre-processing. Instead of generating the .hs files in place in the source tree we put them all in a special build tree in the dist dir (like where we put the .o and .hi files, but not actually the same directory).

I've been meaning to do something like that, to get the generated files under dist, and to separate out the compiler-independent part of the code. There should be no need for the compiler-specific parts to worry about main-is or hs-source-dirs, when the first phase can handle those.

Another benefit is that we could include compiler-independent preprocessor output (where possible) in source distributions. If the package is built in an environment lacking that preprocessor, it can use the shipped output. If the preprocessor is available, it can run it with compiler-specific flags (e.g. happy -agc). Not all preprocessors can produce compiler-independent output, though.

I've got it to a half-way house, I've made PreProcessor into a record data type which records if the pre-processor generates platform independent output. We should add something about whether the result is independent of the haskell implementation or not. ( Incidentally, This PreProcessor abstraction is probably the right place to add things like composing pre-processors, eg .y.pp, .chs.pp etc. I'm not sure if PreProcessor should incorporate the suffix stuff or if it should represent a particular use of a pre-processor independent of some general suffix rule. ) Also, I've still got the platform-independent pre-processed files going back into the src dirs. This isn't nice. The only reason I'm doing that is not to break sdist, however we should probably do that differently anyway. For example at the moment you need to have done a build for the pre-processed sources to exist in the src dir. Perhaps sdist should just generate the platform independent (and impl-independent) sources directly into the temp dir where we make the tarball. Then during builds we can put them all into the dist/build dir. The only downside to this is if generating the sources is expensive. Sometimes it is indeed expensive, happy can take quite a while on large grammars and in Gtk2Hs, generating 100+ .hs files with c2hs takes a minute or two on a fast box. So how about this, we put generated sources into their own dir, not dist/build but somewhere else. We separate platform dependent and independent generated sources. We could even separate impl-dependent sources from independent ones. Then if you do sdist after a build then there is no additional pre-processing needed, but if you do sdist from a clean tree then it will do the pre-processing. Some thing like: gen/platform-dependent/ gen/platform-independent/ghc/ gen/platform-independent/generic/ it would be the gen/platform-independent/ tree that'd be included into the src tarball. That way when it comes to building from a tarball we find we don't have to re-run the pre-processor, assuming the file modification times all work out as expected. I don't think the gen/platform-independent can live under dist since that doesn't need to be anywhere related to the build tree and yet this must be present if we build from a tarball. Suggestions? Duncan

On Wed, 2007-05-02 at 00:31 +0100, Ross Paterson wrote:

On Tue, May 01, 2007 at 03:45:00PM +0100, Duncan Coutts wrote:

...

Also, I've still got the platform-independent pre-processed files going back into the src dirs. This isn't nice. The only reason I'm doing that is not to break sdist, however we should probably do that differently anyway.

For example at the moment you need to have done a build for the pre-processed sources to exist in the src dir. Perhaps sdist should just generate the platform independent (and impl-independent) sources directly into the temp dir where we make the tarball. Then during builds we can put them all into the dist/build dir.

Certainly. Building (using specialized preprocessing) ought to be separated from constructing source packages (which requires portable preprocessor output).

...

The only downside to this is if generating the sources is expensive. Sometimes it is indeed expensive, happy can take quite a while on large grammars and in Gtk2Hs, generating 100+ .hs files with c2hs takes a minute or two on a fast box.

I wonder if optimizing this is worth the increase in complexity (and where does it end -- you need platform-specific versions to avoid the c2hs runs).

I was talking nonsense I realise. Of course for building the sdist we don't run c2hs anyway, we only generate the platform-independent .hs files. So the only one of those that takes time is happy and that never takes that long and people typically do not have dozens of .y files in a project (unlike .chs/.hsc files).

I was after something less ambitious: including portable preprocessor output in a source package so that people can build it even if they don't have the preprocessor. (Their build may be sub-optimal, though.)

Hmm, so then for example most people will end up not getting the fastest possible haddock since the tarball will contain only the generic happy parser. Actually won't everyone using the haddock tarball get the slow parser even if they do have happy installed because the normal thing is to not rebuild the pre-preocessed file if it was modified later?

That could be done by mapping each suffix to a pair of functions

portable :: BuildInfo -> LocalBuildInfo -> Maybe PreProcessor specific :: BuildInfo -> LocalBuildInfo -> Maybe PreProcessor for type Preprocessor = (FilePath, FilePath) -> (FilePath, FilePath) -> Int -> IO ()

The first one would be used in sdist; if the portable preprocessor is available, add its output to the source package.

The second function would be used in build; if the specific preprocessor is unavailable, copy the .hs version of the module (which, if present, should be portable preprocessor output). (Presumably if the portable preprocessor is available the specific one is too, but we don't need to use that.)

Oh ok, so that does deal with the haddock tarball containing the generic parser but me having happy installed on the system and ending up with the fast parser.

BTW, since you're now collecting Haskell files in a single build hierarchy, you could copy the main-is to Main.[l]hs in that hierarchy,

Why would we want to do that?

and also generate the Paths module directly there instead of in autogen.

Yes.

BTW2, building under Hugs will now require two directories under dist: one for the (build-specific) preprocessor output and another for the build outputs, which may be obtained from those using cpphs and ffihugs.

Ah yes, indeed. Duncan