On 05/01/14 14:14, Sven Panne wrote:

My ALUT package is among the ones without documentation, and I have a theory what's wrong (probably a missing installed C library/package libalut-dev, see https://github.com/haskell-openal/ALUT/blob/master/.travis.yml). Two questions:

* As a package maintainer, how can I find out why the documentation is not built/published on Hackage? Logs, etc.?

I mention a link to a GitHub post in my opening post which outlines how to access the log. In case of ALUT, the build log is at [1] http://hackage.haskell.org/package/ALUT-2.3.0.1/reports/1/log and indeed a C library is missing.

* Who should be contacted to polish the server a bit (i.e. install libalut-dev)?

No idea, but I'd love to find out myself! I'm just a disgruntled Hackage user myself.

Currently things are very opaque from the outside, both from a technical and from an organizational point of view, and manually uploading documentation which is effectively guaranteed to be wrong/outdated/etc. sooner or later is not a solution IMHO.

You would be uploading documentation for a specific package version. So for example, if you upload something for ALUT 2.3.0.1 and later upload upload ALUT 2.4, either the docs will be built fine (which would be ideal) or you will have to upload them yourself again for that version. I agree that this is subpar but I think it's better than nothing. There should really be a ‘upload documentation’ button on the site or maybe even a Cabal flag or maybe even ability to bundle documentation in the package tarball when uploading it, to act as a fallback if package fails to build. I suppose this is something that has to be discussed. Ideally, it should just work.

Cheers, S.

[1]: http://hackage.haskell.org/package/ALUT-2.3.0.1/reports/1/log -- Mateusz K.

On 05/01/14 18:15, Peter Selinger wrote:

I agree. Two of my packages are in your list: easyrender and newsynth (both have "Nothing" for a reason in your list).

If a package has Nothing for a reason, that means that no build log is available. From what I've read yesterday, it's a problem with their report system and I think it happens when cabal can't find dependency candidates for your package. I link to a comment on GitHub in my opening post which explains this.

The problem for me is that, although you seem to have access to build logs, I don't. I have not found the way to access the hackage build logs for my packages or their documentation. Could you let me know where I can find them?

I do not have any special access to Hackage, I didn't even log in. See my opening post for how to access the build log or see my reply to Sven for an example. I really think there should be a button on the site for this. If the build logs existed for newsynth, you could do the following: check general build status[1] then check the first report[2] then check the build log for the first report[3]. If the build status is empty, you won't have any reports. Check out [4][5][6] for an example on a package which failed to build and has logs. Effectively, the Nothing in my ‘report’ indicates no build status.

For both packages, the documentation builds just fine on my local machine. It also builds fine in a virtual machine, under Windows and Ubuntu. Since I don't have access to Hackage's build logs, I cannot really figure out why the documentation is not building there. This is what has prevented me from fixing it.

I suspect Hackage fails to resolve your dependencies, at least that what seems to be causing no logs. See [7] for a comment and [8] for an existing GitHub issue (although one without any activity).

I even created "candidates" for the packages, before uploading the packages to the main index. Again, the documentation did not build, and again, I could not find any logs to tell me what went wrong. So the whole "candidate" mechanism has so far been useless to me.

You mentioned that there is a way to upload the documentation manually. I'd love to do that. But how? I don't see any buttons or links on the package maintainer's pages that would allow me to do that.

I post a link in my opening post to a comment about this. See [9]. I just tried to do it for one of my packages (yi-monokai-0.1.1.1) and here are the steps I took: 1. cd ~/programming/yi-monokai 2. cabal configure && cabal build && cabal haddock --hyperlink-source 3. cd dist/doc 4. mv yi-monokai yi-monokai-0.1.1.1-docs 5. tar -c -v -z -Hustar -f yi-monokai-0.1.1.1-docs.tar.gz yi-monokai-0.1.1.1-docs 6. curl -X PUT -H 'Content-Type: application/x-tar' -H 'Content-Encoding: gzip' --data-binary '@yi-monokai-0.1.1.1-docs.tar.gz' 'http://USERNAME:PASSWORD@hackage.haskell.org/package/yi-monokai-0.1.1.1/docs' With these steps, my little package now has documentation. There's some info on format at [10]. I might write a small blog post outlining these steps later as it was not easy to figure out.

Any help appreciated, -- Peter

[1]: http://hackage.haskell.org/package/newsynth-0.1.0.0/reports/ [2]: http://hackage.haskell.org/package/newsynth-0.1.0.0/reports/1 [3]: http://hackage.haskell.org/package/newsynth-0.1.0.0/reports/1/log [4]: http://hackage.haskell.org/package/yi-monokai-0.1.1.1/reports/ [5]: http://hackage.haskell.org/package/yi-monokai-0.1.1.1/reports/1 [6]: http://hackage.haskell.org/package/yi-monokai-0.1.1.1/reports/1/log [7]: https://github.com/haskell/hackage-server/issues/145#issuecomment-30129142 [8]: https://github.com/haskell/hackage-server/issues/142 [9]: https://github.com/haskell/hackage-server/issues/145#issuecomment-29468613 [10]: https://github.com/haskell/hackage-server/issues/56 -- Mateusz K.

i'm sure patches are welcome to improve hackage infrastructure in these ways! :) On Mon, Jan 6, 2014 at 3:00 PM, Peter Selinger wrote:

Thank you, this worked like a charm! (Modulo using dist/doc/html instead of dist/doc). Now all my packages have online documentation.

One day I'd still like to find out why it didn't build automatically. But as long as documentation builder doesn't generate a log saying what problem it thought could not be solved, there is no way of knowing really.

Thanks again, -- Peter

Mateusz Kowalczyk wrote:

...

On 05/01/14 18:15, Peter Selinger wrote:

...

I agree. Two of my packages are in your list: easyrender and newsynth (both have "Nothing" for a reason in your list).

If a package has Nothing for a reason, that means that no build log is available. From what I've read yesterday, it's a problem with their report system and I think it happens when cabal can't find dependency candidates for your package. I link to a comment on GitHub in my opening post which explains this.

...

The problem for me is that, although you seem to have access to build logs, I don't. I have not found the way to access the hackage build logs for my packages or their documentation. Could you let me know where I can find them?

I do not have any special access to Hackage, I didn't even log in. See my opening post for how to access the build log or see my reply to Sven for an example. I really think there should be a button on the site for this. If the build logs existed for newsynth, you could do the following: check general build status[1] then check the first report[2] then check the build log for the first report[3]. If the build status is empty, you won't have any reports.

Check out [4][5][6] for an example on a package which failed to build and has logs.

Effectively, the Nothing in my ‘report’ indicates no build status.

...

For both packages, the documentation builds just fine on my local machine. It also builds fine in a virtual machine, under Windows and Ubuntu. Since I don't have access to Hackage's build logs, I cannot really figure out why the documentation is not building there. This is what has prevented me from fixing it.

I suspect Hackage fails to resolve your dependencies, at least that what seems to be causing no logs. See [7] for a comment and [8] for an existing GitHub issue (although one without any activity).

...

I even created "candidates" for the packages, before uploading the packages to the main index. Again, the documentation did not build, and again, I could not find any logs to tell me what went wrong. So the whole "candidate" mechanism has so far been useless to me.

You mentioned that there is a way to upload the documentation manually. I'd love to do that. But how? I don't see any buttons or links on the package maintainer's pages that would allow me to do that.

I post a link in my opening post to a comment about this. See [9]. I just tried to do it for one of my packages (yi-monokai-0.1.1.1) and here are the steps I took:

1. cd ~/programming/yi-monokai 2. cabal configure && cabal build && cabal haddock --hyperlink-source 3. cd dist/doc 4. mv yi-monokai yi-monokai-0.1.1.1-docs 5. tar -c -v -z -Hustar -f yi-monokai-0.1.1.1-docs.tar.gz yi-monokai-0.1.1.1-docs 6. curl -X PUT -H 'Content-Type: application/x-tar' -H 'Content-Encoding: gzip' --data-binary '@yi-monokai-0.1.1.1-docs.tar.gz' '

http://USERNAME:PASSWORD@hackage.haskell.org/package/yi-monokai-0.1.1.1/docs '

...

With these steps, my little package now has documentation. There's some info on format at [10]. I might write a small blog post outlining these steps later as it was not easy to figure out.

...

Any help appreciated, -- Peter

[1]: http://hackage.haskell.org/package/newsynth-0.1.0.0/reports/ [2]: http://hackage.haskell.org/package/newsynth-0.1.0.0/reports/1 [3]: http://hackage.haskell.org/package/newsynth-0.1.0.0/reports/1/log [4]: http://hackage.haskell.org/package/yi-monokai-0.1.1.1/reports/ [5]: http://hackage.haskell.org/package/yi-monokai-0.1.1.1/reports/1 [6]: http://hackage.haskell.org/package/yi-monokai-0.1.1.1/reports/1/log [7]:

https://github.com/haskell/hackage-server/issues/145#issuecomment-30129142

...

[8]: https://github.com/haskell/hackage-server/issues/142 [9]:

https://github.com/haskell/hackage-server/issues/145#issuecomment-29468613

...

[10]: https://github.com/haskell/hackage-server/issues/56 -- Mateusz K.

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

Am 06.01.2014 11:27, schrieb Malcolm Wallace:

I think the fundamental problem is that Haddock is now built on top of ghc. So if a package cannot be built by ghc (for whatever reason, e.g. missing C library dependency), then it cannot be documented either. This is a good deal less than useful. A documentation generator ought to do a reasonable job, even if the code it is looking at is technically not-compilable.

At work, we have a stand-alone documentation generator for Haskell, which requires no compiler. Haddock also was once stand-alone. I think it might be time to wind the clock backwards and retrieve this desirable property.

I think the clean solution would be a real "GHC as a collection of libraries" instead of "GHCi as a library". Then haddock could use the GHC parser without its type checker etc. for documentation purposes. The haskell-suite might become the project to fill that gap. It might still be difficult to generate docs in the presence of module re-exports, type class instances, missing type signatures, template haskell etc.

Am 06.01.2014 12:59, schrieb Mateusz Kowalczyk:

On 06/01/14 10:27, Malcolm Wallace wrote:

...

At work, we have a stand-alone documentation generator for Haskell, which requires no compiler. Haddock also was once stand-alone. I think it might be time to wind the clock backwards and retrieve this desirable property.

Was Haddock ever stand alone? AFAIK it used to be part of GHC and then David Waern separated it into a separate package. I honestly can't think of how it would ever have been stand-alone (that is not relying on GHC to do part of the work).

The versions with leading zero by Simon Marlow used a custom parser. http://hackage.haskell.org/package/haddock-0.8

On Mon, Jan 6, 2014 at 5:27 AM, Malcolm Wallace wrote:

I think the fundamental problem is that Haddock is now built on top of ghc. So if a package cannot be built by ghc (for whatever reason, e.g. missing C library dependency), then it cannot be documented either. This is a good deal less than useful. A documentation generator ought to do a reasonable job, even if the code it is looking at is technically not-compilable.

At work, we have a stand-alone documentation generator for Haskell, which requires no compiler. Haddock also was once stand-alone. I think it might be time to wind the clock backwards and retrieve this desirable property.

The problem with that was you didn't get documentation if you used any GHC extension added within the past year or so. You can't win.... -- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

On 07/01/14 01:23, Peter Selinger wrote:

Thanks, that's even better!

However, I find that the --contents-location option to cabal haddock does not work properly. Apparently it not only prevents index.html from being built (which makes modest sense), but it also prevents index-frames.html from being built (which does not). So in the frames view, one of the frames will just say "Sorry, it's just not here". That's not very nice. Also, if one loads the "/frames.html" URL directly, it still points to the non-existing index.html, rather than the one given by --contents-location. I think these are bugs.

For example, see http://hackage.haskell.org/package/yi-monokai-0.1.1.1/docs/Yi-Style-Monokai.... and click on "Frames".

Ah, I do not use frames so I did not notice.

One workaround is to run "cabal haddock" twice: first without the --contents-location option, and then with it. Slightly annoying, but it works. It has the disadvantage that if one goes directly to the ".../frames.html" URL, one will still see the old index.html.

Another workaround is to omit the --contents-location option altogether, and to manually create an index.html that is just a forward to the correct location, i.e.:

<meta HTTP-EQUIV="REFRESH" content="0; url=http://hackage.haskell.org/package/newsynth-0.1.0.0"> If your browser does not take you there automatically, please go to <a href=http://hackage.haskell.org/package/newsynth-0.1.0.0>http://hackage.haskell.org/package/newsynth-0.1.0.0</a>.

Only this last solution seems to do the correct thing all the time. For an example, see http://hackage.haskell.org/package/newsynth-0.1.0.0/docs/frames.html

Your instructions have been *very* helpful. But it's a lot of details and finnicky cabal directives to keep track of for doing this all the time. I've turned your instructions into a Makefile to automate this task; see attached. In principle, only the first two lines should need to be customized. Then "make doc-upload" will build the documentation and upload it correctly.

That looks very useful but I have a few questions about it: * Why a Makefile as opposed to a Bash script or something? I don't think there's a way to pass arguments into a Makefile like to Bash scripts, save for environmental variables. I'd much rather prefer a command line flag for a script in my PATH rather than having to change the file each time. * This file makes us change the top two lines (user and package name) and uses ‘read’ for everything else. Why not fully go with one way or the other? Personally, I'd prefer arguments rather than ‘read’ (much easier to repeat commands/script against) but it's up to personal taste. * Can we not grab to project name from the cabal file just like you do with version? * Your ‘--html-location’ has ‘$$pkg’ in it, how come? Is that some kind of escape? * Considering you seem to know what you're doing much more than I am, is there any way to detect cabal sandboxes? I'm trying your makefile now (running with make -f /tmp/Makefile doc-upload) and it's giving me an error: ``` [snip] Haddock coverage: 100% ( 18 / 18) in 'Yi.Style.Monokai' Documentation created: dist/doc/html/yi-monokai/index.html rm -r yi-monokai-0.1.1.1-docs rm: cannot remove ‘yi-monokai-0.1.1.1-docs’: No such file or directory make: *** [yi-monokai-0.1.1.1-docs.tar.gz] Error 1 ``` It works fine if I comment out the ‘rm -r’. Perhaps that should be allowed to fail somehow.

I know, Cabal somehow was supposed to make Makefiles obsolete. Alas. Fortunately, the Makefile is only for the package maintainer, and not for the package user.

-- Peter

I think that cabal-install should really have the option to do all this rather than have people write up scripts like this, resulting in trial and error. -- Mateusz K.

Hi Mateusz, of course you could do the same thing with a bash script. It's a matter of personal taste and workflow. Just for the record, you can pass arguments into a Makefile like this: "make PACKAGE=bla". And "$$" is indeed an escape, to denote a literal dollar sign (as opposed to a Makefile variable). Sorry about the bug. It should be "rm -rf". I'm not sure about the wisdom of adding new functionality to cabal-install just to compensate for bugs in the Hackage documentation builder. It seems that the packages build correctly everywhere else, so there is no reason in principle why the Hackage documentation builder could not be fixed to build them. I see the manual uploading as more of a temporary workaround. As for the bug with --contents-location and frames, it doesn't come up in the Hackage documentation builder because it apparently disables frames. There must be another (undocumented?) option for turning frames off. -- Peter Mateusz Kowalczyk wrote:

...

Your instructions have been *very* helpful. But it's a lot of details and finnicky cabal directives to keep track of for doing this all the time. I've turned your instructions into a Makefile to automate this task; see attached. In principle, only the first two lines should need to be customized. Then "make doc-upload" will build the documentation and upload it correctly.

That looks very useful but I have a few questions about it: * Why a Makefile as opposed to a Bash script or something? I don't think there's a way to pass arguments into a Makefile like to Bash scripts, save for environmental variables. I'd much rather prefer a command line flag for a script in my PATH rather than having to change the file each time. * This file makes us change the top two lines (user and package name) and uses ‘read’ for everything else. Why not fully go with one way or the other? Personally, I'd prefer arguments rather than ‘read’ (much easier to repeat commands/script against) but it's up to personal taste.

* Can we not grab to project name from the cabal file just like you do with version?

* Your ‘--html-location’ has ‘$$pkg’ in it, how come? Is that some kind of escape?

* Considering you seem to know what you're doing much more than I am, is there any way to detect cabal sandboxes?

I'm trying your makefile now (running with make -f /tmp/Makefile doc-upload) and it's giving me an error:

``` [snip] Haddock coverage: 100% ( 18 / 18) in 'Yi.Style.Monokai' Documentation created: dist/doc/html/yi-monokai/index.html rm -r yi-monokai-0.1.1.1-docs rm: cannot remove ‘yi-monokai-0.1.1.1-docs’: No such file or directory make: *** [yi-monokai-0.1.1.1-docs.tar.gz] Error 1 ```

It works fine if I comment out the ‘rm -r’. Perhaps that should be allowed to fail somehow.

...

I know, Cabal somehow was supposed to make Makefiles obsolete. Alas. Fortunately, the Makefile is only for the package maintainer, and not for the package user.

-- Peter

I think that cabal-install should really have the option to do all this rather than have people write up scripts like this, resulting in trial and error.