Getting the haddocks back (was: documentation build failing in hackage?)

On Sun, Oct 12, 2014 at 4:50 AM, Mateusz Kowalczyk
The biggest problem is that there is no official statement as to why the builds stopped and an estimate of when they are coming back.
I agree! My understanding (which is at /least/ 2nd or 3rd hand) is that the doc builds were turned off intentionally because it was a security issue, and that they are unlikely to come back. Now, assuming that is the case, how can we solve the "there is no documentation" issue? Some ideas to kick-start discussion: - Provide an option to include haddocks in the sdist bundle, and extract them on Hackage for display. - Add a 'cabal uploadHaddock' - Run all haddock builds in a VM/docker container / etc.. that mitigates the security concerns. - ??? --Rogan

On 10/13/2014 04:25 PM, Rogan Creswick wrote:
On Sun, Oct 12, 2014 at 4:50 AM, Mateusz Kowalczyk
wrote: The biggest problem is that there is no official statement as to why the builds stopped and an estimate of when they are coming back.
I agree!
My understanding (which is at /least/ 2nd or 3rd hand) is that the doc builds were turned off intentionally because it was a security issue, and that they are unlikely to come back.
Now, assuming that is the case, how can we solve the "there is no documentation" issue? Some ideas to kick-start discussion:
- Provide an option to include haddocks in the sdist bundle, and extract them on Hackage for display. - Add a 'cabal uploadHaddock' - Run all haddock builds in a VM/docker container / etc.. that mitigates the security concerns. - ???
--Rogan
What I heard is that it's the combination of the recent server migration and then running out of disk space. It seems that whoever I ask, they always have heard from someone else… AFAIK there is some movement towards having ‘cabal uploadHaddock’ already. -- Mateusz K.

Mateusz Kowalczyk wrote:
On 10/13/2014 04:25 PM, Rogan Creswick wrote:
Mateusz Kowalczyk
wrote: The biggest problem is that there is no official statement as to why the builds stopped and an estimate of when they are coming back.
Apparently, Haskell.org has a blog since yesterday, where they explain what's up with the documentation builds: https://blog.haskell.org/post/the_new_haskell_org/ Best regards, Heinrich Apfelmus -- http://apfelmus.nfshost.com

Heinrich Apfelmus
Apparently, Haskell.org has a blog since yesterday, where they explain what's up with the documentation builds:
Looks good. This post contains:
And, the doc builder is now working again, and should hopefully stay that way too.
What to do if they don't? E.g., I'm getting this: https://hackage.haskell.org/package/satchmo-2.8.1/reports/ - J.W.

use the curl api to "delete the old docs" to try to kick off a new build https://hackage.haskell.org/api On Tue, Oct 14, 2014 at 10:05 AM, Johannes Waldmann < waldmann@imn.htwk-leipzig.de> wrote:
Heinrich Apfelmus
writes: Apparently, Haskell.org has a blog since yesterday, where they explain what's up with the documentation builds:
Looks good. This post contains:
And, the doc builder is now working again, and should hopefully stay that way too.
What to do if they don't? E.g., I'm getting this: https://hackage.haskell.org/package/satchmo-2.8.1/reports/
- J.W.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

OR just upload your own using the fuuzetsu script :) On Tue, Oct 14, 2014 at 12:04 PM, Carter Schonwald < carter.schonwald@gmail.com> wrote:
use the curl api to "delete the old docs" to try to kick off a new build
https://hackage.haskell.org/api
On Tue, Oct 14, 2014 at 10:05 AM, Johannes Waldmann < waldmann@imn.htwk-leipzig.de> wrote:
Heinrich Apfelmus
writes: Apparently, Haskell.org has a blog since yesterday, where they explain what's up with the documentation builds:
Looks good. This post contains:
And, the doc builder is now working again, and should hopefully stay that way too.
What to do if they don't? E.g., I'm getting this: https://hackage.haskell.org/package/satchmo-2.8.1/reports/
- J.W.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

alternatively: the doc builders could be catching up with the huge backlog :) On Tue, Oct 14, 2014 at 12:04 PM, Carter Schonwald < carter.schonwald@gmail.com> wrote:
OR just upload your own using the fuuzetsu script :)
On Tue, Oct 14, 2014 at 12:04 PM, Carter Schonwald < carter.schonwald@gmail.com> wrote:
use the curl api to "delete the old docs" to try to kick off a new build
https://hackage.haskell.org/api
On Tue, Oct 14, 2014 at 10:05 AM, Johannes Waldmann < waldmann@imn.htwk-leipzig.de> wrote:
Heinrich Apfelmus
writes: Apparently, Haskell.org has a blog since yesterday, where they explain what's up with the documentation builds:
Looks good. This post contains:
And, the doc builder is now working again, and should hopefully stay that way too.
What to do if they don't? E.g., I'm getting this: https://hackage.haskell.org/package/satchmo-2.8.1/reports/
- J.W.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Apparently, Haskell.org has a blog since yesterday, where they explain what's up with the documentation builds: [...] What to do if they don't? E.g., I'm getting this: https://hackage.haskell.org/package/satchmo-2.8.1/reports/
Yep, I've seen the doc building failure in another package today, too I guess, despite the comments section, the blog is mostly a one-way communication channel, so we are left with the options I mentioned in http://thread.gmane.org/gmane.comp.lang.haskell.cafe/113239/focus=113269

Hmm, actually, if you'd like to be an early adopter,
you can file your tickets about haskell insfrastructure here
https://phabricator.haskell.org/maniphest/query/open/
and here is some documentation
https://phabricator.haskell.org/w/projects/haskell.org_infrastructure/
Contributions are welcome!
On Tue, Oct 14, 2014 at 6:09 PM, Mikolaj Konarski
Apparently, Haskell.org has a blog since yesterday, where they explain what's up with the documentation builds: [...] What to do if they don't? E.g., I'm getting this: https://hackage.haskell.org/package/satchmo-2.8.1/reports/
Yep, I've seen the doc building failure in another package today, too I guess, despite the comments section, the blog is mostly a one-way communication channel, so we are left with the options I mentioned in
http://thread.gmane.org/gmane.comp.lang.haskell.cafe/113239/focus=113269

Currently doc building on Hackage seems to be broken again: Three tiny packages (StateVar, ObjectName, Tensor) I've uploaded more than 2 days ago are still without documentation. I think there is a way to upload this manually, but I don't think that's a good idea, because there's no guarantee that the documentation and the package are in sync. Furthermore, I can't even find a description on hackage.haskell.org how to do that (if I wanted to). I'm well aware that most of the Haskell infrastructure depends on volunteers, so things can't be perfect and there are no SLAs etc. What I'm only complaining about is that it's totally unclear what's going on and what the current status is: Is documentation building totally broken at the moment? If yes, when will it be available again? Are there any kind of logs I can see? Are my packages in some hidden documentation builder queue? If yes, knowing its position and/or ETA might be nice. The building doesn't have to be lightning fast, but having some kind of progress indicator would really be nice. Without any visible documentation basically all packages are useless because one can't really know what what they contain, a synopsis is not enough most of the time.

1) once they finally finish updating the hackage deply to current
hackage-server master, there will be more visibility into the build process
2) i've pinged the person who is owning making sure doc builders are
working to ask them to check it out
On Sun, Oct 26, 2014 at 12:56 PM, Sven Panne
Currently doc building on Hackage seems to be broken again: Three tiny packages (StateVar, ObjectName, Tensor) I've uploaded more than 2 days ago are still without documentation. I think there is a way to upload this manually, but I don't think that's a good idea, because there's no guarantee that the documentation and the package are in sync. Furthermore, I can't even find a description on hackage.haskell.org how to do that (if I wanted to).
I'm well aware that most of the Haskell infrastructure depends on volunteers, so things can't be perfect and there are no SLAs etc. What I'm only complaining about is that it's totally unclear what's going on and what the current status is: Is documentation building totally broken at the moment? If yes, when will it be available again? Are there any kind of logs I can see? Are my packages in some hidden documentation builder queue? If yes, knowing its position and/or ETA might be nice. The building doesn't have to be lightning fast, but having some kind of progress indicator would really be nice. Without any visible documentation basically all packages are useless because one can't really know what what they contain, a synopsis is not enough most of the time. _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

2014-10-26 18:07 GMT+01:00 Carter Schonwald
1) once they finally finish updating the hackage deply to current hackage-server master, there will be more visibility into the build process
2) i've pinged the person who is owning making sure doc builders are working to ask them to check it out
Still no documentation. :-( All I'm asking for is some kind of feedback, even a low-tech textual warning on the Hackage starting page would be fine. As it is, things are very opaque...

On Wed, Oct 29, 2014 at 1:04 PM, Sven Panne
1) once they finally finish updating the hackage deply to current hackage-server master, there will be more visibility into the build
2014-10-26 18:07 GMT+01:00 Carter Schonwald
: process 2) i've pinged the person who is owning making sure doc builders are
working
to ask them to check it out
Still no documentation. :-( All I'm asking for is some kind of feedback, even a low-tech textual warning on the Hackage starting page would be fine. As it is, things are very opaque... _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
In case it helps anyone, I've started uploading the Haddocks I generate from Stackage builds to stackage.org (e.g. [1]). I'm still working out a few kinks in the process, for example, I'd forgotten to install the htmldocs PPAs on the Jenkins server, so some of the uploaded packagesets are currently missing docs for GHC-provided packages. However, it mostly works, and feedback is definitely appreciated. Of course, this only provides docs for package in Stackage, so you can consider this email as a not-so-subtle encouragement to add your packages to Stackage[2] :). [1] http://www.stackage.org/haddock/21a32ef62e5e60edb49f29e6baa31931fcbaa8dd/ind... [2] https://github.com/fpco/stackage/wiki/Maintainers-Agreement

I spoke with the person owning the doc builder, and seems like the it was a
problem with how the doc builder would order its work. It was getting
stuck trying to build docs for very old packages instead of newest first.
So that's getting fixed this week.
What textual warning are you looking for?
On Oct 29, 2014 7:04 AM, "Sven Panne"
1) once they finally finish updating the hackage deply to current hackage-server master, there will be more visibility into the build
2014-10-26 18:07 GMT+01:00 Carter Schonwald
: process 2) i've pinged the person who is owning making sure doc builders are
working
to ask them to check it out
Still no documentation. :-( All I'm asking for is some kind of feedback, even a low-tech textual warning on the Hackage starting page would be fine. As it is, things are very opaque...

2014-10-29 14:17 GMT+01:00 Carter Schonwald
I spoke with the person owning the doc builder, and seems like the it was a problem with how the doc builder would order its work. It was getting stuck trying to build docs for very old packages instead of newest first. So that's getting fixed this week.
What textual warning are you looking for?
Exactly the stuff you wrote in this email should be in big fat letters on the front page (and probably even on each page header) of Hackage. As I said, it's totally fine that things take some time (it's an open source effort relying on volunteers after all), but it's crucial that the current state of a central piece of the Haskell ecosystem is very visible. Regarding status.haskell.org: This is too indirect for this purpose IMHO. As somebody who is not closely following each and every email on the various Haskell mailing lists, how should you find out that this site even exists? It's not linked from Hackage AFAICT.

On Wed, 2014-10-29 at 12:04 +0100, Sven Panne wrote:
All I'm asking for is some kind of feedback, even a low-tech textual warning on the Hackage starting page would be fine. As it is, things are very opaque...
+1 It is easy to manually upload docs to Hackage, but at least you need to know that doc builder is not working. Probably status.haskell.org should reflect doc builder status?
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

+1. Why all the secrecy ?

There is no secrecy. Just no one as yet volunteering to write the tools
needed to create the visibility into the doc builder queue progress :-)
Go on #haskell-infrastructure on freenode if you want to help make things
happen. Nothing else matters :)
On Oct 29, 2014 10:09 AM, "Simon Michael"
+1. Why all the secrecy ?
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

2014-10-29 17:37 GMT+01:00 Carter Schonwald
There is no secrecy. Just no one as yet volunteering to write the tools needed to create the visibility into the doc builder queue progress :-) [...]
Nobody asked for a fully automated tool/status page, a plain visible note on Hackage done by hand would be enough. Problems with the doc builders are not new, see e.g. https://github.com/haskell/hackage-server/issues/244 or https://github.com/haskell/hackage-server/issues/265.
participants (10)
-
Carter Schonwald
-
Heinrich Apfelmus
-
Johannes Waldmann
-
Mateusz Kowalczyk
-
Michael Snoyman
-
Mikolaj Konarski
-
Rogan Creswick
-
Simon Michael
-
Sven Panne
-
Yuras Shumovich