For Hacakge 2, the distro ecosystem hasn't had quite as much tooling put into it as, say, documentation, and it has changed a little bit from Hackage 1. Previously, there was a cron job that, among tasks like updating documentation, also downloaded distro info and ran <http://code.haskell.org/~ross/hackage-scripts/DistroInfo.hs> on it; the resulting files were read when generating the package page in <http://code.haskell.org/~ross/hackage-scripts/PackagePage.hs>.

No such job exists now. However, there is something to replace it in Hackage 2. It is not a cron job; it's more push than pull, actually. Distro maintainers can upload distro information directly to Hackage (either manually, using a form, or programmatically using something like curl). It is currently not exposed or documented that well, unfortunately, and it needs a last 10% of work to be fully functional. If you'd like, I think I'd be able to do it this week.

So what does this new API look like? Well, this is the list of distros we know about:
http://hackage.haskell.org/distros/
And the set of packages we know about for Fedora:
http://hackage.haskell.org/distro/Fedora/packages
http://hackage.haskell.org/distro/Fedora/packages.csv
The status for a given package:
http://hackage.haskell.org/distro/Fedora/package/text
Finally, shown on the package page itself:
http://hackage.haskell.org/package/text

There are per-distro ACLs so that the above information can kept up to date by authorized accounts. There are a few issues with this system currently, such as not being able to edit these ACLs over HTTP (!). Once this is fixed, it will be possible to do:
1. Bulk update: PUT a csv formatted like the one above, which is similar to the current format (the current format is a tuple, like http://people.debian.org/~nomeata/cabalDebianMap.txt). This will wipe the previous set of distro statuses for a given distro.
2. Add/update the distro status of an individual package: this is also a PUT, using two form fields (e.g. version=0.11.2.3&uri=https://apps.fedoraproject.org/packages/ghc-text)
3. Delete the distro status of an individual package: this is a DELETE, perhaps meaning that the package is no longer offered

Unlike the previous system, using this *does* require Hackage credentials, and if this data comes from an automatic task, a bot account could possibly be used. It depends on what will work best with the current flow.

Best,
Matt



On Mon, Oct 14, 2013 at 12:08 AM, Jens Petersen <juhp@community.haskell.org> wrote:
Hi, it is great that Hackage 2 finally was deployed in production. :-)

How often does Hackage 2 update/pull distro data?

Hackage 1 had a cronjob I believe which ran every few hours I think.
But the new Hackage does not seem to pull often/regularly.

For Fedora I updated the data on 18 Sept and then 26 Sept.
The update from 18th Sept is reflected in Hackage but not 26th Sept.
Checking more I guess the 18 Sept was still during Hackage1
so it looks like Hackage2 has not updated it distro data since
going into production?

Is there a cronjob setup on the new hackage server currently
for pulling distro package data?

Thanks, Jens

_______________________________________________
cabal-devel mailing list
cabal-devel@haskell.org
http://www.haskell.org/mailman/listinfo/cabal-devel