So you're saying we need to add a db proper to hackage2 server, like SQLite or Postgres so as to make it more performant for interesting features? What's needed to do that? On Monday, October 14, 2013, Matthew Gruen wrote:

Hey, thanks for taking an interest in this. There is kind of a space-time-featureset tradeoff with revdeps, unfortunately. It seems to me like the challenge doesn't come so much from indirect deps, but rather from the sheer number of versions there are! There are 5600+ packages and 33800+ versions, and the set of dependencies of foo-1.2 vs foo-1.3 can be completely different. Not to mention, when you look at the package page for a particular version, you may want to see the revdeps *for that version*, and at this point there are a billion possible combinations. npm's UI, in contrast, has an "always in HEAD" kind of feel.

If you are interested in how the disabled implementation worked, by the way, I have a post here (as gracenotes): https://github.com/haskell/hackage-server/issues/40

So revdeps is great, and there are many different killer uses for it, such as popularity metrics and finding upper version bounds which need updating. In sum, these require keeping around a lot of data. Because there's so much, acid-state is probably not a good place to store it. Not to mention, it's time-consuming to generate it from scratch and space-consuming to keep around data structures to incrementally update it. So there are some tool limitations, but it seems fundamentally tricky to do frugally.

Matt

On Mon, Oct 14, 2013 at 12:20 AM, Jens Petersen < juhp@community.haskell.org> wrote:

...

Not really complaining :) but one of the things I had been looking forward to with Hackage2 was the display of reverse-dependencies, but I gather it was disabled for now (because it loads the server too much when updating the data iirc?).

I was thinking perhaps a compromise would be to show only direct reverse-dependencies rather than including all indirect dependencies too. Perhaps that would make it scale better? I think direct dependency information is the most useful anyway (and also what packdeps provides).

Before doing anything I thought I would ask here first if this makes sense and should be work? I for one would still really like to see basic reverse dependencies info listed on hackage (I noticed yesterday that npm has them too).

Jens

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

It seems to me that all we really need is a list of all first generation revdeps on a given package (any version), each specified maybe as (in pseudodata): type VersionRange = (Version,Version) type RevDep = (PackageName,[VersionRange]) RevDeps = [RevDep] (alt M.Map PackageName [VersionRange]) RevDepStructure = M.Map PackageName RevDeps I can see how things get uglier if we instead do: RevDepStructure = M.Map PackageName (M.Map Version RevDeps) but I think the latter structure is potentially too granular for most uses? --Gershom On Mon, Oct 14, 2013 at 2:44 PM, Duncan Coutts wrote:

On Mon, 2013-10-14 at 12:18 -0400, Carter Schonwald wrote:

...

So you're saying we need to add a db proper to hackage2 server, like SQLite or Postgres so as to make it more performant for interesting features? What's needed to do that?

I don't think that's needed at all.

What we need is to look carefully at what we want from a revdeps feature and then look carefully at what info should be stored (ie cached) and what should be calculated on demand.

It just needs someone to spend some time on the feature, there's no need to panic about the data storage (especially before looking at what is actually needed).

Duncan

...

On Monday, October 14, 2013, Matthew Gruen wrote:

...

Hey, thanks for taking an interest in this. There is kind of a space-time-featureset tradeoff with revdeps, unfortunately. It seems to me like the challenge doesn't come so much from indirect deps, but rather from the sheer number of versions there are! There are 5600+ packages and 33800+ versions, and the set of dependencies of foo-1.2 vs foo-1.3 can be completely different. Not to mention, when you look at the package page for a particular version, you may want to see the revdeps *for that version*, and at this point there are a billion possible combinations. npm's UI, in contrast, has an "always in HEAD" kind of feel.

If you are interested in how the disabled implementation worked, by the way, I have a post here (as gracenotes): https://github.com/haskell/hackage-server/issues/40

So revdeps is great, and there are many different killer uses for it, such as popularity metrics and finding upper version bounds which need updating. In sum, these require keeping around a lot of data. Because there's so much, acid-state is probably not a good place to store it. Not to mention, it's time-consuming to generate it from scratch and space-consuming to keep around data structures to incrementally update it. So there are some tool limitations, but it seems fundamentally tricky to do frugally.

Matt

On Mon, Oct 14, 2013 at 12:20 AM, Jens Petersen < juhp@community.haskell.org> wrote:

...

Not really complaining :) but one of the things I had been looking forward to with Hackage2 was the display of reverse-dependencies, but I gather it was disabled for now (because it loads the server too much when updating the data iirc?).

I was thinking perhaps a compromise would be to show only direct reverse-dependencies rather than including all indirect dependencies too. Perhaps that would make it scale better? I think direct dependency information is the most useful anyway (and also what packdeps provides).

Before doing anything I thought I would ask here first if this makes sense and should be work? I for one would still really like to see basic reverse dependencies info listed on hackage (I noticed yesterday that npm has them too).

Jens

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

-- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel