Oh, great, the email _did_ go out on the mailing lists (what with haskell.org being down I wasn't sure it would). Don Stewart writes:

ivan.miljenovic:

...

Thomas Bereknyei are currently re-writing fgl (just about completely from scratch) and we plan to make an initial release to get feedback on the API in the next few weeks.

However, I'm sending this email out now to warn people that I highly doubt any code that was written for the current version of fgl (http://hackage.haskell.org/package/fgl-5.4.2.2) will work with the new version.

How about you give the library a different name then -- so as not to break all those programs?

A complete rewrite with a new maintainer: fgl-awesome

We considered giving it a new name (fgl', etc.) but figured that in the long term this wouldn't be advantagous. We feel that the situation is analogous to QuickCheck: when the new version came out most people kept using the old one until slowly the momentum shifted and more people started using the new version (without checking in depth, Roel's Hackage mirror reports QC-2.x now has 153 reverse dependencies as opposed to 127 reverse dependencies for QC-1.y). If we changed the name, then the "emotional attachment" that the Haskell community has to FGL being the de-facto graph library means that people would keep using the old version. Whilst we also face the possible problems of people not liking the old version and thus automatically dismissing the new version, I think this is a much more unlikely scenario. The overall philosophy is remaining the same: we're just updating the implementation (letting people pick the Node type rather than hard-coding it to Int, letting people constrain the label types and also providing more scope for optimisations). As such, the new version is incompatible with the old one; however, if you're only _using_ FGL (as opposed to making a new instance of the graph types), then the actual changes to using the new version should be minimal (especially if you're using it internally in which case you can hard-code the various types in and not have to worry about the type family usage at all). As I said, I've directly emailed the maintainers of the packages that have open-ended dependencies on FGL and thus would have compilation problems when we release the new version[s]; 7 out of 12 of those maintainers have already responded within less than 24 hours so I don't think this is likely to be a problem unless some new package is uploaded in the near future without proper ranged dependencies. So as to give you even more of a heads up, here are some more overall plans Thomas and I have for FGL: * FGL will still focus on inductive graphs; however the two current classes are being combined since it doesn't make much sense in general to have a graph that can be decomposed using match but can't be composed using & (there might be specific cases where this is true such as considering the Cabal package dependency "graph"; however this is likely to be defined internally within some project so users should just be careful about how they use it to ensure they don't add arbitrary nodes/edges that don't make sense; note that the new Show and Read instances take advantage of this by using Contexts as the basis of this instance). * The ability to have proper Eq, Show and Read instances for graphs with pre-defined helper functions implementors can use (currently Data.Graph.Inductive.Tree has a "pretty" output for Show but no Read instance, and graphs cannot have equality due to overlapping instance problems). * Splitting the library up into the base classes + sample instances (the fgl package) and a separate fgl-algorithms package (analogous to Dan Doel's vector-algorithms; this will be comprised of what is currently in the Data.Graph.Inductive.Query.* modules as well as the Data.Graph.Analysis.Algorithms.* modules in my Graphalyze library. The Data.Graph.Inductive.Graphviz module will also be scrapped in favour of my graphviz library. * By default, fgl will ship with two default instances: the one currently in Data.Graph.Inductive.PatriciaTree and also a generic Map-based one that lets people choose their own Node type (i.e. the key type in the Map). Any other suitable instances that we can think of (e.g. Thomas has the beginnings of a vector-based one) we'll ship in separate packages (e.g. fgl-vector). If anyone has a good reason to object to any of these plans, we are willing to be persuaded out of them. This is why the 6.z series of fgl will be "technology previews" to slowly build up what FGL does and gauge the reaction of the Haskell community. -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

Don Stewart writes:

ivan.miljenovic:

...

We considered giving it a new name (fgl', etc.) but figured that in the long term this wouldn't be advantagous. We feel that the situation is analogous to QuickCheck: when the new version came out most people kept using the old one until slowly the momentum shifted and more people started using the new version (without checking in depth, Roel's Hackage mirror reports QC-2.x now has 153 reverse dependencies as opposed to 127 reverse dependencies for QC-1.y).

Just remember: you are now the maintainer of a package in the Haskell Platform, and that has some burden on ensuring stability and safety. API-breaking changes are fine, as long as well planned, and it may be up to 12 months before the HP can adopt an API changing release of a package.

Oh, we know that: we envisage it being a few months before anyone starts using the new version of FGL. One change I might make in a released version before then is if I'm successful during AusHack in writing at least a start towards the generic graph classes, we might make a 5.x release of fgl that has the instances of the generic classes and then port at least graphviz over to using the new generic classes. I realise that these changes as well won't necessarily go into the HP straight away, but it should help improve cross-library usage down the track. <off-topic> I was discussing the state of the graph libraries with Edward Kmett yesterday, and here's what I said as what I envisage happening over the next year or so: * The generic graph classes will form a base library for accessing and using graphs. It won't define any graph types itself, but will serve as a base compatability layer between libraries. The focus of this will be a class I'm currently calling GraphLike which let's you access the vertices and edges of any type that matches the formal definition of a graph that it's a set of vertices V and a set of edges E such that forall e \in E, there exists two vertices v1, v2 \in V (not neccessarily distinct) such that there are two mappings "source e = v1" and "target e = v2" (or whatever you want to call them) between each edge and the set of vertices. Basically, this class is an accessor class, so it will work for things like package dependency graph in Cabal, etc. for graph-like types which it might not make sense to add or change vertices and edges in the graph. Whether there will also be a class for adding/removing elements is currently up in the air, as this might not make sense for all graph types. As such, those functions that don't need to edit graphs (e.g. graphToDot in graphviz) will be able to just use these. * The set of FGL libraries will form an initial to intermediary layer of types. If your problem isn't large and the inductive nature is applicable, then FGL will probably be your best bet. * Edward's co-monadic graph library will be an intermediary to advanced layer; it sounds quite interesting but might not be as easy to use due to the types changing when you add a vertex, etc. * fasta on #haskell{,-blah} has suggested that I read through the Boost Graph Library documentation and consider writing a graph library based upon that (and hints that he's once written a high-performance graph library in Haskell on this model). As such, I'll have a look at this down the track during my copious free time :s (Edward thinks it might even fit in to his co-monadic approach, so we'll see). What I'm after here is more "sharing" between graph types and libraries: rather than defining something on just one specific graph type, generalise it as much as you can (to avoid writing a new Graphviz transformation library for every graph type for example). </off-topic> -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

Hello all While new libraries develop at pace, their documentation rarely does; so I'd have to disagree with John's claim that re-naming libraries makes development by new users harder. I'd argue that having tutorials not work for later revisions is more confusing than having various packages doing the same thing. I'd also contend that beginners are better off lagging behind the cutting edge and using Parsec 2, QuickCheck 1, Haskore-vintage, as the earlier version all have comprehensive documentation - Parsec 2 and Haskore have extensive manual/tutorials, QuickCheck 1 was small enough that the original QuickCheck paper covered its use. An advantage of separate names (both for the package name and module name-space) is that its easier to have both packages installed at the same time - the old one to work with while learning the package, the new one if other installed packages depend on it. This can still be done with package-hiding but its less straight-forward. Best wishes Stephen

On 10 June 2010 12:38, sterl wrote:

There's a big range of issues here, and to be honest I'm not sure if our ability to distinguished between them is helped by the title of this thread, which somewhat begs the question. That is to say, it isn't clear to me that calling the proposed changes to the fgl "rewriting a library" is necessarily accurate -- it seems more the case that these are incremental improvements of a library that require breaking API changes.

Except it is a re-write in the truest sense of the word: we started completely from scratch. We did compare our API with the current API (in an attempt to keep function names, etc. the same where possible because I'm hopeless at choosing names) but we didn't exactly take the class as-is and then change it. On the other hand, we were both familiar with the current version of FGL and how it's layed out, so there's probably some implicit influences from there as well. Oh the other hand, it can also be considered as incremental improvements: we wanted to keep the terminology and fundamental concepts as similar as possible to avoid having a jarrring change in how its used. Instead, we focused on improving the current version: using explicit data types for Context and Edge (for why Edge needs a data type of its own, read the Graph section of "Fun with type functions" by Oleg, SPJ and Chung-chieh Shan) rather than tuple aliases; allowing restrictions on the label types (though we've just come across a problem where this doesn't play nicely with mapping functions); increasing the scope for per-instance optimisations, etc. So in a sense we did a re-write that happened to come close to the current definition. This is not to say that this is because the current API is close to an ideal perfect API, but rather because we were focussing on developing something _like_ the current version without worrying about compatability too much.

So on the concrete issue at hand, I'd be for the new fgl version being developed under some new provisional name, and taking pains to provide a compatibility layer where possible. Then, after we see what the changes really are, coming to some informed decision on whether to rebrand it as the real fgl version 6. If so, the old stable fgl can be put up on hackage as fgl98, which lets packages which want to stick with it do so while avoiding any possibility of the dread diamond dependency.

Considering the "rename the old version" issues first: * It won't solve the problem of people not specifying correct constraints on the version of fgl used, since it means they'd have to edit their dependencies to use fgl98 or whatever anyway. * Calling it "fgl98" is on a slippery slope: what happens when GHC-6.14 comes out with Haskell2010 support? Do we then release an fgl2010 version as well? (I believe Ross brought this problem up already). * I'm wanting people to move _off_ of the old version of fgl. The only real advantage (though how practical this will be in the real world is debateable IMHO) is that the current version doesn't use any extensions whereas the new one does (and they're needed to provide the asked-for functionality of letting instance writers constrain the types of labels - i.e. the reason why Set isn't an instance of Functor - and to have custom Node types). Since (like it or not) for the most part when people write Haskell code they use GHC and GHC supports these extensions, I do not think this is that much of a problem (I am open to being convinced otherwise about this though; I think it would be _great_ if there were other Haskell compilers that were as good as if not better than GHC in terms of runtime, etc. ... until I start considering how to manage two different compilers in Gentoo, etc. :p). As for having a temporary name for the testing releases, I am open to doing so, but this in affect pollutes the package name-space with packages that shouldn't/wouldn't be used. I would prefer to host it elsewhere and just tell people to grab a copy and see what they think rather than use a temporary name and then change it later when its "stabilised". It would be preferable IMO that if we were going to change package names then it should be done once and then not changed again.

More broadly, we have to accept that breaking API changes are an irritating but necessary fact of life. As much as the parsec and quickcheck issues have caused some modest pain, there's been equal hassle from things like the strictness behavior of binary, or even the type change in tagsoup. Splitting out Category from Arrow caused me probably the most hassle. In retrospect it was the right thing to do. But how it was done was particularly abrupt and painful. Exceptions got it right in pretty much every respect, but still migration necessarily took some work. We want our packages to grow, including our core packages. Otherwise we get fragmentation and duplicated effort. When we want to grow, but don't know exactly how, then we get experimentation. But experimentation without some organization can lead to the wrong sort of fragmentation -- like the mtl mess, whose resolution now thankfully seems to be in hand.

Right. It's this abrupt change that I'm trying to avoid by publically warning people ahead of time that they should fix their package dependencies and then have a series of preview releases to see what people think. I think to an extent the base-3 to -4 transition coupled with exceptions was a jarring/coming-of-age point for the Haskell community in terms of dependencies. We'd already had the split base issue for base-2 to -3, but that mainly involved using the split-base flag in our .cabal files and adding dependencies on containers, arrays, etc. where needed (and could almost have been automated). However, with the transition to base-4 we really started to get serious about proper versioned dependencies (which is what I was trying to avoid by starting this whole chain of emails) because developers were blindly specifying either just base or "base >= 3" (and in some cases silly things like "base < 5"; this has also occurred in packages that came out after GHC 6.10.1 was released by people that should know better resulting in packages that didn't build with base-4). In cases like QuickCheck, Parsec, etc. this version dependency issue is only half the problem (the other half being diamond dependencies). But with fgl, the problem isn't that severe since there are very few libraries that use FGL; most usages seem to be for applications. As such, the diamond dependency problem isn't that much of a consideration in this case. However, the mtl vs. transformers issue is to an extent a problem here: if users have both versions of fgl installed (which ghc-pkg lets you do), then there will be issues with developers trying to use one but not the other. For this problem, the best solution is probably to make a concerted effort with all package maintainers that use fgl to do a mass upgrade release at the same time the new version of fgl is publically released (in terms of actually being worth using rather than "hey, how about we do it this way? you happy with this now?" preview releases).

Some lessons I think we can learn from the past about changes to widely-used stable APIs: * Clear and documented upgrade paths.

We're planning on writing suitable upgrade documentation.

* Preferably a compat layer (Exceptions and Parsec both did a killer job with this).

Probably not going to happen here unfortunately. However, there are a few pseudo-compatability options that can help resolve this: * When I get the generic graph class written (in about a months' time at AusHack), people should start migrating their code to as low a class in the hierarchy as they can (so if they don't need the inductive nature of fgl, then there's no reason for them to specify doing so in their type signatures). * If you're writing an application rather than a library for graphs, pick an appropriate graph type and stick with it (using various type aliases where necessary). That way, rather than having to have polymorphic type signatures with type family notation (so stuff like "Num (EdgeLabel g), NodeLabel g ~ ()") you can just use the actual type or an alias of the graph type you're using. We might be able to provide this type of alias notation for a default graph type, but it will probably require a different module to be imported than what is currently use; i.e. new fgl still won't be a drop-in replacement for old fgl.

* No, or demonstrably minimal performance regressions.

In this case, the actual library itself is just a type class with a couple of default instances so there should be no regressions. In fact, we're increasing the scope of per-instance optimisations so the default graph type (based upon what is currently in Data.Graph.Inductive.PatriciaTree) may end up being faster in some situations (e.g. mapping over the labels).

* Strong release notes and other documentation, either duplicating or supplementing what existed prior.

Definitely. We're even considering using the new instance-level documentation that Haddock 2.7 provides.

* For particularly long-lived stable APIs, forking off a maintenance-mode-only version may make good sense, especially when the subset of language extensions used differs significantly.

I'm hoping that in this case that won't be neccessary. What might happen is that along with the preview releases (whether fgl-6.x or otherwise), we might slowly start backporting some features (e.g. usage of the generic graph class library) to the 5.y series.

Some lessons to us API consumers who write somewhat-less-core packages: * Upper version bounds.

Pretty please? :p I'm really looking forward to when Cabal supports PVP opt-in so that Hackage will complain if you don't have proper bounds on packages that follow the PVP.

* If at all possible, don't move to the fancy new thing until the fancy new thing is fully baked, and on track to widespread adoption. (early adopters of new mtl implementations, I'm looking at you :-))

To an extent, this is a bit of a mixed bag; in cases like mtl vs transformers, if we didn't have the early adopters then we would have no impetus for _anyone_ to use the new version. That said, in this case, DON'T USE THE NEW VERSION OF FGL UNTIL WE SAY IT'S OK (probably the 7.x series)!!!!!!!!!!!!!!!!!!!!!!!

* If at all possible, try to stay compatible with at least the prior GHC version as well as the current.

At the moment, this is rather easy to do unless you want to take advantage of (or are being bitten by) the new locale-aware stuff in GHC 6.12.

* Don't pull in big packages for small reasons unless really necessary -- minor duplication of trivial code is often the lesser evil.

I would argue that for this reason big packages may want to consider being split up into smaller, more manage-able smaller packages. For example, we're going to split off the Data.Graph.Inductive.Query.* modules into an fgl-algorithms package to make them easier to maintain, etc.

Some lessons for folks exploring new variants: * Don't step on already-used module names.

This depends on the situation; transformers (+ monads-fd) was meant to serve as a drop-in replacement for mtl; this, however, obviously causes problems when people are trying to use one or the other in ghci and have both installed. In this case, for fgl we're wanting to do a library upgrade, so IMO it makes sense to use the same module names.

Some technical issues that will help as time goes on (many already underway): * Depreciation of packages on hackage/redirects. (Makes it easier to establish upgrade / migration / transition paths).

There is already some support for this: packages on Hackage can be explicitly marked as being deprecated and as such won't appear on the default package listing (IIUC) but will still be pulled in by cabal-install if necessary.

* Tree organization of packages on hackage. (Reduces the noise generated by lots of small packages, and so encourages splitting things out).

Not sure what you mean by this. If you're talking about per-category trees, this wont' quite work: some packages will appear in multiple categories (e.g. data structures + graphs) and as such this won't be a tree.

* Wikilike documentation features on hackage (lets users contribute and share upgrade paths, etc. more directly and simply -- hopefully will help with community documentation of packages in general).

Coming soon (as soon as someone codes it)!

* The "local usage" annotation for cabal files to help avoid the dread diamond dependency.

I understand that this requires support in ghc-pkg first.

* A DSL to describe transforms of Haskell programs for at least simple API migrations. Yes, this is a bit more "out there" but it's a great space to explore. The upside is not only better tools to help authors migrate their code, but a strong representation of what exactly the API changes are. So even if the spec language describes things that can't be applied automatically, it can still formalize what authors need to do. A standard format for an API change log as a hackage plugin would be a good start to this.

I would be wary of anything that tried to automagically upgrade my code, since there would most likely be subtleties that it won't get right. -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

On Tue, Jun 8, 2010 at 8:08 AM, Don Stewart wrote:

(... There have been a few cases of major API / rewrites to famous old packages causing problems, including:

* QuickCheck 1 vs 2 * parsec 2 vs 3 * OpenGL ...)

(... * No additional breakages are introduced. ...)

Oh lord yes... just call it fgl3 and leave the fgl package alone. This is a source based community here... so you take a package that has a dependency on another library and you go out and get that to cover the dependency and the API is not the same!!! AND especially if that might be the only thing you will ever use that lib for ... and you have to stop and rewrite the original.. and as someone else said with maybe documentation of that API that is not maybe finished or... NO ... At that point the person will probably just DISCARD the compile on the lib or program that had the dependency.. rather then put the effort in to learn an entire API that doesn't match up.. BAD IDEA!! cheers, gene

On Tuesday 08 June 2010 20:21:54, Gregory Crosswhite wrote:

Or you just put an upper bound on the versions of the fgl library that your program will build against, as you should be doing anyway, and then nothing breaks.

Cheers, Greg

Right. At least, nothing breaks until the next compiler release makes the old library break. But a new package name wouldn't help with that.

The problem is that nothing breaks immediately. Then someone else comes along and transitively depends on your package and on another package, which depends on the newer version. Your users wind up with strange conflicts like that if they are using Parsec 3 they can't use HTTP. Or if they use fc-labels they can't use any library that internally uses mtl, because fc-labels uses transformers. Or worse they want to use a library that used fc-labels internally with another library that used mtl internally. It fragments the library base that you are able to use. Version caps are not the answer. -Edward Kmett On Tue, Jun 8, 2010 at 2:21 PM, Gregory Crosswhite < gcross@phys.washington.edu> wrote:

Or you just put an upper bound on the versions of the fgl library that your program will build against, as you should be doing anyway, and then nothing breaks.

Cheers, Greg

On Jun 8, 2010, at 11:08 AM, Gene A wrote:

On Tue, Jun 8, 2010 at 8:08 AM, Don Stewart wrote:

...

(... There have been a few cases of major API / rewrites to famous old packages causing problems, including:

* QuickCheck 1 vs 2 * parsec 2 vs 3 * OpenGL ...)

...

(... * No additional breakages are introduced. ...)

Oh lord yes... just call it fgl3 and leave the fgl package alone. This is a source based community here... so you take a package that has a dependency on another library and you go out and get that to cover the dependency and the API is not the same!!! AND especially if that might be the only thing you will ever use that lib for ... and you have to stop and rewrite the original.. and as someone else said with maybe documentation of that API that is not maybe finished or... NO ... At that point the person will probably just DISCARD the compile on the lib or program that had the dependency.. rather then put the effort in to learn an entire API that doesn't match up.. BAD IDEA!!

cheers, gene

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Tue, Jun 22, 2010 at 4:54 PM, Gregory Crosswhite < gcross@phys.washington.edu> wrote:

There is no reason that your program couldn't link to multiple versions of the same package so that each library can access the version that it needs. In fact, GHC already does this, doesn't it? For example, I use a mixture of libraries in my programs that link to QuickCheck 1 and QuickCheck 2, and this works just fine.

This works fine as long as no detail of the embedded library leaks into the public API. QuickCheck is typically the least painful library to mix, since you don't typically use the quickcheck properties from multiple quickcheck versions drawn from other packages at runtime.

The only problem I've had is with cabal, which when resolving dependencies seems to only be able to pick out one version of a package; in some cases instead of running "cabal install A B" where A and B depended on different versions of the same package (QuickCheck) I had to instead separately run "cabal install A" and "cabal install B". This isn't a big deal, but I could imagine cases where it could fail to automatically install a package entirely due to conflicting version requirements. This, however, is not because there is an intrinsic problem with installing multiple versions of a library, but simply because cabal sometimes seems to get confused about what it needs to do.

cabal is the only mechanism that the vast majority of Haskell-users know how to use these days. Resolving diamond dependencies safely relies on knowing tha tthe use of different libraries is entirely internal to the library in question -- a detail that is not currently exposed through the cabal file. You can use PackageImports to try and hack around common package names at least in GHC, but it then further confuses purpose and provenance.

So in short, I see no problem with there being multiple versions of a package floating around, and to the extent that an implementation of something can't handle this it seems like this is arguably a bug in that implementation rather than a bug in the package system for allowing the possibility.

There are multiple potential implementation semantics that can be assigned to a diamond dependency. The types could be incompatible. They could be compatible, and the most recent version should be used by all (in case of minor API changes). They could be somewhere in between. I suppose where we differ is in how big of a concern we view 'just cabal having a problem with it' is. All I can say is that every time there has been a major API change where half the community hasn't moved, it has been a practical problem. It become yet another implementation detail that every subsequent developer has needed to consider, and providing support and instances for both is impractical. So while in theory, just bumping the major version number would be sufficient, in practice, I think picking a new package name and namespace would: * fit more closely with the spirit of a ground-up redesign * allow it to succeed or fail on its own merits * avoid torturing new users with cabal install warnings they won't understand * allow libraries that want or need to be agnostic with respect to the change the ability to provide instances for both libraries, providing a smoother upgrade path for their users. Which in the end, seems to be a net positive over: * requiring folks to use one version of fgl entirely internally without exposing it through their API or splinter their user base * causing cabal to panic * ultimately adding yet another hack to the hackage config which indicates implied upper bounds on the stable version of the package, since the current fgl is part of the haskell platform and the new design will not be stable for some time If fgl wasn't part of the haskell platform or the changes were less radical, the balance of net good might go the other way. -Edward Kmett

On Wed, Jun 23, 2010 at 2:57 PM, Gregory Crosswhite < gcross@phys.washington.edu> wrote:

On 6/23/10 2:13 PM, Edward Kmett wrote:

On Tue, Jun 22, 2010 at 4:54 PM, Gregory Crosswhite < gcross@phys.washington.edu> wrote:

...

There is no reason that your program couldn't link to multiple versions of the same package so that each library can access the version that it needs. In fact, GHC already does this, doesn't it? For example, I use a mixture of libraries in my programs that link to QuickCheck 1 and QuickCheck 2, and this works just fine.

This works fine as long as no detail of the embedded library leaks into the public API. QuickCheck is typically the least painful library to mix, since you don't typically use the quickcheck properties from multiple quickcheck versions drawn from other packages at runtime.

Yes, but if details of the package you are using are "leaking" out into the interface then you will have the same kind of problems whenever that package conflicts with any other package, regardless of whether the conflict is with a package of the same name. For example, for a while there was a conflict between mtl and transformers because they shared a package name, and the fact that the two packages had different names didn't make the problem any better.

Yes, and that problem still isn't resolved in another since, since they share the same module names, but as of yet, still provide an incompatible API. I can't (yet) provide 'RightSemiNearRing' instances that work with both the monad transformers from transformers and mtl without deep mojo. The resolution there seems to be to bring transformers and mtl into close enough alignment that we'll be able to finally release a version of the mtl that just imports transformers and monads-fd, and provide a set of guidelines about the fact that in the switch to the next major version of mtl, the non-transformer versions of the monad-transformer stack are just type aliases. In that case the APIs are close enough that with a few breaking changes to existing users on each side, the APIs can be reconciled and the community unfractured. That said, we've had this plan waiting in the wings for months. ;) But cabal can see with exactly which packages each of the dependencies

requires, right? So what is stopping it from just walking through the dependencies and constructing the dependency graph? It should have all of the information it needs to do this.

This becomes somewhat tricky. You can do this more or less with data types and classes, but with instances it is less clear how to do so. Instances (necessarily) kind of silently infect your public interface, and so this form of reasoning is at best global, not local. There has been some chatter about splitting up build dependencies into internal and external dependencies, although I don't know how serious it was, but that would be a move in this direction. To the extent that the full information that cabal needs exists and yet it

is not capable of recognizing this, I would view this as a bug in cabal that we should fix, rather than deciding just to live with this bug and limiting ourselves to a subset of the package dependency functionality.

Regardless, it is unlikely to be fixed before Ivan goes to release his shiny new type-family-driven FGL. =)

So in short, I see no problem with there being multiple versions of a

...

package floating around, and to the extent that an implementation of something can't handle this it seems like this is arguably a bug in that implementation rather than a bug in the package system for allowing the possibility.

There are multiple potential implementation semantics that can be assigned to a diamond dependency. The types could be incompatible. They could be compatible, and the most recent version should be used by all (in case of minor API changes). They could be somewhere in between.

Yes, but again this will happen whenever you use two packages that conflict, regardless of whether they just happen to have the same name or not, as it did for a while with mtl and transformers. Renaming fgl to newfgl won't actually make this situation any better.

If we really are worried so much about these kinds of conflicts, then the real solution is to make sure that none of the modules exported by the new package share the same name as the modules in the old package. And if one is going to do that anyway, then from the perspective of resolving conflicts there isn't any additional benefit to also renaming the package.

Actually, once you've given them different module names keeping the same

[...] package name _is_ an impediment. Because with different package names you could import both and provide instances for both, say, fgl's Graph, and for Ivan's very different type-family driven Graph, but you needlessly sacrifice that upgrade path for your users if you force both packages to have the same name. Another very different consideration is that Erwig's old fgl is likely not going away any time soon. As far as I can tell nothing in the Haskell platform currently exploits type families and fgl is already in the platform. Getting the new library into the platform would take quite a while, even if it were available fully formed, debugged, and had an installed user base today. One much weaker consideration is that out of the 23+ direct dependencies on fgl, fully half of them don't bother to specify an upper bound on the fgl version and would break immediately. That said, those packages are out of compliance with package versioning policy. =) I agree with your point that perhaps cabal should be fixed to support explicit or implicit handling of internal dependencies. I just think that the practicalities in this situation point to fixing the problem with the tools in the room, rather than waving our hands and making it Duncan's problem. ;) -Edward Kmett

Edward Kmett writes:

One much weaker consideration is that out of the 23+ direct dependencies on fgl, fully half of them don't bother to specify an upper bound on the fgl version and would break immediately. That said, those packages are out of compliance with package versioning policy. =)

From what I can tell, fgl is more often used for internal projects or executables rather than to create actual libraries. So the whole

I've contacted all the maintainers of those packages, and have received replies from all but two maintainers promising to fix theirs (still haven't heard from Aditya Mahajan for teams and John Morrice for esotericbot). Note that I'm just as guilty in this regard as I have three packages that depend on fgl without upper bounds... >_> I don't know how many of them have released fixed versions yet (I know I haven't), but with the whole two versions of FGL needed thing: we're going to work hard to help maintainers of packages that use FGL to upgrade to the new version once we're satisfied with it and decide to make the release official. As such, we shouldn't have any problems of packages depending transitively on two versions of FGL because most if not all of them will get upgraded. With FGL, we also have the situation where we have a "famous" library that isn't actually used much: * Out of those 23 direct dependencies on Hackage, 3 of them were by old versions of packages (i.e. they no longer use fgl); these are mpc, jhc and lhc. * One is officially deprecated (scenegraph). * 10 are pure libraries (note that some of these have no reverse dependencies, or if they do it's only by a few packages) (note that I included a package here if its executable seemed to only be for tests). * 6 are executables only. * 3 seem to be a library + an executable; only one of these has a reverse dependency and even then by one other package. Out of the 25 indirect reverse dependencies: * 20 (some of which are older versions only and thus not the latest) may transitively depend upon the old version of mps (either directly or via the hack library); because the dependencies on mps, etc. are open-ended for many of these it's difficult to tell if a new install will still transitively require fgl (as in to actually get it to work). * 1 dependency on cabal-macosx * 2 dependencies on lambdacube-engine (other lambdacube projects) * 2 satchmo projects depending (transitively in one case) upon funsat. As such, if we only consider transitive dependencies, then this issue of one package using new and another old fgl is a non-issue as in all cases as there's only a single point of contact with fgl. problem of worrying about multiple versions appears to be (no offense to anyone) scare-mongering (or looking for possible problems due to bad experiences with parsec, etc.). With the 4 transitive dependencies on fgl above, I'm not sure how many of them actually see any aspects of fgl or if its all hidden inside the internal API of whatever package they use. Thus, once packages have proper reverse dependencies in their .cabal files (as they should), then we should not really have a problem here if a few packages don't upgrade at the same time or at all (except for cases where a distribution only ships one version of each library and wants to ship two different packages that depend upon different versions of fgl). Once again: we're going to do our very best to avoid the issues that arose when parsec, QuickCheck, etc. had new major versions released and packages transitively depended upon two different versions; however if it does arise then it's going to be a very small localised problem. -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

On 6/23/10 8:06 PM, Duncan Coutts wrote:

Consider an example where we want to avoid using two versions of a dependency:

The htar program depends on the tar and zlib packages. The tar and zlib packages depend on bytestring. Both tar and zlib export functions that use the type ByteString. The htar program takes composes functions from the two packages together, passing a bytestring from one to the other.

Okay, I hadn't considered this, though it seems like the real problem in this situation is that someone violated the package version policy: if tar is upgraded to use a newer, incompatible version of bytestring, then it should have an appropriate version bump that causes htar to ignore the new version and use the older version instead. I am getting the impression from these discussions that what we really need in the long term to solve a lot of these problems is to enforce the package versioning policy. Cheers, Greg

On 6/23/10 10:06 PM, Duncan Coutts wrote:

Suppose both the zlib and tar packages specify "build-depends: bytestring-0.9.*". It's entirely possible for me to install zlib, then upgrade to a new bugfix release of bytestring, install tar (using the new bytestring) and then build htar depending on tar+zlib. In this case none of the developers of these packages have done anything wrong. It is situations like this that cabal-install tries to avoid.

Oh! Now I see. The impression that is coming to me now is that the package system is incredibly brittle, and so is very easy to break things even when theoretically following all of the rules. This isn't meant as a critique of the people who designed this system, mind you; it's works very well in practice for most things, and it is hard to see how these problems can be fixed without breaking other things. So I propose the following question to the floor: are there any ideas for changing the package system to make it less brittle?

I think what you mean is that cabal-install should infer from the situation where package A needs 'C< 2' and package B needs 'C>= 2' that it is safe for a package to depend on both A and B and to be built indirectly depending on two versions of C. The fact that people can and often do make packages that work with either C 1 or 2 makes this situation less common. Also, it's not necessarily true, perhaps all it means is that the latest version of B now needs a later C and so my package now cannot use that later version of C because it did indeed pass values of types defined in C between functions defined in A and B (that is, my package has been busted by the latest release of B).

Okay, I see now why it's harder then I had figured it would be. Thank you for taking the time to explain to me why I was wrong. :-) Cheers, Greg

I don't really see this listed on your list, but maybe I missed it. Happstack has been affected by QuickCheck 1 -> QuickCheck 2, parsec 2 -> 3, and HaXml 1.13 -> 1.20. Those packages are common, and people often want to use happstack with other libraries that also use those packages. The problem is that if we upgrade to the newer versions, then people can't use happstack with libraries that use the old version. For example, when we upgrade to QuickCheck 2, gitit no longer installed, because some dependency of gitit still used (and perhap still does) QuickCheck 1. We are now faced with the same problem with HaXml. Do we upgrade to 1.20 and stick with 1.13 for a while? Either way we cause incompatibilities. The the cause of QuickCheck, 99% of the time, it would have been ok to link the app against QC1 and QC2, because no one ever tried to combine the QC1 tests and the QC2 tests. So, in that cause, making QC1 and QC2 completely different packages would have solved our problem. Duncan C, also proposed a system whereby we could declare QuickCheck as an 'internal' dependency -- meaning we never export any QC2 functions or types to the outside word, so it is safe to link against other versions of QC. That would have worked fine as well, but the proposal does not seem to have gotten any traction yet. That is unfortunate, because it seems useful. In the case of HaXml, things are a bit trickier. It is more likely that people are going to want to use the HaXml stuff we export with other libraries that use HaXml, so everyone would have to be using the same version of HaXml then. At the same time, not very many people actually use the HaXml stuff in happstack, even if they are using another library that use HaXml. So, for most people the version mismatch isn't really an issue. So, like QC, it would actually probably be better for use if HaXml 1.13 and 1.20 had unique package names, so we could link against multiple HaXml versions. Of course, in the case of QC2 and HaXml, the ultimate solution is that everyone upgrades to the latest, and then the problem goes away. But, once you start depending on a larger number of packages directly and indirectly, it does not take very long before you run into someone that has not updated to the latest, and then you are kind of stuck... As a maintainer, I have no idea when the right time to switch to HaXml 1.20 is.. and that is an issue. That same would be true of fgl (though, fortunately, happstack doesn't use that yet). The only solution I have at the moment, is to upgrade to HaXml 1.20, and then send patches to any other direct or indirect dependencies of happstack that use HaXml and get them upgraded as well.. Maybe that is the best solution.. though it is also a lot of work. - jeremy On Tue, Jun 8, 2010 at 10:08 AM, Don Stewart wrote:

There have been a few cases of major API  / rewrites to famous old packages causing problems, including:

   * QuickCheck 1 vs 2    * parsec 2 vs 3    * OpenGL

a similar opportunity is present with 'fgl', where the new maintainers are seeking to improve the code.

Below I try to summarise the pros and cons of calling the new rewrite/api 'fgl', in the hope we can identify a path that minimizes disruption to users.

------------------------------------------------------------------------

A group of developers is planning to write a new graph library for Haskell.

   * They maintain an existing package called 'fgl'.    * 'fgl' has a long history: http://web.engr.oregonstate.edu/~erwig/fgl/    * The new library will have different authors and a different API.    * They would like the new library 'fgl'.

It is a controversial step to write a new library and give it the same name as an existing, famous library. Let's look at the arguments.

= Reasons to use the new name =

 * The new code  will be better, and should be preferred. Using the name   'fgl' will ensure adoption.

 * Rewrites are effective if the name is preserved. E.g. QuickCheck 2.

 * It is the maintainer's right to modify APIs as they see fit.

 * Keeping the old fgl around as a separate package, there is then        no real incentive to change/upgrade.

 * Relatively few packages use fgl. So damage is limited.

= Reasons not to use the name =

 * Code that depends on 'fgl' will break.       There are 23 direct and 25 indirect dependencies on fgl.       http://bifunctor.homelinux.net/~roel/cgi-bin/hackage-scripts/revdeps/fgl-5.4...

 * Doesn't matter if the old fgl is still around. If the new code is   better, it will be adopted on its own merits (see e.g.   bytestrings vs packedstring, vector vs uvector)   Let the market decide if it is better, rather than forcing us.

 * The package has been stable for ~10 years -- why change a stable API?    It is already "perfect"

 * The new package really isn't the same package in any sense.

 * Rewrites by new teams damage the brand of famous packages (e.g. parsec 3)

 * No additional breakages are introduced.

 * If you weren't maintainer of 'fgl' this rewrite wouldn't even be   possible to call 'fgl' -- there's a conflict of interest.

 * Maintaining Haskell98 compatability. Keep it simple. (See   regex-posix's mistakes here)

 * Distros that support the Haskell Platform will have to keep an old   version of fgl around for a long time anyway.

Are there any other arguments I'm missing?

-- Don _______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On Tue, Jun 8, 2010 at 7:06 PM, Ivan Lazar Miljenovic wrote:

I recall having a discussion with either you or someone else from the happstack team about why it isn't applicable there, but the QuickCheck problem can be solved in the general case by having its dependency be a compile-time-only option which is disabled by default.

Not really. Cabal likes to recompile things, and it won't remember that I compiled another library with quickcheck enabled. So if library B depends on library A with quickcheck enabled, it may still happen that after I install library A with quickcheck enabled, cabal will decide to recompile A again when building library B, and it will build A without quickcheck, causing B to fail...

However, this is definitely a major problem with parsec, HaXml, etc. and probably what I consider the main argument _against_ keeping the same name for fgl, though this can occur with packages even with a relatively minor API change (e.g. haskell-src-exts going from 1.6 to 1.7; the API change was minor but until all package maintainers upgraded their packages to work with 1.7 there would be some inconsistencies).

Right. Minor dependencies have the same issue. The only difference is that it is generally less work for maintainers to update their code. So maybe it happens faster/more often. But, we still run into that issue with happstack as well. We have fixed it by sending patches upstream. Since they are small patches it is usually easy to get them created and applied than something like HaXml 1.13 -> 1.19. - jeremy

Don Stewart writes:

* Maintaining Haskell98 compatability. Keep it simple. (See regex-posix's mistakes here)

At the risk of resurrecting an old (albeit unfinished) thread, I've just discovered that FGL is actually not Haskell98 compatible anyway, as it utilised the following extensions: MultiParamTypeClasses, OverlappingInstances, FlexibleInstances, ScopedTypeVariables Whether it needs all of these is a different story (ScopedTypeVariables seems to be there just because some type sigs have a forall in them), but it does indeed have them and is thus Haskell98 compatible. So this argument isn't valid ;-) -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

Stephen Tetley writes:

I think it was Hugs compliant as least for some revisions - I seem to remember looking at it before I switched to GHC.

People still use Hugs? :p A bit more seriously: is there any listing anywhere of which extensions Hugs supports? Definitely serious: if anything, I would care more about ensuring compatability with JHC, UHC, etc. than Hugs nowadays. -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com

People still use Hugs? :p

Is there another option for quick prototyping on iPhone?

Ivan Lazar Miljenovic writes:

People still use Hugs? :p

Well, I just did a quick count of Haskell libraries in Debian and Ubuntu (as a sort of comment to Don's blog post), but I forgot Hugs. It seems to be installed on 6000 Ubuntu-respondents, compared to 17000 installations of GHC, for a 25% "market share". But looking at the "using regularly" column GHC gets 725 votes, while Hugs only has 134. -k -- If I haven't seen further, it is by standing in the footprints of giants

wren ng thornton writes:

...

A bit more seriously: is there any listing anywhere of which extensions Hugs supports?

Cabal has a partial listing embedded in its code, though I can't seem to find a textual version at the moment. In general, Hugs has all the features of GHC 6.6: FFI, CPP, MPTCs, FunDeps, OverlappingInstances,... I'm forgetting off-hand whether it has Rank2Types/RankNTypes, but I think so. [..] Of course, since it's GHC6.6-era that means it doesn't support LANGUAGE pragma to enable these features.

I'm not entirely up to speed on Hugs nor Haskell 2010, but it seems that it's fairly close to compliance?

One can dream, eh?

:-) -k -- If I haven't seen further, it is by standing in the footprints of giants

On Tuesday 06 July 2010 07:04:18, wren ng thornton wrote:

Cabal has a partial listing embedded in its code, though I can't seem to find a textual version at the moment. In general, Hugs has all the features of GHC 6.6: FFI, CPP, MPTCs, FunDeps, OverlappingInstances,... I'm forgetting off-hand whether it has Rank2Types/RankNTypes, but I think so. The one notable difference between Hugs and GHC6.6 is that it does not have IncoherentInstances, and instead supports a different/incompatible way of trying to solve that problem.

Another notable difference are BangPatterns.

Of course, since it's GHC6.6-era that means it doesn't support LANGUAGE pragma to enable these features. The -98 flag enables most of the extensions with separate flags for OverlappingInstances, Not-IncoherentInstances, and CPP (or any other preprocessor); and in order to use the FFI you must first run ffihugs in order to compile the bindings. Because the shift from GHC6.6 to GHC6.8 was so significant, it'd be nice if someone put out a maintenance version of Hugs which adds support for LANGUAGE pragma and similar compilation interface issues, even if the underlying code base remains the same. One can dream, eh?

If support for BangPatterns could be added, that would be very convenient. Of course you can do it with seq, but BangPatterns are, well, very convenient compared to seq. Of course my plea for BangPatterns extends to JHC/UHC etc.

...

Definitely serious: if anything, I would care more about ensuring compatability with JHC, UHC, etc. than Hugs nowadays.

Certainly JHC and UHC should be targeted for compatibility. However, doing so seems unlikely to come at the cost of compatibility with Hugs. Hugs implements H98 along with the most venerable and widely used extensions. Consequently, the language Hugs understands happens to be the same one that's the goal of modern Haskell compilers like JHC and UHC. However little maintenance Hugs is getting, it captures all of "classic Haskell". The hallmarks of "new Haskell" like GADTs and type functions are, so far as I'm aware, considered a more distant goal for alternative compilers.

Stephen Tetley writes:

On 3 July 2010 14:00, Ivan Lazar Miljenovic wrote:

...

So this argument isn't valid ;-)

I think it was Hugs compliant as least for some revisions - I seem to remember looking at it before I switched to GHC.

Actually, how would you call it in Hugs? Just start it explicitly with the -98 flag? These modules currently don't load by default in ghci, since all the extensions are specified in the cabal file rather than in the source files with LANGUAGE pragmas. -- Ivan Lazar Miljenovic Ivan.Miljenovic@gmail.com IvanMiljenovic.wordpress.com