The alternative is just that cabal will continue indefinitely to try to install completely broken combinations, and more people will be driven to a fixed package set like stackage LTS. Most of these problems are caused by people being too optimistic about upper bounds and when they realize their mistake and upload a new version, they'll often leave the old versions with the lying bounds intact, which causes cabal to pick old versions without bug fixes, and then give strange build errors. To my knowledge, the few cases where Herbert has actively done a patch to the .cabal file like this without author communication is because the package is in very very widespread use and the author has been incommunicado for many months. As I recall, Max Bolingbroke has a some packages that fit this bill. At least in my case, and in the case of the Haskell core libraries, Herbert has been very conscientious about talking to me, finding problems, auditing what builds across all versions of GHC in recent and not-so-recent memory, and working with me to find the best fix on a case by case basis. He has my explicit consent for any tweaks he has had to make to the build dependencies of my packages and has worked with the core libraries committee very closely for patches to the core libraries. If you have an example of a package you've written that he's patched that you'd rather he left alone, I'm sure he'd be happy to oblige. I am, however, as of yet unaware of any such overreach and I'm rather disinclined to view the enormous amount of effort Herbert has poured into keeping the ecosystem working smoothly as anything but a good thing. The price of doing nothing here is quite high. -Edward On Sun, Jan 18, 2015 at 6:05 PM, Vincent Hanquez wrote:

On 18/01/2015 09:56, kyra wrote:

...

Hi, guys,

It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel.

While this is harmless I wonder what idea stands behind this?

This is not harmless. This is a security issue by itself, as now packages get changes transparently given a url, you might have a different package one day, which trigger hash check failure. or signed tag verification failure.

This has also the effect of not changing the bounds in the repository, so for example, next time you upload a tweak'ed packages, you effectively revert the change done on hackage only.

This is also done without the consent of the maintainer of a given package, nor that the maintainer is actually notified when that happens, or allow to prevent it happening. This is pretty big start from the other similar policy for taking over packages, that insist on a very long period of repeated communication with the author and then the community.

The whole thing is at best ill advised, -- Vincent

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

Hello Vincent, On 2015-01-19 at 01:39:09 +0100, Vincent Hanquez wrote: [...]

...

To my knowledge, the few cases where Herbert has actively done a patch to the .cabal file like this without author communication is because the package is in very very widespread use and the author has been incommunicado for many months. As I recall, Max Bolingbroke has a some packages that fit this bill.

A simple counter example, that I noticed after the fact [1]

That was more of an oversight of mine as I got blocked on entropy being broken[2] which transitively broke several of your packages for GHC 7.0/7.2 users (and this got fixed yesterday). I usually did inform upstreams in the cases I had to edit the meta-data (sometimes before, and sometimes after the fact). However, now that entropy got fixed and I got a clearer picture of which failures have to be fixed in your packages, I've filed a couple of tickets[3] against your packages to record what actions are needed to restore some install-plans. Moreover, as already stated elsewhere in this thread, an email notification system is in the works to automate informing upstreams to some extent.[4] Does this procedure meet your expectations better? [...] Cheers, hvr [2] https://github.com/TomMD/entropy/issues/23#issuecomment-70705615 [3]: https://github.com/vincenthz/hs-tls/issues/99 https://github.com/vincenthz/hs-certificate/issues/42 https://github.com/vincenthz/hs-cprng-aes/issues/10 https://github.com/vincenthz/hs-crypto-numbers/issues/16 https://github.com/vincenthz/hs-crypto-pubkey/issues/19 https://github.com/vincenthz/hs-cryptohash/issues/33 https://github.com/vincenthz/hs-hourglass/issues/15 [4] https://github.com/haskell/hackage-server/issues/230

On 18/01/2015 15:51, David Feuer wrote:

It would be best to be sure to make the maintainer (if there is one) aware of such changes. That said, not every package has a responsive maintainer, and *someone* has to do this work, and do it promptly. A signed hash failure does not introduce a security hole, unless you count a sort of semi-manual, avoidable denial of service.

Not sure how you got "security hole" from what I said, but a failing hash or signature, means that the build system breaks while cabal install stuff and that I have to manually inspect what the change is. If you can't pin down a special tarball when doing a download (i.e. it can changes under your feet, one day to the other), then it's an issue. Lots of people would be *horrified* to download some {c,c++,python,ruby,...} library-a.b.c.tar.gz and found anything changed inside without changing the exact name for it.

If you don't trust Herbert and Austin, you probably shouldn't bother trying to use Haskell anyway.

lol ? Do you mean that I should switch language, if security is remotely important to me ? As much as Herbert and Austin are doing awesome work in general, I certainly do not blindly trust them. -- Vincent

Wow, I thought this was a joke at first. HVRs work is terribly useful for enabling old and new versions of GHC to work with hackage, and I've very much appreciated the build reports he generates. I'm sure he'd be happy to leave packages alone for anyone who prefers their packages remain broken. Thanks hvr for fixing the dozens and dozens of versions of just the lens package in order to sort the base-4.8 rollout in particular that would have taken an age to do manually. On Sun, Jan 18, 2015, 5:00 PM Brandon Allbery wrote:

On Sun, Jan 18, 2015 at 7:57 PM, Vincent Hanquez wrote:

...

Lots of people would be *horrified* to download some {c,c++,python,ruby,...} library-a.b.c.tar.gz and found anything changed inside without changing the exact name for it.

...

Funny thing, that, because it happens pretty often and packagers have to have "silent update" (upstream silently repackaged without changing the version) rules to deal with it.

-- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net _______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On Mon, Jan 19, 2015 at 7:06 AM, Herbert Valerio Riedel wrote:

The reason I wouldn't be happy is that the effects of a "broken" package (especially the more popular it becomes) can't be contained easily. The

I wonder if Hackage can be extended to support an out-of-band "broken" flag that can be applied to such packages, and cabal-install then refuse to use those packages (possibly with an option to override). -- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

We have a deprecation flag on hackage now, it just doesn't do anything at last check. On Mon, Jan 19, 2015 at 9:59 AM, Brandon Allbery wrote:

On Mon, Jan 19, 2015 at 7:06 AM, Herbert Valerio Riedel wrote:

...

The reason I wouldn't be happy is that the effects of a "broken" package (especially the more popular it becomes) can't be contained easily. The

I wonder if Hackage can be extended to support an out-of-band "broken" flag that can be applied to such packages, and cabal-install then refuse to use those packages (possibly with an option to override).

-- brandon s allbery kf8nh sine nomine associates allbery.b@gmail.com ballbery@sinenomine.net unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

_______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On Mon, 2015-01-19 at 09:59 -0500, Brandon Allbery wrote:

On Mon, Jan 19, 2015 at 7:06 AM, Herbert Valerio Riedel wrote:

...

The reason I wouldn't be happy is that the effects of a "broken" package (especially the more popular it becomes) can't be contained easily. The

I wonder if Hackage can be extended to support an out-of-band "broken" flag that can be applied to such packages, and cabal-install then refuse to use those packages (possibly with an option to override).

This can be achieved by editing the .cabal file, and Herbert has done so in at least one case. It's just a matter of making the constraints impossible, e.g. base > 1 && < 1. It could possibly be done more obviously or directly, e.g. adding a dep on something impossible (though note that we don't currently allow adding deps). Deprecation is orthogonal I'd say. People may need to rely on deprecated packages while they migrate away. Duncan

On Tue, Jan 20, 2015 at 06:22:47PM +0100, Herbert Valerio Riedel wrote:

On 2015-01-20 at 17:43:24 +0100, Duncan Coutts wrote:

[...]

...

This can be achieved by editing the .cabal file, and Herbert has done so in at least one case. It's just a matter of making the constraints impossible, e.g. base > 1 && < 1. It could possibly be done more obviously or directly, e.g. adding a dep on something impossible (though note that we don't currently allow adding deps).

Fwiw, I've converged to using the following construct to blacklist a package version:

library -- [reason why this package version had to be blacklisted] build-depends: base<0

[...rest of content...]

IMHO this is somewhat obvious than any other unsatisfiable constructs

Indeed, that is a much better way to signal blacklisting/deprecation/etc. I first found the unsatisfiable-dependencies solution in the random package. It was only due to being able to find a bug mentioning that the version in question should be deprecated that I realised it was done on purpose. /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus Code as if whoever maintains your program is a violent psychopath who knows where you live. -- Anonymous

On 19/01/15 01:05, Vincent Hanquez wrote:

This is not harmless. This is a security issue by itself, as now packages get changes transparently given a url, you might have a different package one day, which trigger hash check failure. or signed tag verification failure.

Correct me if I'm wrong, but editing version bounds on hackage doesn't actually affect the tarball (and its checksum). The modified cabal file is downloaded separately as part of the index. Not saying it doesn't introduce its own problems, but the hash check should continue to pass. Roman

On Sun, 2015-01-18 at 21:31 -0800, Vincent Hanquez wrote:

On 18/01/2015 20:23, Roman Cheplyaka wrote:

...

On 19/01/15 01:05, Vincent Hanquez wrote:

...

This is not harmless. This is a security issue by itself, as now packages get changes transparently given a url, you might have a different package one day, which trigger hash check failure. or signed tag verification failure. Correct me if I'm wrong, but editing version bounds on hackage doesn't actually affect the tarball (and its checksum). The modified cabal file is downloaded separately as part of the index.

yes, that's right. I meant to say that what you're downloading through cabal get tweaked by cabal, but the end result is pretty much the same

...

Not saying it doesn't introduce its own problems, but the hash check should continue to pass. of the tarball yes, not of your compilation tree, and maybe not the resulting binary.

So we have thought about it a bit when designing this feature and we don't think that is correct. If you freeze your solution (e.g. with cabal freeze) then changes to the metadata should have no effect on compilation. We're pretty restrictive about what changes are allowed and we don't think there's anything that can cause changes except for the versions of deps picked (and hence if you freeze then nothing). Apart from version constraints we pretty much only allow tweaks to metadata like description, maintainer email etc. If you don't freeze then currently you have no guarantee that you'll not get different solutions each time you "cabal update" to a new hackage snapshot (or indeed installing things can change what solutions you get since we prefer already installed instances). One caveat is that it's possible to accidentally constrain deps and make an existing working solution impossible. This could make a frozen solution no longer build, which is annoying but doesn't make a broken binary. If this proves to be a real issue then we can deal with it (simply ignore constraints for frozen solutions). Duncan

On Mon, Jan 19, 2015 at 2:51 AM, Herbert Valerio Riedel wrote:

Just to clarify: the source tarball are *never* modified to specifically make sure hashes don't randomly change (which would trip all sorts of warnings for distribution packagers). In fact, there's also a `cabal get` option if you want to unpack with the original unaltered .cabal file:

--pristine Unpack the original pristine tarball, rather than updating the .cabal file with the latest revision from the package archive.

Why is this option needed if the tarball is never modified? This documentation states that cabal modifies the cabal file on the local system. We all appreciate Herbert's efforts. The problem is the tools he is working with are limited. The local cabal should operate the same way as hackage: always keeping the meta-data separate. Please correct me if I am mis-representing the situation.

On Sun, 2015-01-18 at 15:05 -0800, Vincent Hanquez wrote:

On 18/01/2015 09:56, kyra wrote:

...

Hi, guys,

It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel.

While this is harmless I wonder what idea stands behind this?

This is not harmless. This is a security issue by itself, as now packages get changes transparently given a url, you might have a different package one day, which trigger hash check failure. or signed tag verification failure.

Note that hackage never changes the content of package tarballs. The checksums on those are stable. Guaranteed.

This has also the effect of not changing the bounds in the repository, so for example, next time you upload a tweak'ed packages, you effectively revert the change done on hackage only.

Communicating changes upstream is certainly something we need to work on to be able to use this as widely as it'd be helpful. Up until recently we've only used the metadata editing feature with core packages (or the maintainers themselves have done it). Recently Herbert has been going a bit wider and if we are now running into issues of communication with maintainers then I think this says that now is the time to address that properly. So that includes: * this discussion * wider communication with maintainers of just what is and is not possible (since we're actually deliberately rather conservative) * a proper notification and opt-in/opt-out system for maintainers to avail themselves of the helpful service that the trustees can provide. Duncan

I just want to say that I appreciate your work. You're a hero for doing this very thankless but very beneficial work. On 01/19/2015 12:31 AM, Herbert Valerio Riedel wrote:

On 2015-01-18 at 18:56:42 +0100, kyra wrote:

...

It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel. That would be me...

Just to clarify: I those aren't re-uploads of existing package versions, but rather meta-data revisions. The actual source-code is *not* touched, (and Hackage is very restricting in what's allowed to be changed in a .cabal file)

This is also mentioned on http://hackage.haskell.org/packages/trustees/

...

While this is harmless I wonder what idea stands behind this? The idea, as others in this thread have already pointed out, is to add missing lower/upper package bounds to packages so that the Cabal install-plan solver can fulfill its task.

Over the last week, I've been analyzing the current situation (motivated by the upcoming GHC 7.10 release which trips up quite a few packages with missing bounds on base/deepseq/time), and trying to come up with some tooling[1] to help the otherwise IMO rather unmanageable job of a Hackage trustee, as well as surgically fixing up the meta-data where I had empirical proof that the .cabal meta-data was incorrect (and thus leading the solver to select a broken install-plan).

Here's a very simple buildreport for illustration:

https://ghc.haskell.org/~hvr/buildreports/pretty.html

The purpose of my meta-data edits were/are (in order of decreasing priority):

1. First and foremost, no currently working/valid install-plan was lost.

2. A simple constraint-less `cabal install pretty` ought to result *either* in a valid install-plan, *or* cabal failing to find any install-plan.

3. An major-version constraint `cabal install 'pretty == a.b.*'` ought to result *either* in a valid install-plan, *or* cabal failing to find any install-plan.

4. A package shall not cause indirect *build*-failures of other packages directly or indirectly depending on this package[3]

The build-report linked helps directly with 2. and 3. as it shows that `pretty` failed to `cabal install pretty` (as well as `cabal install 'pretty == 1.1.*'`) for GHC 7.0 and GHC 7.2. Case 4. could be detected by running the buildtool on many Hackage packages depending on `pretty`, and look for failures to build `pretty` during the `cabal install --dep` step (which would result in a "FAIL (deps)" cells).

I've encountered a couple of situations, where fixing the meta-data of an important package low in the dependency graph fixed a couple of other packages depending on that package. However, this also means that the more popular a package is, the higher its responsibility is to make sure its build-depend meta-data is correct, as such packages have the potential to instantly break large branches of Hackage:

One example for that was the release of deepseq-1.4, which changes the default-implementation of `rnf` which a few packages relied on, but the rather important `text` package was among of those. This affected most of the released ~50 `text` package versions, and since `deepseq` is compatible with all GHC 7.* versions, as soon an install-plan was selected which included `deepseq-1.4` with a non-latest version of `text` the build would fail. And since it's not uncommon to restrict the upper bound of `text`[2], all those packages depending on a non-latest `text` would have broken install-plans (unless `deepseq < 1.4` was selected).

I hope this answers your question.

Cheers, hvr

[1]: https://github.com/hvr/hackage-matrix-builder

(WIP example reports can be seen in

https://ghc.haskell.org/~hvr/buildreports/0INDEX.html)

[2]: http://packdeps.haskellers.com/reverse/text

[3]: This can happen due to additional package-version constraints causing other (but broken) configurations to be selected by the solver not detected in case 2. and 3. _______________________________________________ Libraries mailing list Libraries@haskell.org http://www.haskell.org/mailman/listinfo/libraries

On 20.01.2015 14:47, Duncan Coutts wrote:

...

Hi, guys,

It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel. BTW, where do you see these changes? I thought we had fixed the "latest uploads" page and rss to only show real uploads, and not edits to

On Sun, 2015-01-18 at 20:56 +0300, kyra wrote: metadata.

Not that we want to hide the edits, they need to be visible, but we shouldn't be confusing them with new package uploads in the UI anywhere.

Duncan

.

I've wrote a small program producing old-hackage-log-like-file from hackage index file ("Recent additions" page is too short). Regads, Kyra