You are right, I skipped over that this was actually a server-side exploit - sure, end-to-end signing will help here. On 30/01/13 19:47, Edward Z. Yang wrote:

...

As long as we upload packages via plain HTTP, signing won't help though.

HTTPS doesn't really change anything if the server is compromised, it only prevents bad things from happening in transit. Sign the packages with GPG (or equivalent) before upload. The server never sees the package author's private key, only the public key. Server and/or client can warn or fail if the public key doesn't match their previous credentials or the signature verification fails. On Wed, Jan 30, 2013 at 11:44 AM, Niklas Hambüchen wrote:

As long as we upload packages via plain HTTP, signing won't help though.

On Wed 30 Jan 2013 19:27:32 GMT, Edward Z. Yang wrote:

...

https://status.heroku.com/incidents/489

Unsigned Hackage packages are a ticking time bomb.

Cheers, Edward

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

IMHO Hackage and Cabal should support package signing even if they aren't package managers. On Wed, Jan 30, 2013 at 6:59 PM, Joachim Breitner wrote:

Hi,

Am Mittwoch, den 30.01.2013, 11:27 -0800 schrieb Edward Z. Yang:

...

https://status.heroku.com/incidents/489

Unsigned Hackage packages are a ticking time bomb.

another reason why Cabal is no package manager¹.

(Ok, I admit that I don’t review every line of diff between the Haskell packages I uploads. But thanks to http://hdiff.luite.com/ I at least glance over them most of the time – a hurdle that malicious code would have to take. And once a package has entered a distribution like Debian (which it only can with a valid cryptographic signatures), checksums and signatures are used in many places to (mostly) guarantee that the package reaches the user unmodified.)

Greetings, Joachim

¹ http://ivanmiljenovic.wordpress.com/2010/03/15/repeat-after-me-cabal-is-not-...

-- Joachim "nomeata" Breitner Debian Developer nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Felipe.

On Wed, Jan 30, 2013 at 10:43 PM, Edward Z. Yang wrote:

This argument seems specious. Whether or not cabal-install is or not intended to be a package manager, users expect it to act like one (as users expect rubygems to be a package manager), and, at the end of the day, that is what matters.

But playing along with their delusion might make it harder to change their minds.

Edward

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Wed, Jan 30, 2013 at 10:48 PM, Edward Z. Yang wrote:

...

...

This argument seems specious. Whether or not cabal-install is or not intended to be a package manager, users expect it to act like one (as users expect rubygems to be a package manager), and, at the end of the day, that is what matters.

But playing along with their delusion might make it harder to change

Excerpts from Ramana Kumar's message of Wed Jan 30 14:46:26 -0800 2013: their

...

minds.

Looking at the library ecosystems of the most popular programming languages, I think this ship has already sailed.

I was talking only about Haskell and cabal. There is a viable alternative to using cabal as a package manager on Arch Linux (the Arch-Haskell package repositories). There is also the Stackage project that might make this possible on more distributions with real package managers. But if you keep calling cabal a package manager, eventually you'll have to write the patches to make it one.

Edward

Ertugrul Söylemez writes:

People are using Hackage!

+1. And I keep telling people to use it. Sure, it'd be better if they used .debs, .rpms, or whatever goes on Mac and Windows. But that would mean I would need to build those packages, including maintaining systems with the respective OSes. I haven't even managed to do it for the systems I do use. The most simple and obvious threat is here that some random evil person gets a Hackage account, uploads a new version of a common package with a trojan, and waits for unsuspecting users to download and install it.

My proposal is:

1. Build the necessary machinery into Cabal to allow signing [...]

*MY* proposal is that: 0. Hackage sends an email to the previous uploader whenever a new version of a package is uploaded by somebody else. At least that way, I would be notified if it happened to my packages, and I would be able to check up on the situation, and rectify it. This is not to say that cryptographic signing is the wrong thing to do, but a very simple thing like this, which would probably take all of five minutes to implement, would reduce risk by a substantial amount. -k -- If I haven't seen further, it is by standing in the footprints of giants

Vincent Hanquez wrote:

For example, previous maintainer might be away from email for a long time potentially leaving a trojan version for days/weeks, or changed email address..

And that may even be more harmful, because an insecure system with a false sense of security is worse than an insecure system alone. Let's do it properly. Greets, Ertugrul -- Not to be or to be and (not to be or to be and (not to be or to be and (not to be or to be and ... that is the list monad.

Joachim Breitner wrote:

...

And that may even be more harmful, because an insecure system with a false sense of security is worse than an insecure system alone.

Let's do it properly.

but don’t overengineer it either. Simply adding to hackage the possibility to store a .asc file next to the tar.gz file that contains the cryptographic signature would be a great start, and allow us to develop a WoT model later on.

(I try to resist from wondering whether this could go into hackage1 or only hackage2, and in the latter case, whether that means that we actually have the time to overengineer the system.)

In fact, a lot would already be gained by a simple „warn if foo-2.0 is signed with a different key than the version of foo already installed“ on cabal-install and people having a closer look at uploads from different people. Not much infrastructure needed there.

That was exactly my suggestion actually. It requires the ability to make and check signatures. The making can be done with external tools like GnuPG, but the checking has to be done by cabal-install. To detect changed keys there also needs to be a trust database, which can be a simple directory in ~/.cabal/ where files are named after the fingerprint of the key it contains. The most important part is a sensible user interface. The whole process should be invisible to the user, until there is a signature error. The first installation of a package will actually generate a handful of signature errors, because the keys are not known yet. This shouldn't be too hard to implement and requires only a small change to Hackage and cabal-install's upload command to begin. Greets, Ertugrul -- Not to be or to be and (not to be or to be and (not to be or to be and (not to be or to be and ... that is the list monad.

Vincent Hanquez wrote:

...

That was exactly my suggestion actually. It requires the ability to make and check signatures. The making can be done with external tools like GnuPG, but the checking has to be done by cabal-install. To detect changed keys there also needs to be a trust database, which can be a simple directory in ~/.cabal/ where files are named after the fingerprint of the key it contains.

The most important part is a sensible user interface. The whole process should be invisible to the user, until there is a signature error. The first installation of a package will actually generate a handful of signature errors, because the keys are not known yet.

This shouldn't be too hard to implement and requires only a small change to Hackage and cabal-install's upload command to begin.

That's not a proper solution, and definitively in the warm fuzzy feeling department.

What if you install a package for the first time and this package has just been re-uploaded maliciously with a different key and a payload ? What if you're relying on hackage mirrors, what stop this mirror to regenerate all signatures with a new key ?

It also make maintainers change difficult, and doing genuine non-maintainer upload.

See the last point of my post. The last step is to implement proper web of trust functionality, so that some keys can be declared to be signing keys. Then a set of trusted keys can be shipped together with cabal-install. That step is optional, because at least now I can fetch developer keys by other means like a key server. According to my solution Cabal warns for new and changed keys and asks whether to trust them showing a fingerprint. Greets, Ertugrul -- Key-ID: E5DD8D11 "Ertugrul Soeylemez " FPrint: BD28 3E3F BE63 BADD 4157 9134 D56A 37FA E5DD 8D11 Keysrv: hkp://subkeys.pgp.net/

Ertugrul Söylemez writes:

And that may even be more harmful, because an insecure system with a false sense of security is worse than an insecure system alone.

Yes. As is clear to all, the current low level of security means that nobody are _actually_ downloading stuff of Hackage, thank God. Hackage just exists for...well, I forget, but certainly not to distribute software. Right. Sarcasm aside, to some extent, this is true. I used to have a cron job 'cabal install'ing my packages off Hackage to ensure that they would compile with the current offering of their dependencies. But I decided it was way too risky, and don't do it anymore.

Let's do it properly.

You mean like how it was decisively dealt with when this was discussed in 2008? https://github.com/haskell/cabal/issues/207 Or maybe more the way it was firmly handled when it was brought up again in 2010? http://www.haskell.org/pipermail/haskell-cafe/2010-December/087050.html This looks increasingly like that time of year when the problem is pointed out, the crypto geeks get together to construct the Optimal Solution, and then everybody lose interest and move on to greener pastures for a while. Well, I don't think the perfect solution exists, and even if it could be identified, it might not be implemented, and even if were implemented, it might not be used. We've just been incredibly lucky that nothing really bad has happened so far. Let's hope it lasts. -k -- If I haven't seen further, it is by standing in the footprints of giants

Vincent Hanquez writes:

On 01/31/2013 08:16 AM, Ketil Malde wrote:

...

At least that way, I would be notified if it happened to my packages, and I would be able to check up on the situation, and rectify it.

you wouldn't in real cases,

I wouldn't what? Be notified? Rectify it?

it just fix the most obvious and simple attack vector.

but consider:

Ah, those _real_ cases. Fine, let's see what you got.

* someone intercepting your upload http stream, and replacing dynamically your package.

* someone gaining malicious access to hackage and planting stuff inside packages.

* a rogue hackage admin.

* a rogue hackage mirror admin.

How often do these things happen, I wonder? I guess a bit more rarely than malware taking control of people's computers, which would simply allow people access to upload passwords _and_ signing keys. So much for that perfect security scheme, huh? Anyway: I don't want to sound negative here, so as a constructive proposal, please replace one of my packages on Hackage, making use of one of those techniques. No hurry, I'll wait. We have this discussion every now and then, and in spite of grand schemes of crypto and signing and whatnot, we _still_ have security holes you could drive a truck through. Why not at least do _something_ to fix that, before we return to the more _interesting_ pursuit of the Perfect Security Scheme? -k -- If I haven't seen further, it is by standing in the footprints of giants

On 31/01/13 09:16, Ketil Malde wrote:

*MY* proposal is that:

0. Hackage sends an email to the previous uploader whenever a new version of a package is uploaded by somebody else.

At least that way, I would be notified if it happened to my packages, and I would be able to check up on the situation, and rectify it.

This is not to say that cryptographic signing is the wrong thing to do, but a very simple thing like this, which would probably take all of five minutes to implement, would reduce risk by a substantial amount.

That is an excellent idea, and it should be very simple to add. Of course it doesn't stop all attacks, but it does stop the most obvious one. And it might also prevent some honest mistakes or errors in communication where someone uploads a forked package without permission. Twan

On Thu, Jan 31, 2013 at 9:26 AM, Vincent Hanquez wrote:

On 01/31/2013 06:27 AM, Ertugrul Söylemez wrote:

...

In any case there is no valid excuse for the lack of crypto. It's too easy to attack Hackage, so we need some crypto regardless of what we interpret it as.

My proposal is:

1. Build the necessary machinery into Cabal to allow signing keys and packages and verifying the signatures, ideally through GnuPG. Cabal would benefit from that even without cabal-install and Hackage.

Seems there's lots of suggestion of using gnupg, which is a perfectly valid answer if cabal was unix only, but i'm not sure it's a valid option considering windows. Sure you can install gnupg somehow, but sounds to me it's going the same problem as gtk2hs on windows.

One better way, would be to tap in the 2, work in progress, gnupg haskell replacement:

http://hackage.haskell.org/**package/openpgphttp://hackage.haskell.org/package/openpgp http://hackage.haskell.org/**package/hOpenPGPhttp://hackage.haskell.org/package/hOpenPGP

AFAIK, both packages are not yet handling anything related to WoT, but just do the signing/verification (which is same status as my ad-hoc experiment)

In this case I think this is the wrong approach. There must be at least one way to work within a trust model that is not fragile. Whether this is fully supported on all platforms is actually not very important. I have pointed out why simply signing packages is fragile and how git is better suited for this task. We are not going to reimplement all the good infrastructure that already exists (gpg, git), so making that a requirement is not a good idea IMO. Basic verification of signatures should work on Windows, I agree. But the underlying WoT should be a little bit more sophisticated. This means it has to be based on standard tools, or it will never happen. Alexander

______________________________**_________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/**mailman/listinfo/haskell-cafehttp://www.haskell.org/mailman/listinfo/haskell-cafe

On Thu, Jan 31, 2013 at 11:48 AM, Vincent Hanquez wrote:

On 01/31/2013 08:54 AM, Alexander Kjeldaas wrote:

...

On Thu, Jan 31, 2013 at 9:26 AM, Vincent Hanquez wrote:

On 01/31/2013 06:27 AM, Ertugrul Söylemez wrote:

...

In any case there is no valid excuse for the lack of crypto. It's too

...

easy to attack Hackage, so we need some crypto regardless of what we interpret it as.

My proposal is:

1. Build the necessary machinery into Cabal to allow signing keys and packages and verifying the signatures, ideally through GnuPG. Cabal would benefit from that even without cabal-install and Hackage.

Seems there's lots of suggestion of using gnupg, which is a perfectly valid answer if cabal was unix only, but i'm not sure it's a valid option considering windows. Sure you can install gnupg somehow, but sounds to me it's going the same problem as gtk2hs on windows.

One better way, would be to tap in the 2, work in progress, gnupg haskell replacement:

http://hackage.haskell.org/****package/openpgphttp://hackage.haskell.org/**package/openpgp <http://**hackage.haskell.org/package/**openpgphttp://hackage.haskell.org/package/openpgp

...

http://hackage.haskell.org/****package/hOpenPGPhttp://hackage.haskell.org/**package/hOpenPGP <http://**hackage.haskell.org/package/**hOpenPGPhttp://hackage.haskell.org/package/hOpenPGP

...

AFAIK, both packages are not yet handling anything related to WoT, but just do the signing/verification (which is same status as my ad-hoc experiment)

In this case I think this is the wrong approach. There must be at least

one way to work within a trust model that is not fragile. Whether this is fully supported on all platforms is actually not very important.

I have pointed out why simply signing packages is fragile and how git is better suited for this task. We are not going to reimplement all the good infrastructure that already exists (gpg, git), so making that a requirement is not a good idea IMO.

Basic verification of signatures should work on Windows, I agree. But the underlying WoT should be a little bit more sophisticated. This means it has to be based on standard tools, or it will never happen.

I think you misunderstood me.

Having a fully working pgp package, means you have full control of the pgp stack, you don't rely on hard-to-get out tools, and it can be integrated with cabal directly for a full WoT experience.

Also git doesn't solve the hackage problem, there's not necessarily a one-to-one mapping between packages and their repositories.

Regarding git, I don't see anyone even suggesting that hackage should support multiple 'signed-off-by's, signing diffs between packages, and signing releases. I don't see anyone supporting 3rd-parties signing packages at will. All of this is significant engineering effort that just won't happen. And even if it does, it still doesn't solve the problem, because all of the above is required to protect our base ghc, cabal-install tools anyways. If cabal-install is hacked, hackage is hacked. And even when all of this is done, a design where there is a set of trusted developers is flawed. The set of trusted developers will increase over time, and there will be the need to restrict which developers can release which package. How do you protect that information, which keys can sign which packages? That information must itself be signed off by someone. If not, ouch some obscure developer in the trusted set hijacked an important package because he was owned. That signing in itself *requires* either consensus or a trusted third party. Consensus is a superior method and IMO required. Actually, all modifications to important packages requires consensus by at least two developers to avoid the threat posed by "owned" developers. So although it is not a waste of time to "secure" hackage, it is always going to be slightly fragile as long as ghc is fragile for example, and it just isn't ever going to *really* solve the problem. Git has the ability to solve all of this. So a better use of resources IMO is to skip securing hackage by own means, but to standardize on using git as the infrastructure for securing hackage. Then the same infrastructure that should secure ghc should secure hackage. To explain this in concrete terms, to do this, all that is needed is *something like* the following: 1. Each package on hackage gets its own repository (say on github or similar). (I've actually put all of hackage in a single git repository - it works fine, so there are no relevant size problems) 2. Uploads to hackage either happen through commits to the git repository, or an old-style upload to hackage automatically creates a new anonymous branch in the git repository. 3. The git repository is authorative. Signing releases, code reviews etc. all happens through the git repositories. This gives us all the flexibility of a git-style trust model. 4. If a package already has a git repository, that repository can be used instead of the 'hackage' one. 5. Who owns which package names can be held in a separate meta-tree git repository, and can have consensus requirements on commits. 6. This special meta-tree can also contain suggested verification keys for commits to the other hackage git trees. It can even contain keys that protect Haskell namespaces in general, so that no hackage package can overwrite a protected Haskell namespace. 7. As backward compatibility, the meta-tree can sign simple hashes of already existing packages on hackage. I'll just note that none of the above requires any changes to Hackage at all. That is the modification story. On the verification side there are multiple options: 1. There could be some git magic script that downloads the signed git tag objects only (small data set). Then another script would generate a git-compatible SHA1 of the extracted tarball, given that the tarball was fetched from hackage. 2. Or cabal-install could fetch directly from git repositories and use standard git verification. 3. Or a trusted machine creates tarballs from the git repositories, signs them and uploads them to hackage. At the same time, a set of distrustful machines do the same thing by running cron-jobs that check the trusted machine. This is slightly fragile, but would "mostly work". Alexander

-- Vincent

On 1/02/2013, at 3:32 PM, Kevin Quick wrote:

Without details of git's trust/verification model, it's difficult to see how this particular SCM tool provides the trust capabilities being discussed any better than a more focused solution. Additionally, the use of git is also difficult for many Windows users (80MB installed footprint, last I tried).

Put it on an outboard 1.5TB drive (price about $100) and it will take about 1/18,000th of the space. And Windows Update downloads about that amount or more in patches every time I log in (about once a fortnight). So it can't really be the size that makes git-on-Windows difficult.

Excerpts from Joachim Breitner's message of Wed Jan 30 14:57:28 -0800 2013:

I’m not against cryptographically signed packages on hackage. In fact, I would whole-heatedly appreciate it, as it would make my work as a package maintainer easier.

I was taking the opportunity to point out an advantage of established package management systems, to shamelessly advertise my work there, as not everyone sees distro-packaged libraries as a useful thing.

Yes. In fact, I am a sysadmin for a large shared hosting environment, and the fact that programming language libraries tend not to be distro-packaged is an endless headache for us. We would like it if everything were just packaged properly! On the other hand, working in these circumstances has made me realize that there is a huge tension between the goals of package library authors and distribution managers (a package library author is desires ease of installation of their packages, keeping everyone up-to-date as possible and tends to be selfish when it comes to the rest of the ecosystem, whereas the distribution manager values stability, security, and global consistency of the ecosystem.) So there is a lot of work to be done here. Nevertheless, I believe we are in violent agreement that cryptographically signed Hackage packages should happen as soon as possible! Edward

Marc Weber writes:

The only safe way is acceptnig keys from people you know don't view pdf using adobe reader,

I don’t…

who don't browse the web (neither use flash) etc.

I only browse the web (in general) from a diskless virtual machine.

And then still you also have to know that their email account password is reasonable strong ..

I don’t think you need to know that: you need to know that their signing key password is strong, that they’ve never entered it into a machine with a keylogger running, and no-one has shoulder surfed them. Oh, and that no-one can blackmail or torture them :-P -- Jón Fairbairn Jon.Fairbairn@cl.cam.ac.uk

Vincent Hanquez wrote:

I agree this is terrible, I've started working on this, but this is quite a bit of work and other priorities always pop up.

https://github.com/vincenthz/cabal https://github.com/vincenthz/cabal-signature

My current implementation generate a manifest during sdist'ing in cabal, and have cabal-signature called by cabal on the manifest to create a manifest.sign.

The main issue i'm facing is how to create a Web of Trust for doing all the public verification bits.

You don't need it yet. See my other post. Once the basic infrastructure for signatures is established, you can allow the user to have a set of trusted keys. The idea is that users can ask for keys and/or import keys from key servers. In the worst case they accept keys when installing a package. Once you have such a trust database you can allow users to select, whether a key is to be trusted for signing other keys. Then you have basically everything to establish both hierarchial trust relationships (like CAs) and webs of trust. Greets, Ertugrul -- Not to be or to be and (not to be or to be and (not to be or to be and (not to be or to be and ... that is the list monad.

On Fri, Feb 01, 2013 at 01:07:33PM +0100, Christopher Done wrote:

Hey dude, it looks like we made the same project yesterday:

http://www.reddit.com/r/haskell/comments/17njda/proposal_a_trivial_cabal_pac...

Yours is nice as it doesn't depend on GPG. Although that could be a nice thing because GPG manages keys. Dunno.

Another diff is that mine puts the .sig inside the .tar.gz, yours puts it separate.

Nice to see a productive discussion on this. /me really need to read reddit more :) Couple of details, no the signature is going inside the tarball too. the signature process happens during the sdisting after building the manifest. My reason for doing is, which i suspect similar to yours, is that I don't need to modify hackage this way and the uploading stays the same. Also in my case, cabal-signature is called by cabal, not by the user. I can't see this effort working without forcing everyone to use it (transparently in the background) For gpg, i don't know what's the right answer. One on hand it's solving all the problems related to this already, but on the other portability issue. I was thinking maybe one way to verify the key that i use for signing, would be to tie it to a personal gpg key (by signing the key with a gpg key) to benefit from all the facilities that gpg provides. It would provide a cheap way to switch model later, without being tied to a gpg signing process. -- Vincent

+1 for keeping this alive. Apart from the initial hype, now this issue is slowly losing attention but I think we should always keep the risk we are exposed to. Being I will sound pessimistic, but we should learn from the "competitors" mistakes :) Cheers, A. On 12 February 2013 08:49, Bob Ippolito wrote:

The Python and Ruby communities are actively working on improving the security of their packaging infrastructure. I haven't paid close attention to any of the efforts so far, but anyone working on cabal/hackage security should probably take a peek. I lurk on Python's catalog-sig list and here's the interesting bits I've noticed from the past few weeks:

[Catalog-sig] [Draft] Package signing and verification process http://mail.python.org/pipermail/catalog-sig/2013-February/004832.html

[Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security http://mail.python.org/pipermail/catalog-sig/2013-February/004994.html

Python PyPi Security Working Document:

https://docs.google.com/document/d/1e3g1v8INHjHsUJ-Q0odQOO8s91KMAbqLQyqj20CS...

Rubygems Threat Model: http://mail.python.org/pipermail/catalog-sig/2013-February/005099.html

https://docs.google.com/document/d/1fobWhPRqB4_JftFWh6iTWClUo_SPBnxqbBTdAvbb...

TUF: The Update Framework https://www.updateframework.com/

On Fri, Feb 1, 2013 at 4:07 AM, Christopher Done wrote:

...

Hey dude, it looks like we made the same project yesterday:

http://www.reddit.com/r/haskell/comments/17njda/proposal_a_trivial_cabal_pac...

Yours is nice as it doesn't depend on GPG. Although that could be a nice thing because GPG manages keys. Dunno.

Another diff is that mine puts the .sig inside the .tar.gz, yours puts it separate.

=)

...

On 01/30/2013 07:27 PM, Edward Z. Yang wrote:

...

https://status.heroku.com/incidents/489

Unsigned Hackage packages are a ticking time bomb.

I agree this is terrible, I've started working on this, but this is quite a bit of work and other priorities always pop up.

https://github.com/vincenthz/cabal https://github.com/vincenthz/cabal-signature

My current implementation generate a manifest during sdist'ing in cabal, and have cabal-signature called by cabal on the manifest to create a manifest.sign.

The main issue i'm facing is how to create a Web of Trust for doing all

On 31 January 2013 09:11, Vincent Hanquez wrote: the

...

public verification bits.

-- Vincent

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe