Making cabal-install SSL capable

Gershom B

27 Apr 2015 27 Apr '15

11:55 p.m.

So there are many discussions over various hackage security schemes, and there are a variety of takes on the different elements of how we could make package distribution more secure. However, everyone seems to agree that it would be unambiguously better if the cabal install executable were able to communicate over ssl. I looked at the previous discussion on this topic [1], and it seems that HsOpenSSL and tls were both considered. I don’t have any experience with how cross-platform compatible HsOpenSSL is (i.e. if it is sufficiently easy to use for both Windows and OS X that we can just encourage people to “cabal install cabal-install” and things will just work). I don’t know if anyone else can speak to this? Furthermore, of course, redistributing cabal-install binaries could potentially be more of a pain with links to external c libraries. I’m not quite sure how much an issue this would be. Meanwhile, tls is certainly cross-platform, but there is the question about how trustworthy it is, as it is not nearly as widely used and vetted as openssl. Also, we have the option of simply shelling out to curl, wget, or the appropriate powershell command (on windows 7 or above you get those by default). So rather than rely on either HsOpenSSL or tls, we could also teach cabal to probe for one of the appropriate executables on first run, save that configuration, and warn if no such executable is available (allowing the user to fall back to http with warnings indefinitely). I would like to pursue getting SSL into cabal by any of these three avenues. What do people feel about the relative tradeoffs of these options? Honestly, I lean towards simply using the tls package, because https is ultimately only going to be a complimentary aspect of our security architecture plans and not central to it. And a pure-haskell dependency is the most logical approach. If people find too much fault with that approach, I would be inclined to shell out as the next option, with HsOpenSSL as the last option only because I worry about too many “unknown unknowns” of the sort I listed above. But if others have more experience with these approaches, proposals are welcome! —Gershom

Show replies by date

Michael Snoyman

28 Apr 28 Apr

12:08 a.m.

+1. Oddly enough, I was just in the process of filing a bug report related to this on something I discovered yesterday, namely that `cabal upload` uses HTTP basic authentication over HTTP, exposing username/password to anyone sniffing the connection. I offered Duncan last week that I'd port cabal-install over to http-client/http-client-tls to add SSL support. That offer still stands. On Tue, Apr 28, 2015 at 6:55 AM Gershom B wrote:

...

So there are many discussions over various hackage security schemes, and there are a variety of takes on the different elements of how we could make package distribution more secure.

However, everyone seems to agree that it would be unambiguously better if the cabal install executable were able to communicate over ssl.

I looked at the previous discussion on this topic [1], and it seems that HsOpenSSL and tls were both considered. I don’t have any experience with how cross-platform compatible HsOpenSSL is (i.e. if it is sufficiently easy to use for both Windows and OS X that we can just encourage people to “cabal install cabal-install” and things will just work). I don’t know if anyone else can speak to this? Furthermore, of course, redistributing cabal-install binaries could potentially be more of a pain with links to external c libraries. I’m not quite sure how much an issue this would be. Meanwhile, tls is certainly cross-platform, but there is the question about how trustworthy it is, as it is not nearly as widely used and vetted as openssl.

Also, we have the option of simply shelling out to curl, wget, or the appropriate powershell command (on windows 7 or above you get those by default).

So rather than rely on either HsOpenSSL or tls, we could also teach cabal to probe for one of the appropriate executables on first run, save that configuration, and warn if no such executable is available (allowing the user to fall back to http with warnings indefinitely).

I would like to pursue getting SSL into cabal by any of these three avenues. What do people feel about the relative tradeoffs of these options? Honestly, I lean towards simply using the tls package, because https is ultimately only going to be a complimentary aspect of our security architecture plans and not central to it. And a pure-haskell dependency is the most logical approach. If people find too much fault with that approach, I would be inclined to shell out as the next option, with HsOpenSSL as the last option only because I worry about too many “unknown unknowns” of the sort I listed above. But if others have more experience with these approaches, proposals are welcome!

—Gershom _______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel

Herbert Valerio Riedel

4:12 a.m.

On 2015-04-28 at 06:08:38 +0200, Michael Snoyman wrote: [...]

...

I offered Duncan last week that I'd port cabal-install over to http-client/http-client-tls to add SSL support. That offer still stands.

I did a quick check trying to find out the additional dependencies (relative to what 'cabal-install' currently depends on) http-client-tls would pull in (it seems http-client and tls each roughly account for half the ~50 deps below): async-2.0.2 base64-bytestring-1.0.0.1 blaze-builder-0.4.0.1 byteable-0.1.1 cereal-0.4.1.1 clock-0.4.5.0 cryptohash-0.11.6 data-default-class-0.0.1 data-default-instances-base-0.0.1 data-default-instances-containers-0.0.1 data-default-instances-old-locale-0.0.1 dlist-0.7.1.1 data-default-instances-dlist-0.0.1 data-default-0.5.3 cookie-0.4.1.4 hashable-1.2.3.2 case-insensitive-1.2.0.4 hourglass-0.2.9 asn1-types-0.3.0 asn1-encoding-0.9.0 asn1-parse-0.9.0 crypto-pubkey-types-0.4.3 http-types-0.8.6 mime-types-0.1.0.6 pem-0.2.2 primitive-0.6 securemem-0.1.7 crypto-cipher-types-0.0.9 cipher-aes-0.2.10 cipher-des-0.0.6 cipher-rc4-0.1.4 socks-0.5.4 streaming-commons-0.1.12 transformers-compat-0.4.0.4 exceptions-0.8.0.2 utf8-string-1 publicsuffixlist-0.1 http-client-0.4.11.1 vector-0.10.12.3 crypto-random-0.0.9 crypto-numbers-0.2.7 crypto-pubkey-0.2.8 x509-1.5.0.1 x509-store-1.5.0 x509-system-1.5.0 x509-validation-1.5.1 tls-1.2.17 connection-0.2.4 http-client-tls-0.2.2 In contrast, I was surprised to see, that extending the HTTP package (or maybe just writing a 'HTTPS'-companion package) to use HsOpenSSL seems to pull in 'HsOpenSSL' as the only additional package... For comparision here's what http-streams (which I'm not suggesting right now, as I think going the 'HTTP'+'HsOpenSSL'-route would be better currently) would pull in (which could have a few deps less if it didn't pull in 'aeson'...): HsOpenSSL-0.11.1.1 base64-bytestring-1.0.0.1 blaze-builder-0.4.0.1 bytestring-builder-0.10.6.0.0 dlist-0.7.1.1 hashable-1.2.3.2 case-insensitive-1.2.0.4 primitive-0.6 scientific-0.3.3.8 attoparsec-0.12.1.6 syb-0.4.4 unordered-containers-0.2.5.1 http-common-0.8.2.0 vector-0.10.12.3 aeson-0.8.0.2 zlib-bindings-0.1.1.5 io-streams-1.3.0.0 openssl-streams-1.2.1.0 http-streams-0.8.3.1 Cheers, hvr

Michael Snoyman

4:21 a.m.

On Tue, Apr 28, 2015 at 11:12 AM Herbert Valerio Riedel wrote:

...

On 2015-04-28 at 06:08:38 +0200, Michael Snoyman wrote:

[...]

...
I offered Duncan last week that I'd port cabal-install over to http-client/http-client-tls to add SSL support. That offer still stands.

I did a quick check trying to find out the additional dependencies (relative to what 'cabal-install' currently depends on) http-client-tls would pull in (it seems http-client and tls each roughly account for half the ~50 deps below):

async-2.0.2 base64-bytestring-1.0.0.1 blaze-builder-0.4.0.1 byteable-0.1.1 cereal-0.4.1.1 clock-0.4.5.0 cryptohash-0.11.6 data-default-class-0.0.1 data-default-instances-base-0.0.1 data-default-instances-containers-0.0.1 data-default-instances-old-locale-0.0.1 dlist-0.7.1.1 data-default-instances-dlist-0.0.1 data-default-0.5.3 cookie-0.4.1.4 hashable-1.2.3.2 case-insensitive-1.2.0.4 hourglass-0.2.9 asn1-types-0.3.0 asn1-encoding-0.9.0 asn1-parse-0.9.0 crypto-pubkey-types-0.4.3 http-types-0.8.6 mime-types-0.1.0.6 pem-0.2.2 primitive-0.6 securemem-0.1.7 crypto-cipher-types-0.0.9 cipher-aes-0.2.10 cipher-des-0.0.6 cipher-rc4-0.1.4 socks-0.5.4 streaming-commons-0.1.12 transformers-compat-0.4.0.4 exceptions-0.8.0.2 utf8-string-1 publicsuffixlist-0.1 http-client-0.4.11.1 vector-0.10.12.3 crypto-random-0.0.9 crypto-numbers-0.2.7 crypto-pubkey-0.2.8 x509-1.5.0.1 x509-store-1.5.0 x509-system-1.5.0 x509-validation-1.5.1 tls-1.2.17 connection-0.2.4 http-client-tls-0.2.2

In contrast, I was surprised to see, that extending the HTTP package (or maybe just writing a 'HTTPS'-companion package) to use HsOpenSSL seems to pull in 'HsOpenSSL' as the only additional package...

For comparision here's what http-streams (which I'm not suggesting right now, as I think going the 'HTTP'+'HsOpenSSL'-route would be better currently) would pull in (which could have a few deps less if it didn't pull in 'aeson'...):

HsOpenSSL-0.11.1.1 base64-bytestring-1.0.0.1 blaze-builder-0.4.0.1 bytestring-builder-0.10.6.0.0 dlist-0.7.1.1 hashable-1.2.3.2 case-insensitive-1.2.0.4 primitive-0.6 scientific-0.3.3.8 attoparsec-0.12.1.6 syb-0.4.4 unordered-containers-0.2.5.1 http-common-0.8.2.0 vector-0.10.12.3 aeson-0.8.0.2 zlib-bindings-0.1.1.5 io-streams-1.3.0.0 openssl-streams-1.2.1.0 http-streams-0.8.3.1

Cheers, hvr

I have no intention of playing the "minimal dependency" game (though I don't mind dropping data-default, which accounts for 6 of the dependencies listed there). I will point out- as Gershom already did- that in many cases it's likely easier to install a few extra Haskell packages than it is to pull in OpenSSL as a dependency, especially on Windows. (And that's ignoring the fact that http-client-openssl exists.) As a historical point of interest, I originally wrote http-client (or, as it was called at the time, http-enumerator) because I was trying to add OpenID support to an application, and the openid package[1] had done exactly what you've described: add HsOpenSSL to the HTTP package. I could never get a single connection to work with that combination. But maybe a brand new approach at writing that code will work. Michael [1] http://hackage.haskell.org/package/openid

Thomas Tuegel

7:56 a.m.

On Tue, Apr 28, 2015 at 3:21 AM, Michael Snoyman wrote:

...

I have no intention of playing the "minimal dependency" game (though I don't mind dropping data-default, which accounts for 6 of the dependencies listed there). I will point out- as Gershom already did- that in many cases it's likely easier to install a few extra Haskell packages than it is to pull in OpenSSL as a dependency, especially on Windows. (And that's ignoring the fact that http-client-openssl exists.)

Considering users with cabal-install already, that many dependencies is only a small maintenance problem. Bootstrapping will not be a small problem. Gershom also pointed out the availability, on most platforms, of utilities which already provide HTTP support. That route seems to be the shortest distance between the problem and a solution.

...

As a historical point of interest, I originally wrote http-client (or, as it was called at the time, http-enumerator) because I was trying to add OpenID support to an application, and the openid package[1] had done exactly what you've described: add HsOpenSSL to the HTTP package. I could never get a single connection to work with that combination. But maybe a brand new approach at writing that code will work.

This report is yet another reason to disfavor HsOpenSSL. We already know it will add to installation difficulties. To that I will add: Gershom questioned the trustworthiness of tls, but do we really trust OpenSSL either? Do we know anything about the usability of the HTTP+tls combination? -- Thomas Tuegel

Michael Snoyman

8:38 a.m.

On Tue, Apr 28, 2015 at 2:56 PM Thomas Tuegel wrote:

...

...
I have no intention of playing the "minimal dependency" game (though I don't mind dropping data-default, which accounts for 6 of the dependencies

On Tue, Apr 28, 2015 at 3:21 AM, Michael Snoyman wrote: listed

...
there). I will point out- as Gershom already did- that in many cases it's likely easier to install a few extra Haskell packages than it is to pull in OpenSSL as a dependency, especially on Windows. (And that's ignoring the fact that http-client-openssl exists.)

Considering users with cabal-install already, that many dependencies is only a small maintenance problem. Bootstrapping will not be a small problem.

Can you explain what the bootstrapping issue is here? If it's what I think you're getting at, I think I have a simple solution to that: https://www.stackage.org/lts/build-plan?package=http-client&_accept=application/x-sh Note that this script generation isn't really highly tested, I put it together a few weeks back as a proof-of-concept to help out someone playing around with LTS Haskell + Nix. Michael

Herbert Valerio Riedel

9:22 a.m.

On 2015-04-28 at 10:21:04 +0200, Michael Snoyman wrote: [...]

...

I have no intention of playing the "minimal dependency" game (though I don't mind dropping data-default, which accounts for 6 of the dependencies listed there). I will point out- as Gershom already did- that in many cases it's likely easier to install a few extra Haskell packages than it is to pull in OpenSSL as a dependency, especially on Windows. (And that's ignoring the fact that http-client-openssl exists.)

While I do appreciate the technical issues resulting from HsOpenSSL on Windows[1], I'll be frank and admit that there's a "political" aspect that worries me with such a large number of added dependencies imported into the cabal project in one go, as that would promote (or at the very least bias) one specific family of multiple competing HTTP-client abstractions into the Haskell Platform through the back-door (assuming the idea of the HP hasn't been abandoned -- I may not be up to date regarding that debate), and make it a fait accompli without having actually had any agreement on it (which I admit may never be reached, as the associated communities involved have grown quite large by now and may disagree quite violently on basic design choices...). That's why I suggested HTTP+HsOpenSSL (which tbh is not my favorite HTTP library), as that would be the neutral choice regarding HTTP-libraries at the foundational core library level. Alternatively, Gershom's suggestion to shell out to curl(1)/wget(1)/etc would equally achieve impartiality regarding HTTP-libraries (and probably work better on Windows too) PS: We shouldn't forget that there's also an existing deployed cabal-install user-base we can't get rid off so easily, which may still leak unencrypted basic-auth credentials for the years to come. Just saying... [1]: Are those issues documented somewhere btw? Could that it be addressed via minghc? Cheers, hvr

Gershom B

9:40 a.m.

I don’t think the issue of http libraries as a foundational core library is very important. I suspect it would be possible to do a minimal http transport over tls, just as one can over HsOpenSSL, if that is how we decided to go. In the meantime, here are the first-order dependencies of tls alone, for comparison. I imagine there are a few others pulled in as well, in the second-order deps. cereal >= 0.4 , byteable , data-default-class -- crypto related , cryptohash >= 0.6 , crypto-random >= 0.0 && < 0.1 , crypto-numbers , crypto-cipher-types >= 0.0.8 , crypto-pubkey >= 0.2.8 , crypto-pubkey-types >= 0.4 , cipher-rc4 , cipher-des , cipher-aes >= 0.2 && < 0.3 -- certificate related , asn1-types >= 0.2.0 , asn1-encoding , x509 >= 1.5.0 && < 1.6.0 , x509-store >= 1.5.0 , x509-validation >= 1.5.1 && < 1.6.0 , async More troublesome, I recall that the current scheme for cabal is that it is fully bootstrappable by only depending on GHC base libs and “vendored” dependencies? In that case, perhaps we really should lean only towards the “shell out” solution. Or, I suppose, we could hide the ssl implementation behind a flag — but that feels rather awkward in comparison. So let’s try this: can anybody suggest any _downside_ to the “shell out” solution? —Gershom On April 28, 2015 at 9:22:22 AM, Herbert Valerio Riedel (hvriedel@gmail.com) wrote:

...

On 2015-04-28 at 10:21:04 +0200, Michael Snoyman wrote:

[...]

...
I have no intention of playing the "minimal dependency" game (though I don't mind dropping data-default, which accounts for 6 of the dependencies listed there). I will point out- as Gershom already did- that in many cases it's likely easier to install a few extra Haskell packages than it is to pull in OpenSSL as a dependency, especially on Windows. (And that's ignoring the fact that http-client-openssl exists.)

While I do appreciate the technical issues resulting from HsOpenSSL on Windows[1], I'll be frank and admit that there's a "political" aspect that worries me with such a large number of added dependencies imported into the cabal project in one go, as that would promote (or at the very least bias) one specific family of multiple competing HTTP-client abstractions into the Haskell Platform through the back-door (assuming the idea of the HP hasn't been abandoned -- I may not be up to date regarding that debate), and make it a fait accompli without having actually had any agreement on it (which I admit may never be reached, as the associated communities involved have grown quite large by now and may disagree quite violently on basic design choices...).

That's why I suggested HTTP+HsOpenSSL (which tbh is not my favorite HTTP library), as that would be the neutral choice regarding HTTP-libraries at the foundational core library level. Alternatively, Gershom's suggestion to shell out to curl(1)/wget(1)/etc would equally achieve impartiality regarding HTTP-libraries (and probably work better on Windows too)

PS: We shouldn't forget that there's also an existing deployed cabal-install user-base we can't get rid off so easily, which may still leak unencrypted basic-auth credentials for the years to come. Just saying...

[1]: Are those issues documented somewhere btw? Could that it be addressed via minghc?

Cheers, hvr _______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel

Thomas Tuegel

9:48 a.m.

On Tue, Apr 28, 2015 at 8:40 AM, Gershom B wrote:

...

More troublesome, I recall that the current scheme for cabal is that it is fully bootstrappable by only depending on GHC base libs and “vendored” dependencies?

Well, not really. We have an ugly bootstrapping script that downloads the other dependencies: https://github.com/haskell/cabal/blob/master/cabal-install/bootstrap.sh Speaking of which... On Tue, Apr 28, 2015 at 7:38 AM, Michael Snoyman wrote:

...

On Tue, Apr 28, 2015 at 2:56 PM Thomas Tuegel wrote:

...
Considering users with cabal-install already, that many dependencies is only a small maintenance problem. Bootstrapping will not be a small problem.

Can you explain what the bootstrapping issue is here? If it's what I think you're getting at, I think I have a simple solution to that:

https://www.stackage.org/lts/build-plan?package=http-client&_accept=application/x-sh

Note that this script generation isn't really highly tested, I put it together a few weeks back as a proof-of-concept to help out someone playing around with LTS Haskell + Nix.

Sorry, yes, the bootstrapping issue I referred to was installing cabal-install starting from just a bare GHC installation, as you correctly inferred. I think that script you generated is nicer than what we do now; that would be good to have, regardless of where we come down on the TLS situation. Is the source code that generates that script available? Thanks, -- Thomas Tuegel

Michael Snoyman

9:52 a.m.

On Tue, Apr 28, 2015 at 4:48 PM Thomas Tuegel wrote:

...

On Tue, Apr 28, 2015 at 8:40 AM, Gershom B wrote:

...
More troublesome, I recall that the current scheme for cabal is that it is fully bootstrappable by only depending on GHC base libs and “vendored” dependencies?

Well, not really. We have an ugly bootstrapping script that downloads the other dependencies: https://github.com/haskell/cabal/blob/master/cabal-install/bootstrap.sh

Speaking of which...

On Tue, Apr 28, 2015 at 7:38 AM, Michael Snoyman wrote:

...
On Tue, Apr 28, 2015 at 2:56 PM Thomas Tuegel wrote:

...
Considering users with cabal-install already, that many dependencies is only a small maintenance problem. Bootstrapping will not be a small problem.

Can you explain what the bootstrapping issue is here? If it's what I

think

...
you're getting at, I think I have a simple solution to that:

https://www.stackage.org/lts/build-plan?package=http-client&_accept=application/x-sh

...
Note that this script generation isn't really highly tested, I put it together a few weeks back as a proof-of-concept to help out someone

playing

...
around with LTS Haskell + Nix.

Sorry, yes, the bootstrapping issue I referred to was installing cabal-install starting from just a bare GHC installation, as you correctly inferred. I think that script you generated is nicer than what we do now; that would be good to have, regardless of where we come down on the TLS situation. Is the source code that generates that script available?

Thank you :) Yes, the code is part of the stackage-server code base, available here: https://github.com/fpco/stackage-server/blob/master/Handler/BuildPlan.hs#L40 When I was working on this for Nix, I offered to make the code a standalone library/tool instead. That offer's still open if it would be useful to others. Michael

Thomas Tuegel

11:08 a.m.

On Tue, Apr 28, 2015 at 8:52 AM, Michael Snoyman wrote:

...

Thank you :)

Yes, the code is part of the stackage-server code base, available here:

https://github.com/fpco/stackage-server/blob/master/Handler/BuildPlan.hs#L40

When I was working on this for Nix, I offered to make the code a standalone library/tool instead. That offer's still open if it would be useful to others.

I would like to take you up on that offer! I think it would need a few tweaks for our purposes, but this is very close to a feature I was planning to add to cabal-install. Our dependency requirements are already growing, and they will continue to do so, whether in small steps or leaps and bounds. Thanks, -- Thomas Tuegel

Michael Snoyman

12:10 p.m.

On Tue, Apr 28, 2015, 6:08 PM Thomas Tuegel wrote:

...

On Tue, Apr 28, 2015 at 8:52 AM, Michael Snoyman wrote:

...
Thank you :)

Yes, the code is part of the stackage-server code base, available here:

https://github.com/fpco/stackage-server/blob/master/Handler/BuildPlan.hs#L40

...
When I was working on this for Nix, I offered to make the code a

standalone

...
library/tool instead. That offer's still open if it would be useful to others.

I would like to take you up on that offer! I think it would need a few tweaks for our purposes, but this is very close to a feature I was planning to add to cabal-install. Our dependency requirements are already growing, and they will continue to do so, whether in small steps or leaps and bounds.

Cool, I'll be happy to help. I've opened up an issue about this: https://github.com/fpco/stackage-server/issues/95 Can you add some details about the kinds of changes you're looking for and we can start sketching out a plan? Michael

Michael Snoyman

10:11 a.m.

On Tue, Apr 28, 2015 at 4:40 PM Gershom B wrote:

...

In that case, perhaps we really should lean only towards the “shell out” solution. Or, I suppose, we could hide the ssl implementation behind a flag — but that feels rather awkward in comparison.

So let’s try this: can anybody suggest any _downside_ to the “shell out” solution?

Since I wasn't explicit (and my comments could be read to say otherwise): I am +1 on whatever approach ends up getting SSL support into cabal. If the least problematic approach ends up being "shell out", I'm OK with it (I even did that myself[1] in the distant past). I believe that shelling out would work for uploading as well as downloading, correct? Michael [1] http://hackage.haskell.org/package/http-wget

Michael Snoyman

9:50 a.m.

On Tue, Apr 28, 2015 at 4:22 PM Herbert Valerio Riedel wrote:

...

On 2015-04-28 at 10:21:04 +0200, Michael Snoyman wrote:

[...]

...
I have no intention of playing the "minimal dependency" game (though I don't mind dropping data-default, which accounts for 6 of the dependencies listed there). I will point out- as Gershom already did- that in many cases it's likely easier to install a few extra Haskell packages than it is to pull in OpenSSL as a dependency, especially on Windows. (And that's ignoring the fact that http-client-openssl exists.)

While I do appreciate the technical issues resulting from HsOpenSSL on Windows[1], I'll be frank and admit that there's a "political" aspect that worries me with such a large number of added dependencies imported into the cabal project in one go, as that would promote (or at the very least bias) one specific family of multiple competing HTTP-client abstractions into the Haskell Platform through the back-door (assuming the idea of the HP hasn't been abandoned -- I may not be up to date regarding that debate), and make it a fait accompli without having actually had any agreement on it (which I admit may never be reached, as the associated communities involved have grown quite large by now and may disagree quite violently on basic design choices...).

It may be worth looking at the reverse dependencies and download counts on this. But more importantly: I am most certainly not proposing that http-client be part of the platform. I discussed this already with Duncan last week: I have no idea why having the libraries in the platform should be necessary for a dependency of an executable.

...

That's why I suggested HTTP+HsOpenSSL (which tbh is not my favorite HTTP library), as that would be the neutral choice regarding HTTP-libraries at the foundational core library level. Alternatively, Gershom's suggestion to shell out to curl(1)/wget(1)/etc would equally achieve impartiality regarding HTTP-libraries (and probably work better on Windows too)

So to translate: since there are two good libraries for doing HTTP client side calls that both have TLS support, we can use neither of them, and must instead continue using the library we know to be inferior, *and* do extra dev work to add in TLS support, an endeavor which at least anecdotally has failed once before? By the way: I'm unaware of any violent disagreement around http-client vs http-streams. There are certainly differences between the libraries which I'd be happy to explore with others (in a separate thread, this isn't the place), but I think the possibility for a community brawl on this is slightly overstated.

...

PS: We shouldn't forget that there's also an existing deployed cabal-install user-base we can't get rid off so easily, which may still leak unencrypted basic-auth credentials for the years to come. Just saying...

I agree on that front. I think that Hackage should turn away all uploads that aren't TLS-secured, and should make that change ASAP.

...

[1]: Are those issues documented somewhere btw? Could that it be addressed via minghc?

There's not really anything to document. It's a matter of having to install the DLLs correctly, which people find difficult without a package manager. I'm in favor of supporting this situation better with MinGHC, but it would need someone else to spearhead it; I frankly have no itch to scratch here, since I'm already quite happily using the tls package. Michael

Herbert Valerio Riedel

10:32 a.m.

On 2015-04-28 at 15:50:26 +0200, Michael Snoyman wrote: [...]

...

...
PS: We shouldn't forget that there's also an existing deployed cabal-install user-base we can't get rid off so easily, which may still leak unencrypted basic-auth credentials for the years to come. Just saying...

...

I agree on that front. I think that Hackage should turn away all uploads that aren't TLS-secured, and should make that change ASAP.

Well, even if you do that, you can only reject the upload-request *after* the http client has already leaked the basic-auth credentials over a non-secured http channel... :-/ So the only thing this measure would buy us IMHO is that CLI users would get an incentive to upgrade their cabal-install tooling (if they use e.g. `cabal upload`), but it wouldn't protect against accidentally falling back to an older cabal-install version picked up by accident (and then again compromising the credentials). I.e. this measure on its own wouldn't remove the unsecured basic-auth eavesdropping attack-window completely, only make it smaller. Cheers, hvr

Michael Snoyman

12:13 p.m.

On Tue, Apr 28, 2015 at 5:32 PM Herbert Valerio Riedel wrote:

...

On 2015-04-28 at 15:50:26 +0200, Michael Snoyman wrote:

[...]

...
...
PS: We shouldn't forget that there's also an existing deployed cabal-install user-base we can't get rid off so easily, which may still leak unencrypted basic-auth credentials for the years to come. Just saying...

...
I agree on that front. I think that Hackage should turn away all uploads that aren't TLS-secured, and should make that change ASAP.

Well, even if you do that, you can only reject the upload-request *after* the http client has already leaked the basic-auth credentials over a non-secured http channel... :-/

So the only thing this measure would buy us IMHO is that CLI users would get an incentive to upgrade their cabal-install tooling (if they use e.g. `cabal upload`), but it wouldn't protect against accidentally falling back to an older cabal-install version picked up by accident (and then again compromising the credentials). I.e. this measure on its own wouldn't remove the unsecured basic-auth eavesdropping attack-window completely, only make it smaller.

Cheers, hvr

Your analysis is accurate. There are some interesting approaches we could take to further mitigate things. For example: newer versions of cabal-install could automatically set an incorrect username/password in the ~/.cabal/config file, and create a new set of fields (ssl-username/ssl-password?) that it would recognize. A more radical approach would be to have Hackage simply turn off port 80. But that's probably too extreme; having a 301 redirect from HTTP to HTTPS is really necessary for a good user experience, and even that redirect would still expose the password vulnerability window. Michael

Herbert Valerio Riedel

29 Apr 29 Apr

3 a.m.

On 2015-04-28 at 18:13:47 +0200, Michael Snoyman wrote: [...]

...

Your analysis is accurate. There are some interesting approaches we could take to further mitigate things. For example: newer versions of cabal-install could automatically set an incorrect username/password in the ~/.cabal/config file, and create a new set of fields (ssl-username/ssl-password?) that it would recognize.

Now that you mention that; the typical default config entry looks like: remote-repo: hackage.haskell.org:http://hackage.haskell.org/packages/archive For a TLS-capable cabal-install (for which I'll assume it'll use digest-auth), I'd suggest (as a strawman proposal - so please criticize!) that it a) when creating a default-config, write remote-repo: hackage.haskell.org:https://hackage.haskell.org/packages/archive b) if a cabal upload operation is performed, 1) if remote-repo points to a http:// url, Warn the user in a visible scary way that can't easily be ignored that he/she should switch to a https:// url; however, proceed w/ the requested operation using digest-auth 2) if remote-repo points to a https:// url, operate w/o warning (could use basic-auth instead of digest-auth, if we wanted to avoid an additional round-trip at the HTTP-layer) c) for non-authorised operations, emit less-scary warnings as a friendly reminder to update the repo-url to a https:// capable one So once a the cabal-config file points to a https://, url, the risk of older non-TLS-capable cabal-installs leaking credentials via unencrypted HTTP would be solved (at the expense of older cabal-install versions not being able to communicate via https:// urls at all...) Cheers, hvr

Michael Snoyman

3:52 a.m.

On Wed, Apr 29, 2015 at 10:00 AM Herbert Valerio Riedel wrote:

...

On 2015-04-28 at 18:13:47 +0200, Michael Snoyman wrote:

[...]

...
Your analysis is accurate. There are some interesting approaches we could take to further mitigate things. For example: newer versions of cabal-install could automatically set an incorrect username/password in the ~/.cabal/config file, and create a new set of fields (ssl-username/ssl-password?) that it would recognize.

Now that you mention that; the typical default config entry looks like:

remote-repo: hackage.haskell.org:http:// hackage.haskell.org/packages/archive

For a TLS-capable cabal-install (for which I'll assume it'll use digest-auth), I'd suggest (as a strawman proposal - so please criticize!) that it

a) when creating a default-config, write

remote-repo: hackage.haskell.org:https:// hackage.haskell.org/packages/archive

b) if a cabal upload operation is performed,

1) if remote-repo points to a http:// url, Warn the user in a visible scary way that can't easily be ignored that he/she should switch to a https:// url; however, proceed w/ the requested operation using digest-auth

2) if remote-repo points to a https:// url, operate w/o warning (could use basic-auth instead of digest-auth, if we wanted to avoid an additional round-trip at the HTTP-layer)

c) for non-authorised operations, emit less-scary warnings as a friendly reminder to update the repo-url to a https:// capable one

So once a the cabal-config file points to a https://, url, the risk of older non-TLS-capable cabal-installs leaking credentials via unencrypted HTTP would be solved (at the expense of older cabal-install versions not being able to communicate via https:// urls at all...)

Cheers, hvr

I'd probably be more aggressive, and simply rewrite the cabal.config for users (with a message) to change http://hackage... to https://hackage. But I'm OK with just give a bunch of warnings to encourage the user to do it him/herself.

Yitzchak Gale

30 Apr 30 Apr

6:12 a.m.

As a user of both http-client and tls, I can vouch for having very good experience with both, and I thank Michael for his offer. I prefer that option. However - Both extra Haskell dependencies and C dependencies are a problem for cabal-install. The reason is that cabal is a basic requirement to bootstrap a Haskell installation from scratch. So either of those will make life very much harder for distro packagers, builders of Haskell Platform or other such from-scratch installation options, and anyone who needs to get a Haskell tool chain installed on a platform that is non-standard in some way. So I suggest that we have basic cabal-install - perhaps stripped down even more than it is now - which is designed for easy porting, with no C dependency and minimal Haskell dependencies. Furthermore, that basic cabal-install does need some SSL option - probably Gershom's shell-out option, even though I don't prefer that for every day use, because it can be implemented with minimal dependencies. The reason is the same - for porting. Once you have cabal-install-basic installed, the only thing you may ever use it for is a one-time installation of full cabal-install. But that's a very sensitive installation - if that is compromised, you're hosed. So if there's any time you need SSL, it's then. In summary - I vote for cabal-install-basic with no C deps, very minimal Haskell deps, possibly stripped-down cabal functionality if needed, and a simple SSL shell-out option designed for maximum portability. And then, one or more full-featured cabal-install packages, with one of them being the tls option. Thanks, Yitz

Thomas Tuegel

8:25 a.m.

On Thu, Apr 30, 2015 at 5:12 AM, Yitzchak Gale wrote:

...

Both extra Haskell dependencies and C dependencies are a problem for cabal-install.

The reason is that cabal is a basic requirement to bootstrap a Haskell installation from scratch. So either of those will make life very much harder for distro packagers, builders of Haskell Platform or other such from-scratch installation options, and anyone who needs to get a Haskell tool chain installed on a platform that is non-standard in some way.

Michael is putting together some tools for us that should make added Haskell dependencies a non-issue from the standpoint of distributing cabal-install. We already have a bootstrap script for this purpose, but the new tools should make it more robust. So, I don't think we should continue to be concerned by that aspect of additional Haskell dependencies.

...

In summary - I vote for cabal-install-basic with no C deps, very minimal Haskell deps, possibly stripped-down cabal functionality if needed, and a simple SSL shell-out option designed for maximum portability. And then, one or more full-featured cabal-install packages, with one of them being the tls option.

I am emphatically against any plan that involves optional dependencies or distributing two versions of anything. Our users, especially newcomers, will be confused. Which cabal-install do I need? Which cabal-install do I have? Why are there two versions? Why doesn't that version do this? etc. Especially because one version would be used once and necessarily be discarded, we would need to make the 2-step installation automatic. That leads us back to a bootstrap script, which we're already doing. If our current bootstrap facilities are inadequate, then we need to address that instead. Distributing two versions also means maintaining two code paths. Maintaining and testing two *security sensitive* code paths. I would much rather see us put all our eggs in the http-client basket than distribute two versions. I say this even as someone who favors the pipe-out-to-curl option. I have had a bad experience with every HTTP client library on Hackage (that I know of) and I would take any one of them over any one of them, plus curl. -- Thomas Tuegel

Duncan Coutts

28 Apr 28 Apr

10:58 a.m.

On Mon, 2015-04-27 at 23:55 -0400, Gershom B wrote:

...

So there are many discussions over various hackage security schemes, and there are a variety of takes on the different elements of how we could make package distribution more secure.

However, everyone seems to agree that it would be unambiguously better if the cabal install executable were able to communicate over ssl.

Yeah.

...

I looked at the previous discussion on this topic [1], and it seems that HsOpenSSL and tls were both considered. I don’t have any experience with how cross-platform compatible HsOpenSSL is (i.e. if it is sufficiently easy to use for both Windows and OS X that we can just encourage people to “cabal install cabal-install” and things will just work). I don’t know if anyone else can speak to this? Furthermore, of course, redistributing cabal-install binaries could potentially be more of a pain with links to external c libraries. I’m not quite sure how much an issue this would be. Meanwhile, tls is certainly cross-platform, but there is the question about how trustworthy it is, as it is not nearly as widely used and vetted as openssl.

Right. I don't have any great suggestion there. This is what has prevented us getting anywhere in the past.

...

Also, we have the option of simply shelling out to curl, wget, or the appropriate powershell command (on windows 7 or above you get those by default).

I think that's quite a reasonable 90% solution. I was looking at implementing this the other day after talking about it with Michael. When looking at the code I noticed (as did Michael) that rather embarrassingly the current code is actually using basic auth, when it should have been using digest auth. I've sent a PR: https://github.com/haskell/cabal/pull/2563

...

So rather than rely on either HsOpenSSL or tls, we could also teach cabal to probe for one of the appropriate executables on first run, save that configuration, and warn if no such executable is available (allowing the user to fall back to http with warnings indefinitely).

I don't even think it needs any saved configuration. Cabal is pretty good at probing for and running exes.

...

I would like to pursue getting SSL into cabal by any of these three avenues. What do people feel about the relative tradeoffs of these options? Honestly, I lean towards simply using the tls package, because https is ultimately only going to be a complimentary aspect of our security architecture plans and not central to it. And a pure-haskell dependency is the most logical approach. If people find too much fault with that approach, I would be inclined to shell out as the next option, with HsOpenSSL as the last option only because I worry about too many “unknown unknowns” of the sort I listed above. But if others have more experience with these approaches, proposals are welcome!

My suggestion is that in the short term we use an external curl binary if it happens to be available, and fallback to digest auth if not. If/when we are in a position to have dependencies on decent http(s) libraries then we should use those. Also, for those users without curl (or without an HTTPS-enabled version), the digest auth can be improved. We can use digest auth with the "auth-int" QoP, which includes replay protection and integrity protection of the message body (ie the .tar.gz). It's not as good as TLS, but digest auth (with "auth" or "auth-int" QoP) isn't actually known to be broken yet. (It's an oldish standard that relies on MD5, but has both server and client nonces to protect against chosen plaintext attacks). Duncan

Gregory Collins

11:37 a.m.

On Tue, Apr 28, 2015 at 7:58 AM, Duncan Coutts

...

wrote:

...

...
However, everyone seems to agree that it would be unambiguously better if the cabal install executable were able to communicate over ssl.

Yeah.

Strongly +1'd. Following this track, if cabal's http handling is going to be abstract enough to be able to shell out to curl, there's no reason not to add an optional dependency on http-client (either haskell-tls or openssl) or http-streams, guarded by a flag. G -- Gregory Collins

Bardur Arantsson

11:51 a.m.

On 28-04-2015 16:58, Duncan Coutts wrote:

...

On Mon, 2015-04-27 at 23:55 -0400, Gershom B wrote:

...
I would like to pursue getting SSL into cabal by any of these three avenues. What do people feel about the relative tradeoffs of these options? Honestly, I lean towards simply using the tls package, because https is ultimately only going to be a complimentary aspect of our security architecture plans and not central to it. And a pure-haskell dependency is the most logical approach. If people find too much fault with that approach, I would be inclined to shell out as the next option, with HsOpenSSL as the last option only because I worry about too many “unknown unknowns” of the sort I listed above. But if others have more experience with these approaches, proposals are welcome!

My suggestion is that in the short term we use an external curl binary if it happens to be available, and fallback to digest auth if not. If/when we are in a position to have dependencies on decent http(s) libraries then we should use those.

+1 I wonder what curl's defaults are wrt. accepting self-signed certs and/or certs without a valid trust chain in the system's default certificate store. I guess someone would have to go through all the options and see what's necessary here. (I guess there might be some issues here with old versions of "curl" vs. newer versions, but at least some of the curl options have an "Added in X.Y.Z" which I could be used to guide decisions here.) Regards,

3673

Age (days ago)

3675

Last active (days ago)

List overview

Download

22 comments

8 participants

participants (8)

Bardur Arantsson
Duncan Coutts
Gershom B
Gregory Collins
Herbert Valerio Riedel
Michael Snoyman
Thomas Tuegel
Yitzchak Gale

Making cabal-install SSL capable

tags

participants (8)