Ongoing IHG work to improve Hackage security

All, The IHG members identified Hackage security as an important issue some time ago and myself and my colleague Austin have been working on a design and implementation. The details are in this blog post: http://www.well-typed.com/blog/2015/04/improving-hackage-security We should have made more noise earlier about the fact that we're working on this. We saw that it was important to finally write this up now because other similar ideas are under active discussion and we don't want to cause too much unnecessary duplication. The summary is this: We're implementing a system to significantly improve Hackage security. It's based on a sensible design (The Update Framework) by proper crypto experts. The basic system is fully automatic and covers all packages on Hackage. A proposed extension would give further security improvements for individual packages at the cost of a modest effort from package authors. http://theupdateframework.com/ It will also allow the secure use of untrusted public Hackage mirrors, which is the simplest route to better Hackage reliability. As a bonus we're including incremental index downloads to reduce `cabal update` wait times. And it's all fully backwards compatible. I should also note that our IHG funding covers the first phase of the design, and for the second phase we would very much welcome others to get involved with the detailed design and implementation (or join the IHG and contribute further funding). -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

I've read the blog post, and am still trying to understand the implications
of TUF. However, it's incredibly difficult to give solid review of a system
which is stated to be "based on TUF," without knowing what the delta
implied by that is. For now, I can only ask simple questions:
1. Is there any timeline for the changes needed to Hackage and
cabal-install?
2. Is there any idea of how much extra code will need to be maintained
going forward for this? This is an important point, given that both Hackage
and Cabal are having trouble keeping up with demand already.
3. Is there any mitigation of eavesdropping attacks on the authorization
headers sent to Hackage by uploaders for digest authentication?
4. Is there any mitigation against a compromise of the Hackage server
itself, either the code base or the system?
Overall, I'm quite wary of a solution stated as "experts devised this, it's
good, we'll implement it, everyone stop worrying." I take your point about
crypto-humility, but I'm not confident that an approach based on TUF
addresses that concern since it involves a new implementation (or
copy-pasted implementation) of crypto primitives together with unspecified
changes to TUF.
Note: wary is *not* a code word for opposed, but I strongly believe that
anything we do here warrants far more discussion, and that can't happen
until more details are explained.
On Thu, Apr 16, 2015 at 12:33 PM Duncan Coutts
All,
The IHG members identified Hackage security as an important issue some time ago and myself and my colleague Austin have been working on a design and implementation.
The details are in this blog post:
http://www.well-typed.com/blog/2015/04/improving-hackage-security
We should have made more noise earlier about the fact that we're working on this. We saw that it was important to finally write this up now because other similar ideas are under active discussion and we don't want to cause too much unnecessary duplication.
The summary is this:
We're implementing a system to significantly improve Hackage security. It's based on a sensible design (The Update Framework) by proper crypto experts. The basic system is fully automatic and covers all packages on Hackage. A proposed extension would give further security improvements for individual packages at the cost of a modest effort from package authors.
http://theupdateframework.com/
It will also allow the secure use of untrusted public Hackage mirrors, which is the simplest route to better Hackage reliability. As a bonus we're including incremental index downloads to reduce `cabal update` wait times. And it's all fully backwards compatible.
I should also note that our IHG funding covers the first phase of the design, and for the second phase we would very much welcome others to get involved with the detailed design and implementation (or join the IHG and contribute further funding).
-- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/
-- You received this message because you are subscribed to the Google Groups "Commercial Haskell" group. To unsubscribe from this group and stop receiving emails from it, send an email to commercialhaskell+unsubscribe@googlegroups.com. To post to this group, send email to commercialhaskell@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/commercialhaskell/1429176835.25663.30.came... . For more options, visit https://groups.google.com/d/optout.

Hi,
On 16 April 2015 at 11:33, Duncan Coutts
All,
The IHG members identified Hackage security as an important issue some time ago and myself and my colleague Austin have been working on a design and implementation.
The details are in this blog post:
http://www.well-typed.com/blog/2015/04/improving-hackage-security
Thank you, this is very exciting. But won't the post-release .cabal update feature interfere with "package index as an append-only log" concept? IIUC, right now it is implemented as a destructive update of the corresponding package index entry, so making the package index immutable will break backwards compatibility.

On Thu, 2015-04-16 at 15:14 +0200, Mikhail Glushenkov wrote:
Hi,
On 16 April 2015 at 11:33, Duncan Coutts
wrote: All,
The IHG members identified Hackage security as an important issue some time ago and myself and my colleague Austin have been working on a design and implementation.
The details are in this blog post:
http://www.well-typed.com/blog/2015/04/improving-hackage-security
Thank you, this is very exciting. But won't the post-release .cabal update feature interfere with "package index as an append-only log" concept? IIUC, right now it is implemented as a destructive update of the corresponding package index entry, so making the package index immutable will break backwards compatibility.
Yes, we can use the tar file in an append-only way while allowing metadata updates because that's the tar file format supports that. The tar file format was originally designed for tape drives where rewinding and updating old entries was far too expensive. So the tar file format allows appending updated file entries to the end of the archive. Compliant tar tools (including the standard unix tools, and cabal-install) understand this and take the last entry in the archive as the current file content. -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Hi,
On 16 April 2015 at 15:34, Duncan Coutts
Compliant tar tools (including the standard unix tools, and cabal-install) understand this and take the last entry in the archive as the current file content.
Thanks. I looked at the code again, and while this is not explicitly mentioned in comments, we get this behaviour for free by relying on Map.fromList.

On Thu, 2015-04-16 at 15:56 +0200, Mikhail Glushenkov wrote:
Hi,
On 16 April 2015 at 15:34, Duncan Coutts
wrote: Compliant tar tools (including the standard unix tools, and cabal-install) understand this and take the last entry in the archive as the current file content.
Thanks. I looked at the code again, and while this is not explicitly mentioned in comments, we get this behaviour for free by relying on Map.fromList.
Sorry, I should have added more comments there. I was aware of this issue when I wrote the tar package (indeed I found out more about the history of the tar format than is really healthy for anyone). -- Duncan Coutts, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/
participants (3)
-
Duncan Coutts
-
Michael Snoyman
-
Mikhail Glushenkov