Ok, on my "https" branch here: https://github.com/gbaz/Cabal/tree/https We have some initial support for multibackend downloads and uploads both. Lots of bits are still missing, but the basic downloads work with curl and wget both, and the basic uploads with curl. The code regarding uploading build reports (which isn't in use anyway) is now commented out temporarily. Any work at adding other backends or further improving these (note the many TODO comments littering the code) can hopefully build on this. In particular, note there's an HttpTransportType, and a function `findHttpTransport`. In turn both `getHTTP` and `uploadToURI` now call this function, then case on the result to dispatch to different transports. All the transport specific logic should now be localized to those two places. This code is still rather ugly and could use some cleanup. But it should provide an extension point for using other https layers as well. On top of that, I haven't tackled the issue of making choice of transport configurable instead of just doing auto-discovery in a fixed order. It seems to me that this is something that could be added to the config file, or a command line flag, or both. So anyone that wants to fixate on that part of the problem, feel free to jump in :-) For my part, I wanted to get this out there to help anyone else interested in adding a transport. At this point, I intend mainly to just clean up what exists, fill in the powershell (and possibly bitsadmin) transport for windows, and then reenable HTTP as a fallback layer in some fashion (with warnings or confirmation or something). (I had commented out the layer entirely to make sure I wasn't missing any uses of it). Again, normally I wouldn't be sharing something in this halfway shape, but it seems useful to make it available for others interested in doing related work. Cheers, Gershom On Tue, May 5, 2015 at 10:15 AM, Gershom B wrote:

Note that I’ve started some refactorings to support curl/wget/etc already. I’m not sure how best to keep these patches from stepping on one another’s toes. I’ll try to get my stuff to an unfinished, untested stopping point where nonetheless there is a clear extension point, and then perhaps a fork off of that branch would make sense?

—Gershom

...

Hi,

On 4 May 2015 at 08:31, Michael Snoyman wrote:

...

Just a little update on this. I pinged the author of publicsuffixlist (necessary for proper cookie domain handling) about removing the data-default dependency, and after discussion we decided to just merge

...

...

code into http-client instead. With that change, the full dependency

On May 5, 2015 at 7:29:21 AM, Mikhail Glushenkov ( the.dead.shall.rise@gmail.com) wrote: the list

...

...

for http-client-openssl is in fact smaller that http-streams:

I think we would accept a patch making cabal-install depend on http-client or http-streams if it was enabled with a flag. Refactorings required to make this work would also make it simpler to add support for curl/wget/whatever. _______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/cabal-devel

The “Transport” data type you suggest seems like a good refactor. I’m less certain about the use of ProgramDB — the documentation suggests it will only be populated from command line flags in the “configure” step, or from .cabal files. Since upload/download do not take place in such settings, I don’t think it will contain useful information yet. So it might be the right approach to use it, but it would also require, I suspect, modifying other parts of the cabal-install executable to populate it properly? —Gershom On May 6, 2015 at 3:32:44 AM, Mikhail Glushenkov (the.dead.shall.rise@gmail.com) wrote:

On 6 May 2015 at 00:26, Gershom B wrote:

...

Ok, on my "https" branch here: https://github.com/gbaz/Cabal/tree/https

Looks like a good first step. Consider using machinery in Distribution.Simple.Program.Db for searching for curl/wget in path or in user-provided location.

I'd also use something like

data Transport = { getHttp :: URI -> FilePath -> IO (Int, String), putHttp :: URI -> FilePath -> ...}

configureTransport :: ProgramDB -> IO Transport

instead of switching on TransportType in getHttp/uploadToUri.

On Wed, May 6, 2015 at 11:53 AM, Mikhail Glushenkov wrote:

On 6 May 2015 at 17:43, Gershom B wrote:

...

I’m less certain about the use of ProgramDB — the documentation suggests it will only be populated from command line flags in the “configure” step, or from .cabal files. Since upload/download do not take place in such settings, I don’t think it will contain useful information yet. So it might be the right approach to use it, but it would also require, I suspect, modifying other parts of the cabal-install executable to populate it properly?

You may need to use a separate ProgramDb (populated at runtime) just for curl/wget and friends. The point is that we already have some code for locating executables that supports setting program locations on command line ('--with-PROG-path') and in the config file, so it'd be nice to reuse it.

The saved program configuration is not serialized in `dist/setup-config`, [1] so it must be regenerated at runtime every time, anyway. [1]. https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple.hs#L4... -- Thomas Tuegel

On my branch now, everything "should" work: https://github.com/gbaz/Cabal/tree/https By "should" I mean the refactor is in place, and we have support for curl, wget, powershell, and insecure-http. You can pass a global flag to set your preferred transport (and it will not try insecure-http unless you explictly request it). If you use any transport but insecure-http it will convert http to https urls automatically. Things that remain outstanding 1) ETags are not enabled on the new transports -- this is just some tedious work to emit them from the programs and parse them out. 2) http-transport is a global option and can be passed, but it doesn't seem to wind up in the documentation correctly. Not sure how to configure the options properly. 3) Proxy support is not fully tested on curl/wget -- it _did_ work on my machine at one point, but it lives behind a particularly perverse proxy and ceased working in a way I can't diagnose and might just be due to a hopeless proxy. 4) I haven't properly tested upload support on wget, which is fiddly since it needs to do its own multipart encoding just like insecure-http. 5) Proxies are not enabled on powershell -- this is some tedious work to automate parsing out the proxy settings better, since the "correct powershell" way to handle proxies with user/passwords doesn't seem to actually work. 6) Powershell also needs real escaping, in case package names/urls perversely have quotation marks in them. Anyway, this touched a bunch of stuff in small ways and could use some review and _especially_ some testers across various platforms and transports to see what's still missing, If any brave souls want to try out the various transports, or pick up some of the remaining work, please do! This was a nice exercise, but I could really use another set or two of eyes/hands to take it to 100%. (note that even though I branched before the digest fix, I ported it over to this branch, so that the insecure-http transport _should_ be using digest auth properly already). --Gershom On Thu, May 7, 2015 at 5:02 AM, Mikhail Glushenkov < the.dead.shall.rise@gmail.com> wrote:

Hi,

On 6 May 2015 at 20:08, Thomas Tuegel wrote:

...

The saved program configuration is not serialized in `dist/setup-config`, [1] so it must be regenerated at runtime every time, anyway.

We don't want to configure all known built-in programs on each 'cabal upload', though, so we should introduce a 'configTransport' function to be used in 'uploadAction' instead of 'configCompiler'.

I've been happily using the curl backend for a bit now and it seems to work fine, so I'd like to get this polished and committed. Since the curl backend works, then even if the wget and powershell backends aren't fully tested, the workaround can just be to tell users to install curl and make sure it is in their path (or to pass the insecure-http option that falls back to the existing implementation). So perhaps we can just patch cabal to warn on using wget or powershell backends that if there are issues they should try one of the fully finished options, and then we can merge? Alternately, someone can jump in and try to give a hand with the other two backends -- they're very close. -g On Fri, May 8, 2015 at 12:46 PM, Mikhail Glushenkov < the.dead.shall.rise@gmail.com> wrote:

On 8 May 2015 at 00:14, Gershom B wrote:

...

On my branch now, everything "should" work: https://github.com/gbaz/Cabal/tree/https

Thanks for working on this. I only had time to take a brief glance, but everything looks good so far. Hope we can merge your patches soon.

...

(note that even though I branched before the digest fix, I ported it over to this branch, so that the insecure-http transport _should_ be using digest auth properly already).

You could've just rebased (which is a good practice anyway).

Done: https://github.com/haskell/cabal/pull/2613 :-) --Gershom On Mon, May 18, 2015 at 4:15 PM, Mikhail Glushenkov < the.dead.shall.rise@gmail.com> wrote:

Hi,

On 18 May 2015 at 20:22, Gershom B wrote:

...

So perhaps we can just patch cabal to warn on using wget or powershell backends that if there are issues they should try one of the fully finished options, and then we can merge?

Sure, please send us a pull request.