Local mirrors of hackage

older
ICFP 2016 Final Call for Papers

Victor Miller

24 Feb 2016 24 Feb '16

11:23 a.m.

I have a computer that I work on which isn't directly connected to the internet, that I want to use Haskell on. In order to do that I want a local version of hackage. I was previously able to do this using mirror_hackage.py, but since hackage 2 came around this no longer works. Doing this would involve running something on a computer that *is* attached to the internet, and then transferring a bunch of files over to the other computer. Can someone point me to a recipe for doing this? Victor

Show replies by date

Matthew Pickering

24 Feb 24 Feb

11:27 a.m.

Hello Victor, It is easily possible with a small shell script. Here is another discussion about the same topic - https://mail.haskell.org/pipermail/ghc-devs/2015-September/009958.html Matt On Wed, Feb 24, 2016 at 4:23 PM, Victor Miller wrote:

...

I have a computer that I work on which isn't directly connected to the internet, that I want to use Haskell on. In order to do that I want a local version of hackage. I was previously able to do this using mirror_hackage.py, but since hackage 2 came around this no longer works. Doing this would involve running something on a computer that *is* attached to the internet, and then transferring a bunch of files over to the other computer. Can someone point me to a recipe for doing this?

Victor _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

Roman Cheplyaka

11:36 a.m.

On 02/24/2016 06:23 PM, Victor Miller wrote:

...

I have a computer that I work on which isn't directly connected to the internet, that I want to use Haskell on. In order to do that I want a local version of hackage. I was previously able to do this using mirror_hackage.py, but since hackage 2 came around this no longer works. Doing this would involve running something on a computer that *is* attached to the internet, and then transferring a bunch of files over to the other computer. Can someone point me to a recipe for doing this?

I don't know how exactly you want to use Haskell on your computer, but if you know in advance which packages you'll need, you can download only those packages (and their versions) and place them into cabal or stack cache directory. That will be much more efficient than mirroring whole of hackage. Roman

John Wiegley

1:16 p.m.

...

...
...
...
...
Victor Miller writes:

...

I have a computer that I work on which isn't directly connected to the internet, that I want to use Haskell on. In order to do that I want a local version of hackage. I was previously able to do this using mirror_hackage.py, but since hackage 2 came around this no longer works. Doing this would involve running something on a computer that *is* attached to the internet, and then transferring a bunch of files over to the other computer. Can someone point me to a recipe for doing this?

I wrote https://github.com/jwiegley/hackage-mirror to do just this, with relatively high performance for incremental updates. Also, if you're ever a Nix user, you can modify the core Haskell builder to reference whatever directory you mirror to first, before reaching out to the Internet for tarballs. I've been using this effectively for several years now. -- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

Victor Miller

2:37 p.m.

John, Thanks very much. Victor On Wed, Feb 24, 2016 at 1:16 PM, John Wiegley wrote:

...

...
...
...
...
...
Victor Miller writes:

...
I have a computer that I work on which isn't directly connected to the internet, that I want to use Haskell on. In order to do that I want a local version of hackage. I was previously able to do this using mirror_hackage.py, but since hackage 2 came around this no longer works. Doing this would involve running something on a computer that *is* attached to the internet, and then transferring a bunch of files over to the other computer. Can someone point me to a recipe for doing this?

I wrote https://github.com/jwiegley/hackage-mirror to do just this, with relatively high performance for incremental updates.

Also, if you're ever a Nix user, you can modify the core Haskell builder to reference whatever directory you mirror to first, before reaching out to the Internet for tarballs. I've been using this effectively for several years now.

-- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

Vo Minh Thu

3:24 p.m.

Here is a script that I use. The whole mirror can also be downloaded through rsync if that helps. https://github.com/noteed/nginx-hackage 2016-02-24 20:37 GMT+01:00 Victor Miller :

...

John, Thanks very much.

Victor

On Wed, Feb 24, 2016 at 1:16 PM, John Wiegley wrote:

...
...
...
...
...
> Victor Miller writes:

...
I have a computer that I work on which isn't directly connected to the internet, that I want to use Haskell on. In order to do that I want a local version of hackage. I was previously able to do this using mirror_hackage.py, but since hackage 2 came around this no longer works. Doing this would involve running something on a computer that *is* attached to the internet, and then transferring a bunch of files over to the other computer. Can someone point me to a recipe for doing this?

I wrote https://github.com/jwiegley/hackage-mirror to do just this, with relatively high performance for incremental updates.

Also, if you're ever a Nix user, you can modify the core Haskell builder to reference whatever directory you mirror to first, before reaching out to the Internet for tarballs. I've been using this effectively for several years now.

-- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

Victor Miller

4:31 p.m.

John, Thanks very much for hackage-mirror. I'm running it now and it appears to be working as advertised. Thanks to everyone else who responded. Victor On Wed, Feb 24, 2016 at 1:16 PM, John Wiegley wrote:

...

...
...
...
...
...
Victor Miller writes:

...
I have a computer that I work on which isn't directly connected to the internet, that I want to use Haskell on. In order to do that I want a local version of hackage. I was previously able to do this using mirror_hackage.py, but since hackage 2 came around this no longer works. Doing this would involve running something on a computer that *is* attached to the internet, and then transferring a bunch of files over to the other computer. Can someone point me to a recipe for doing this?

I wrote https://github.com/jwiegley/hackage-mirror to do just this, with relatively high performance for incremental updates.

Also, if you're ever a Nix user, you can modify the core Haskell builder to reference whatever directory you mirror to first, before reaching out to the Internet for tarballs. I've been using this effectively for several years now.

-- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

John Wiegley

4:45 p.m.

...

...
...
...
...
Victor Miller writes:

...

John, Thanks very much for hackage-mirror. I'm running it now and it appears to be working as advertised. Thanks to everyone else who responded.

I committed a fix very recently to the 'pipes' branch that you may need, since there is one package on Hackage that is missing and thus cannot be downloaded. -- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

Victor Miller

3 Mar 3 Mar

12:26 p.m.

Hi John, I just got around to installing the downloaded hackage mirror, and I noticed that its structure is different than the one that I downloaded last June -- in the old one there are files 00-index.cache 00-index.tar and a bunch of subdirectories, one per package. Within each of those is a version subdirectory which contains a cabal file and a tar.gz file of the package files. In the new one there are files 00-checksums.bak 00-checksums.dat 00-index.tar.gz and a directory called package That directory contains directories called package-name-version and a corresponding file called package-name-version.tar.gz one for each package and version My version of cabal 1.22 doesn't understand that. Do I need a new version of cabal, or do I have to process the files somehow? Victor On Wed, Feb 24, 2016 at 4:45 PM, John Wiegley wrote:

...

...
...
...
...
...
Victor Miller writes:

...
John, Thanks very much for hackage-mirror. I'm running it now and it appears to be working as advertised. Thanks to everyone else who responded.

I committed a fix very recently to the 'pipes' branch that you may need, since there is one package on Hackage that is missing and thus cannot be downloaded.

-- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

Victor Miller

1:47 p.m.

Aha, I figured out the problem. I needed to add the lines remote-repo: myname:file://path-to-hackage remote-repo-cache: cache-file to my ~/.cabal/config Now it works fine. I wish that the instructions were a little less obscure. Victor On Thu, Mar 3, 2016 at 12:26 PM, Victor Miller wrote:

...

Hi John, I just got around to installing the downloaded hackage mirror, and I noticed that its structure is different than the one that I downloaded last June -- in the old one there are files

00-index.cache 00-index.tar and a bunch of subdirectories, one per package. Within each of those is a version subdirectory which contains a cabal file and a tar.gz file of the package files.

In the new one there are files 00-checksums.bak 00-checksums.dat 00-index.tar.gz

and a directory called package

That directory contains directories called package-name-version and a corresponding file called package-name-version.tar.gz

one for each package and version

My version of cabal 1.22 doesn't understand that. Do I need a new version of cabal, or do I have to process the files somehow?

Victor

On Wed, Feb 24, 2016 at 4:45 PM, John Wiegley wrote:

...
...
...
...
...
> Victor Miller writes:

...
John, Thanks very much for hackage-mirror. I'm running it now and it appears to be working as advertised. Thanks to everyone else who responded.

I committed a fix very recently to the 'pipes' branch that you may need, since there is one package on Hackage that is missing and thus cannot be downloaded.

-- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2

3365

Age (days ago)

3373

Last active (days ago)

List overview

Download

9 comments

5 participants

participants (5)

John Wiegley
Matthew Pickering
Roman Cheplyaka
Victor Miller
Vo Minh Thu