Hackage2 server ready for testing

Ian Lynagh

24 Aug 2012 24 Aug '12

11:17 a.m.

Hi all, All the bugs identified in the Hackage 2 server as blockers have now been fixed, and I have a mirror of Hackage running here: http://new-hackage.haskell.org/ I suggest that we proceed with: * A 3 week testing period, to be announced on haskell@ and libraries@ * Assuming no new blocking bugs are found which justify additional testing time, disable uploads to the Hackage 1 server * Throw the testing Hackage 2 instance away, and make a new mirror from the Hackage 1 server * Point hackage.haskell.org at the new Hackage 2 instance I think that a 3 week testing period will be sufficient as I expect most testing will happen in the first week or two, or the last few days, so any extra time is essentially wasted. I was actually initially going to propose 2 weeks, but given the timing of ICFP et al I think an extra week is justified. The reason I propose throwing away this test instance is that I think it's important that people are able to test uploading, and therefore uploading dummy packages and dummy versions should be encouraged. Does that plan sound reasonable to everyone? Is anyone willing to volunteer to be an admin for the new Hackage server please? (people need to have an admin add their account to the uploaders group before they can upload). A couple more notes: The mirror is a week or two old, so the latest package uploads won't be on it. I don't plan to set up continuous mirroring, as I think it may be more useful for people to be able to upload new versions to both servers themselves, for testing. The documentation builder has built (or failed to build) docs for the latest versions of all packages, but it's still working through older versions. It will therefore take up to 35mins before it notices that a new package has been uploaded, and builds docs for it. Finally, I would like to thank the Industrial Haskell Group (http://industry.haskell.org/) for funding this work. Thanks Ian -- Ian Lynagh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Show replies by date

Johan Tibell

24 Aug 24 Aug

1 p.m.

I just tried to upload a package, after registering for an account, and I get this error, without being prompted for username and password as I would have expected: Error: Forbidden No access for this page. This is after selecting a file and clicking the upload button.

Ian Lynagh

1:12 p.m.

On Fri, Aug 24, 2012 at 10:00:02AM -0700, Johan Tibell wrote:

...

I just tried to upload a package, after registering for an account, and I get this error, without being prompted for username and password as I would have expected:

Error: Forbidden No access for this page.

This is after selecting a file and clicking the upload button.

I can't reproduce that. Are you sure you weren't already logged in? If you think it's a bug, please give me full reproduction instructions, starting from a freshly started browser (so it doesn't have any passwords cached) and registering a fresh account. Thanks Ian

Johan Tibell

1:24 p.m.

On Fri, Aug 24, 2012 at 10:12 AM, Ian Lynagh wrote:

...

I can't reproduce that. Are you sure you weren't already logged in?

If you think it's a bug, please give me full reproduction instructions, starting from a freshly started browser (so it doesn't have any passwords cached) and registering a fresh account.

The reason is probably that the browser saved my password after I failed to log in after creating an account (I guess an admin have to approve?).

Ian Lynagh

1:30 p.m.

On Fri, Aug 24, 2012 at 10:24:44AM -0700, Johan Tibell wrote:

...

On Fri, Aug 24, 2012 at 10:12 AM, Ian Lynagh wrote:

...
I can't reproduce that. Are you sure you weren't already logged in?

If you think it's a bug, please give me full reproduction instructions, starting from a freshly started browser (so it doesn't have any passwords cached) and registering a fresh account.

The reason is probably that the browser saved my password after I failed to log in after creating an account (I guess an admin have to approve?).

Technically admins don't have to approve accounts, but they do have to add them to the 'uploaders' group, which is pretty much the same thing from the user's point of view. Thanks Ian

Erik Hesselink

3:36 p.m.

Hi Ian, Looks great, thanks for the work! One question: I see the reverse dependencies have been disabled, darcs says for 'performance problems'. Do you have some details? Do you think it is hard to fix? I wouldn't mind doing some permission granting for the testing period, by the way. Erik On Fri, Aug 24, 2012 at 5:17 PM, Ian Lynagh wrote:

...

Hi all,

All the bugs identified in the Hackage 2 server as blockers have now been fixed, and I have a mirror of Hackage running here: http://new-hackage.haskell.org/

I suggest that we proceed with:

* A 3 week testing period, to be announced on haskell@ and libraries@ * Assuming no new blocking bugs are found which justify additional testing time, disable uploads to the Hackage 1 server * Throw the testing Hackage 2 instance away, and make a new mirror from the Hackage 1 server * Point hackage.haskell.org at the new Hackage 2 instance

I think that a 3 week testing period will be sufficient as I expect most testing will happen in the first week or two, or the last few days, so any extra time is essentially wasted. I was actually initially going to propose 2 weeks, but given the timing of ICFP et al I think an extra week is justified.

The reason I propose throwing away this test instance is that I think it's important that people are able to test uploading, and therefore uploading dummy packages and dummy versions should be encouraged.

Does that plan sound reasonable to everyone?

Is anyone willing to volunteer to be an admin for the new Hackage server please? (people need to have an admin add their account to the uploaders group before they can upload).

A couple more notes:

The mirror is a week or two old, so the latest package uploads won't be on it. I don't plan to set up continuous mirroring, as I think it may be more useful for people to be able to upload new versions to both servers themselves, for testing.

The documentation builder has built (or failed to build) docs for the latest versions of all packages, but it's still working through older versions. It will therefore take up to 35mins before it notices that a new package has been uploaded, and builds docs for it.

Finally, I would like to thank the Industrial Haskell Group (http://industry.haskell.org/) for funding this work.

Thanks Ian -- Ian Lynagh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

Ian Lynagh

30 Aug 30 Aug

11:19 a.m.

Hi Erik, On Fri, Aug 24, 2012 at 09:36:23PM +0200, Erik Hesselink wrote:

...

One question: I see the reverse dependencies have been disabled, darcs says for 'performance problems'. Do you have some details? Do you think it is hard to fix?

There were 2 aspects to it. First, it is slow: I think there's a comment somewhere saying that it takes 2 seconds whenever a package is added. IIRC there are 23000+ packages in Hackage, so doing a full Hackage mirror would take approximately an extra half day with this enabled. Second, it looked like it might be the cause of some space leaks. It's possible that this was fixed by me making space usage fixes elsewhere since removing it. I doubt that it is hard to fix, but I haven't looked at it at all. Thanks Ian

Matthew Gruen

31 Aug 31 Aug

1:16 p.m.

Reverse dependencies can mean different things. The current implementation answers these questions: 1. What are the packages for which the latest version depends directly on a given version of foo? 2. What are the packages for which only previous or deprecated versions depend directly on a given version of foo? 3. For which packages do any versions depend either directly and indirectly on any version of bar? (this is what takes 2 seconds to regenerate, as incremental transitive closure wasn't implemented) 4. Which packages are most depended-upon (only direct, or both direct and indirect)? It seems to me as though 1 is the most important question to answer. This involves knowing, for any package and version, the list of package-version pairs which depend on it. This involves using a fair bit of memory, it seems. Additionally, when a package is uploaded, removed, or changed, the packages which now depend on it also need to be updated. Another question answered by packdeps is: 5. Which packages depend on a previous or deprecated version of foo but not the latest version? (and vice versa) This can also be answered by the above data structure. It is currently implemented using nested Data.Maps, which is serialized in the database. There are some opportunities for trimming and optimization. Best, Matt On Fri, Aug 24, 2012 at 3:36 PM, Erik Hesselink wrote:

...

Hi Ian,

Looks great, thanks for the work! One question: I see the reverse dependencies have been disabled, darcs says for 'performance problems'. Do you have some details? Do you think it is hard to fix?

I wouldn't mind doing some permission granting for the testing period, by the way.

Erik

On Fri, Aug 24, 2012 at 5:17 PM, Ian Lynagh wrote:

...
Hi all,

All the bugs identified in the Hackage 2 server as blockers have now been fixed, and I have a mirror of Hackage running here: http://new-hackage.haskell.org/

I suggest that we proceed with:

* A 3 week testing period, to be announced on haskell@ and libraries@ * Assuming no new blocking bugs are found which justify additional testing time, disable uploads to the Hackage 1 server * Throw the testing Hackage 2 instance away, and make a new mirror from the Hackage 1 server * Point hackage.haskell.org at the new Hackage 2 instance

I think that a 3 week testing period will be sufficient as I expect most testing will happen in the first week or two, or the last few days, so any extra time is essentially wasted. I was actually initially going to propose 2 weeks, but given the timing of ICFP et al I think an extra week is justified.

The reason I propose throwing away this test instance is that I think it's important that people are able to test uploading, and therefore uploading dummy packages and dummy versions should be encouraged.

Does that plan sound reasonable to everyone?

Is anyone willing to volunteer to be an admin for the new Hackage server please? (people need to have an admin add their account to the uploaders group before they can upload).

A couple more notes:

The mirror is a week or two old, so the latest package uploads won't be on it. I don't plan to set up continuous mirroring, as I think it may be more useful for people to be able to upload new versions to both servers themselves, for testing.

The documentation builder has built (or failed to build) docs for the latest versions of all packages, but it's still working through older versions. It will therefore take up to 35mins before it notices that a new package has been uploaded, and builds docs for it.

Finally, I would like to thank the Industrial Haskell Group (http://industry.haskell.org/) for funding this work.

Thanks Ian -- Ian Lynagh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

Ben Millwood

24 Aug 24 Aug

7:13 p.m.

On Fri, Aug 24, 2012 at 4:17 PM, Ian Lynagh wrote:

...

Is anyone willing to volunteer to be an admin for the new Hackage server please? (people need to have an admin add their account to the uploaders group before they can upload).

Is this going to be just "get e-mail, press corresponding button"? If so I would be happy to help. If it's more complicated than that then, well, I'd still be happy to help, but I worry that it's a lot of responsibility (and I'd understand if I was insufficiently trusted in The Community to do it).

Simon Hengel

25 Aug 25 Aug

1:12 p.m.

Hi, with Firefox, I sometimes get an "XML Parsing error" on the package index. Exact symptoms: Firefox shows an error message (black and red text on yellow background) while loading/rendering. After a short time (< 1s) the page still loads successfully. Steps to reproduce: - point Firefox to http://new-hackage.haskell.org/packages/ - repeatedly reload the page (ctrl-r), until the symptoms occur I ran into this with Firefox 14.0.1 on Linux. Cheers, Simon

Ian Lynagh

30 Aug 30 Aug

10:57 a.m.

On Sat, Aug 25, 2012 at 07:12:41PM +0200, Simon Hengel wrote:

...

Hi, with Firefox, I sometimes get an "XML Parsing error" on the package index.

Exact symptoms:

Firefox shows an error message (black and red text on yellow background) while loading/rendering. After a short time (< 1s) the page still loads successfully.

Steps to reproduce:

- point Firefox to http://new-hackage.haskell.org/packages/

- repeatedly reload the page (ctrl-r), until the symptoms occur

Thanks for the report. Unfortunately, I can't reproduce this. I tried reloading several times in a web browser, but hit no problems. Then I downloaded the URL 1000 times in series. Then I downloaded 10 copies in parallel 100 times. All 2000 copies are identical and end with a </html> tag. Thanks Ian

Simon Hengel

6 Sep 6 Sep

5 p.m.

Hi Ian, sorry for the late reply. If I'm not explicitly on TO/CC it is likely that I miss that mail. On Thu, Aug 30, 2012 at 03:57:50PM +0100, Ian Lynagh wrote:

...

On Sat, Aug 25, 2012 at 07:12:41PM +0200, Simon Hengel wrote:

...
Hi, with Firefox, I sometimes get an "XML Parsing error" on the package index.

Exact symptoms:

Firefox shows an error message (black and red text on yellow background) while loading/rendering. After a short time (< 1s) the page still loads successfully.

Steps to reproduce:

- point Firefox to http://new-hackage.haskell.org/packages/

- repeatedly reload the page (ctrl-r), until the symptoms occur

Thanks for the report. Unfortunately, I can't reproduce this.

I tried reloading several times in a web browser, but hit no problems.

I think it is a Firefox problem, at least I can't reproduce it with Chromium. Did you try with Firefox? Cheers, Simon

Ian Lynagh

7 Sep 7 Sep

8:39 a.m.

On Thu, Sep 06, 2012 at 11:00:47PM +0200, Simon Hengel wrote:

...

On Thu, Aug 30, 2012 at 03:57:50PM +0100, Ian Lynagh wrote:

...
On Sat, Aug 25, 2012 at 07:12:41PM +0200, Simon Hengel wrote:

...
Firefox shows an error message (black and red text on yellow background) while loading/rendering. After a short time (< 1s) the page still loads successfully.

Thanks for the report. Unfortunately, I can't reproduce this.

I tried reloading several times in a web browser, but hit no problems.

I think it is a Firefox problem, at least I can't reproduce it with Chromium. Did you try with Firefox?

I was using iceweasel. Do you have a proxy set in firefox? Thanks Ian -- Ian Lynagh, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Simon Hengel

9:27 a.m.

On Fri, Sep 07, 2012 at 01:39:18PM +0100, Ian Lynagh wrote:

...

On Thu, Sep 06, 2012 at 11:00:47PM +0200, Simon Hengel wrote:

...
On Thu, Aug 30, 2012 at 03:57:50PM +0100, Ian Lynagh wrote:

...
On Sat, Aug 25, 2012 at 07:12:41PM +0200, Simon Hengel wrote:

...
Firefox shows an error message (black and red text on yellow background) while loading/rendering. After a short time (< 1s) the page still loads successfully.

Thanks for the report. Unfortunately, I can't reproduce this.

I tried reloading several times in a web browser, but hit no problems.

I think it is a Firefox problem, at least I can't reproduce it with Chromium. Did you try with Firefox?

I was using iceweasel. Do you have a proxy set in firefox?

No. I tried with a vanilla Firefox profile, and it is still reproducible here (FF 15.0 now, the build that comes with the latest Ubuntu). Cheers, Simon

Matthew Gruen

9:35 a.m.

If I may take a stab, I think this issue is related to the use of strict XHTML. When you refresh, your browser renders what you have so far; if this is non-matching XHTML (that is, invalid XML), it will render an error. I think rendering strict and serving as transitional would fix this. In the long run, there is likely more merit in being HTML5-compliant than anything else for HTML resources. On Fri, Sep 7, 2012 at 9:27 AM, Simon Hengel wrote:

...

...
On Thu, Sep 06, 2012 at 11:00:47PM +0200, Simon Hengel wrote:

...
On Thu, Aug 30, 2012 at 03:57:50PM +0100, Ian Lynagh wrote:

...
On Sat, Aug 25, 2012 at 07:12:41PM +0200, Simon Hengel wrote:

...
Firefox shows an error message (black and red text on yellow background) while loading/rendering. After a short time (<

1s) the

...
...
...
page still loads successfully.

Thanks for the report. Unfortunately, I can't reproduce this.

I tried reloading several times in a web browser, but hit no

On Fri, Sep 07, 2012 at 01:39:18PM +0100, Ian Lynagh wrote: problems.

...
...
I think it is a Firefox problem, at least I can't reproduce it with Chromium. Did you try with Firefox?

I was using iceweasel. Do you have a proxy set in firefox?

No. I tried with a vanilla Firefox profile, and it is still reproducible here (FF 15.0 now, the build that comes with the latest Ubuntu).

Cheers, Simon

_______________________________________________ cabal-devel mailing list cabal-devel@haskell.org http://www.haskell.org/mailman/listinfo/cabal-devel

4638

Age (days ago)

4652

Last active (days ago)

List overview

Download

14 comments

6 participants

participants (6)

Ben Millwood
Erik Hesselink
Ian Lynagh
Johan Tibell
Matthew Gruen
Simon Hengel