Status update on {code, trac, projects, planet, community}.haskell.org

newer
WGP 2011: Call for Papers

older
`cabal install darcs` fail !

Duncan Coutts

1 Feb 2011 1 Feb '11

8:33 p.m.

All, As you will be aware, some of the *.haskell.org websites have been down recently, specifically: code.haskell.org trac.haskell.org projects.haskell.org planet.haskell.org community.haskell.org These are all hosted on the community server. The community server was hacked on the 26th January and we took it offline. The server was running an old version of debian that was no longer supported with security updates. (Ironically, two days previously the infrastructure team had been discussing the fact that nobody seemed to have any time available to do the planned migration to a new host). The hacker replaced sshd which we noticed because the ssh host signature changed and it started prompting for passwords (we use key-based rather than password based logins). MSR kindly allowed Ian to take time off from the GHC 7.0.2 release to work on migrating the services to the new host (that had previously been partially prepared). Thanks to Ian's efforts and some help from other members of the infrastructure team, the migration is now nearly complete. planet.haskell.org should now be working, along with email, project mailing lists and trac instances. We have not yet re-enabled user login accounts, nor re-enabled access to code repositories. The support ticket system is not yet enabled. We will send a further update when these are re-enabled, or procedures for people to re-enable them are finalised. On a more positive note, the new VM that we are using is a lot more powerful than the old one, in particular about 5x more memory. As many of you will have experienced, services on the old server tended to go AWOL which was primarily due to running up against memory limits, causing services to get killed off. In the case of the web server it was mainly due to having to use a very small number of concurrent connections, again to minimise memory use. So all in all we expect the new server to be a good deal more reliable. Duncan (On behalf of the Haskell infrastructure team)

Show replies by date

Roman Cheplyaka

3 Feb 3 Feb

3:37 a.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

* Duncan Coutts [2011-02-02 01:33:22+0000]

...

These are all hosted on the community server. The community server was hacked on the 26th January and we took it offline. The server was running an old version of debian that was no longer supported with security updates. (Ironically, two days previously the infrastructure team had been discussing the fact that nobody seemed to have any time available to do the planned migration to a new host). The hacker replaced sshd which we noticed because the ssh host signature changed and it started prompting for passwords (we use key-based rather than password based logins).

Might be related: http://sourceforge.net/blog/sourceforge-attack-full-report/ -- Roman I. Cheplyaka :: http://ro-che.info/ Don't worry what people think, they don't do it very often.

Duncan Coutts

6:45 a.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

On Thu, 2011-02-03 at 10:37 +0200, Roman Cheplyaka wrote:

...

* Duncan Coutts [2011-02-02 01:33:22+0000]

...
These are all hosted on the community server. The community server was hacked on the 26th January and we took it offline. The server was running an old version of debian that was no longer supported with security updates. (Ironically, two days previously the infrastructure team had been discussing the fact that nobody seemed to have any time available to do the planned migration to a new host). The hacker replaced sshd which we noticed because the ssh host signature changed and it started prompting for passwords (we use key-based rather than password based logins).

Might be related: http://sourceforge.net/blog/sourceforge-attack-full-report/

Yes, it's quite possible. One difference to note is that we use ssh key based logins, not passwords. We suspect this saved us from the worst case scenarios. Nevertheless, while we don't have to reset passwords, we are concerned about the potential that the attacker replaced or added to users ~/.ssh/authorized_keys lists, which is why we have not yet re-enabled user accounts. We will try and provide as full a picture as we can when we're satisfied we've got as much info and confidence as we're likely to get. Duncan

Duncan Coutts

7 Feb 7 Feb

2:37 p.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

On Wed, 2011-02-02 at 01:33 +0000, Duncan Coutts wrote:

...

As you will be aware, some of the *.haskell.org websites have been down recently, specifically:

code.haskell.org trac.haskell.org projects.haskell.org planet.haskell.org community.haskell.org

[...]

...

We have not yet re-enabled user login accounts, nor re-enabled access to code repositories. The support ticket system is not yet enabled.

We will send a further update when these are re-enabled, or procedures for people to re-enable them are finalised.

We have restored read-only access to the majority of projects on: code.haskell.org (code repositories) projects.haskell.org (project webspace) A small number have been held back because they contain .js, .html or .tar.gz snapshots that are not recorded in any repository. We will ask project owners to check these before they are restored. Duncan (On behalf of the Haskell infrastructure team)

Duncan Coutts

15 Feb 15 Feb

9:12 p.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

On Wed, 2011-02-02 at 01:33 +0000, Duncan Coutts wrote:

...

All,

As you will be aware, some of the *.haskell.org websites have been down recently, specifically:

code.haskell.org trac.haskell.org projects.haskell.org planet.haskell.org community.haskell.org

[...]

...

We have not yet re-enabled user login accounts, nor re-enabled access to code repositories. We will send a further update when these are re-enabled, or procedures for people to re-enable them are finalised.

Logging in ========== We have restored ssh logins for around 250 user accounts (ie darcs push will work). If you are not one of those 250 and you cannot log in then you will need to email support@community.haskell.org. Give your real name, your unix user name and attach your current ssh public key. Once you have logged in ======================= Personal webspace ----------------- public URL: http://code.haskell.org/~$username/ server-side: ~/public_html(-disabled) You will notice that your ~/public_html directory has been renamed to ~/public_html-disabled. There is a slim possibility that the data was altered when the server was compromised. We recommend that you check it first and then to restore use: mv ~/public_html-disabled ~/public_html Code repositories ----------------- public URL: http://code.haskell.org/$projname/ server-side: /srv/code/$projname/ or: /srv/srv-from-nun/code/{checked-failed,checked-strayfiles}/$projname/ Similarly, many code repositories (44) have not been re-enabled. Ones that we could check automatically have already been restored. If the /srv/code/$project directory for your project is empty or missing then you will find it in one of the directories in /srv/srv-from-nun/code/, either checked-failed/ if "darcs check" failed on that repository, or in checked-strayfiles/ if the repository contains extra unrecorded files that we could not check automatically. You should check that you are satisfied that the repository contains just what you expect and then email support@community.haskell.org to ask for it to be moved back to the usual location. Project websites ---------------- public URL: http://projects.haskell.org/$projname/ server-side: /srv/projects/${projname/ or: /srv/srv-from-nun/projects/$projname/ If the /srv/projects/$project directory for your project is empty or missing then will find the project website in /srv/srv-from-nun/projects/$project. You should check that you are satisfied that the website directory contains just what you expect and then email support@community.haskell.org to ask for it to be moved back to the usual location. Explanation =========== We believe that when the server was compromised, the attacker was mainly interested in collecting usernames and passwords. Since we do not use password based logins, we think the attacker was not successful in this. However we are unable to trust any of the ~/.ssh/authorized_keys because the attacker could have modified them to give access at a later date. We were able to verify the ~/.ssh/authorized_keys for around 250 users by comparing the current file against the key that was originally submitted in the account creation request. People who have added keys or changed keys since initial account creation have not had their login access restored and they must resend their current key. For html, css, javascript files etc, there was the slight concern that the attacker may have defaced sites or made malicious files available for download. While we have not found any instance of this so far, we need the help of project owners to check this. Duncan (On behalf of the Haskell infrastructure team)

Henning Thielemann

16 Feb 16 Feb

12:58 p.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

Thank you a lot for bringing code.haskell.org back! I missed it a lot! However, I still get $ ssh code.haskell.org @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The RSA host key for code.haskell.org has changed, and the key for the according IP address 178.63.91.44 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. ... How can I verify that it is safe to continue logging in? Could you restore the old host key? On Wed, 16 Feb 2011, Duncan Coutts wrote:

...

Code repositories -----------------

public URL: http://code.haskell.org/$projname/ server-side: /srv/code/$projname/ or: /srv/srv-from-nun/code/{checked-failed,checked-strayfiles}/$projname/

Similarly, many code repositories (44) have not been re-enabled. Ones that we could check automatically have already been restored.

If the /srv/code/$project directory for your project is empty or missing then you will find it in one of the directories in /srv/srv-from-nun/code/, either checked-failed/ if "darcs check" failed on that repository, or in checked-strayfiles/ if the repository contains extra unrecorded files that we could not check automatically.

You should check that you are satisfied that the repository contains just what you expect and then email support@community.haskell.org to ask for it to be moved back to the usual location.

This causes a lot manual work for you. It would be easier for you and us to allow project maintainers to move the repositories from srv-from-nun to /src/code.

Roel van Dijk

17 Feb 17 Feb

3:31 a.m.

I believe code.haskell.org has moved to a new machine. Its IP address also changed, which causes your ssh to issue a warning. You can fix it by deleting the code.haskell.org entry from your local ~/.ssh/known_hosts file. On 16 February 2011 18:58, Henning Thielemann wrote:

...

Thank you a lot for bringing code.haskell.org back! I missed it a lot! However, I still get

$ ssh code.haskell.org @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The RSA host key for code.haskell.org has changed, and the key for the according IP address 178.63.91.44 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. ...

How can I verify that it is safe to continue logging in? Could you restore the old host key?

Duncan Coutts

16 Feb 16 Feb

7:59 p.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

On Wed, 2011-02-16 at 02:12 +0000, Duncan Coutts wrote:

...

...
We have not yet re-enabled user login accounts, nor re-enabled access to code repositories. We will send a further update when these are re-enabled, or procedures for people to re-enable them are finalised.

Logging in ==========

We have restored ssh logins for around 250 user accounts (ie darcs push will work).

Several people have asked about the new host key. Yes, there is a new RSA host key for the community server, the fingerprint of which is: 21:b8:59:ff:39:69:58:7a:51:ef:c1:d8:c6:24:6e:f7 ssh will likely give you a scary warning and you'll need to delete the old entry in your ~/.ssh/known_hosts file. You don't need to enter a new one, just delete the old one. When you next log into the server, ssh will ask you if you're happy with the new key. If you're paranoid, you can double check that it matches the key fingerprint above. Duncan

Henning Thielemann

17 Feb 17 Feb

1:30 p.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

Duncan Coutts schrieb:

...

Several people have asked about the new host key. Yes, there is a new RSA host key for the community server, the fingerprint of which is:

21:b8:59:ff:39:69:58:7a:51:ef:c1:d8:c6:24:6e:f7

ssh will likely give you a scary warning and you'll need to delete the old entry in your ~/.ssh/known_hosts file. You don't need to enter a new one, just delete the old one. When you next log into the server, ssh will ask you if you're happy with the new key. If you're paranoid, you can double check that it matches the key fingerprint above.

Do you think it is paranoid? Unfortunately it has become quite common to ignore SSH warnings because admins often do not care about restoring keys when updating the operating system or moving the machine, even not telling users that the host key has changed. But if I had ignored the SSH warning on code.haskell.org recently I might have logged in and from there maybe to other servers, thus giving my passwords to the attackers. I think generally that just deleting a host from known_hosts in response to an SSH warning and blindly accepting a new host key is not a fix. Am I too afraid?

Vo Minh Thu

1:53 p.m.

2011/2/17 Henning Thielemann :

...

Duncan Coutts schrieb:

...
Several people have asked about the new host key. Yes, there is a new RSA host key for the community server, the fingerprint of which is:

21:b8:59:ff:39:69:58:7a:51:ef:c1:d8:c6:24:6e:f7

ssh will likely give you a scary warning and you'll need to delete the old entry in your ~/.ssh/known_hosts file. You don't need to enter a new one, just delete the old one. When you next log into the server, ssh will ask you if you're happy with the new key. If you're paranoid, you can double check that it matches the key fingerprint above.

Do you think it is paranoid? Unfortunately it has become quite common to ignore SSH warnings because admins often do not care about restoring keys when updating the operating system or moving the machine, even not telling users that the host key has changed. But if I had ignored the SSH warning on code.haskell.org recently I might have logged in and from there maybe to other servers, thus giving my passwords to the attackers. I think generally that just deleting a host from known_hosts in response to an SSH warning and blindly accepting a new host key is not a fix. Am I too afraid?

Hi, Regarding you giving passwords when logging in other marchines, I think it would not be the case if you only use key authentication from machines to machines. Your private key can be only on your local machine and you can use an ssh agent to do log from machines to machines. Cheers, Thu

Vincent Hanquez

1:54 p.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

On Thu, Feb 17, 2011 at 07:30:23PM +0100, Henning Thielemann wrote:

...

Do you think it is paranoid? Unfortunately it has become quite common to ignore SSH warnings because admins often do not care about restoring keys when updating the operating system or moving the machine, even not telling users that the host key has changed. But if I had ignored the SSH warning on code.haskell.org recently I might have logged in and from there maybe to other servers, thus giving my passwords to the attackers. I think generally that just deleting a host from known_hosts in response to an SSH warning and blindly accepting a new host key is not a fix. Am I too afraid?

If sshd has been compromised, so is the original host private key. It would be kind of pointless (security wise) to restore it on the new server. -- Vincent

Duncan Coutts

18 Feb 18 Feb

6:36 a.m.

New subject: Status update on {code, trac, projects, planet, community}.haskell.org

On Thu, 2011-02-17 at 19:30 +0100, Henning Thielemann wrote:

...

Duncan Coutts schrieb:

...
Several people have asked about the new host key. Yes, there is a new RSA host key for the community server, the fingerprint of which is:

21:b8:59:ff:39:69:58:7a:51:ef:c1:d8:c6:24:6e:f7

ssh will likely give you a scary warning and you'll need to delete the old entry in your ~/.ssh/known_hosts file. You don't need to enter a new one, just delete the old one. When you next log into the server, ssh will ask you if you're happy with the new key. If you're paranoid, you can double check that it matches the key fingerprint above.

Do you think it is paranoid?

Sorry, I didn't mean it literally (or pejoratively).

...

Unfortunately it has become quite common to ignore SSH warnings because admins often do not care about restoring keys when updating the operating system or moving the machine, even not telling users that the host key has changed. But if I had ignored the SSH warning on code.haskell.org recently I might have logged in and from there maybe to other servers, thus giving my passwords to the attackers. I think generally that just deleting a host from known_hosts in response to an SSH warning and blindly accepting a new host key is not a fix. Am I too afraid?

No, you're quite right. It was these warnings that initially alerted us to the problem. Duncan

5207

Age (days ago)

5223

Last active (days ago)

List overview

Download

11 comments

6 participants

participants (6)

Duncan Coutts
Henning Thielemann
Roel van Dijk
Roman Cheplyaka
Vincent Hanquez
Vo Minh Thu