921
edits
Great Hack of 2008: Difference between revisions
m
no edit summary
No edit summary |
mNo edit summary |
||
| Line 20: | Line 20: | ||
==== Friday ==== | ==== Friday Morning ==== | ||
| Line 47: | Line 47: | ||
* 10:30 - final mass hey goes out before shutdown | * 10:30 - final mass hey goes out before shutdown | ||
* 10:32 - access to the network is cut off. | * 10:32 - access to the network is cut off. | ||
==== Friday Afternoon ==== | |||
D_fens, svan & drag0n go down to the server room and start tracing through the rest of the logs. The rest of us can't do much with no access. | |||
Dano sends out a mail to the DCU RedBrick list | |||
From: Damien Rathigan | |||
Date: Fri, 25 Apr 2008 14:54:58 +0100 | |||
To: redbricksoc@list.dcu.ie | |||
Subject: [Redbricksoc] downtime | |||
hey all its me your beloved secretary, | |||
There will be down time for unplanned maintenance of the network. We'll keep | |||
you up to date with a dramatic blow by blow dialog to let you all know whats | |||
happening! We're working on it and hopefully will not take too long! In the | |||
meanwhile stretch out your legs get some coffee, go for a walk, smell a flower | |||
or something,get high on life! And we'll be in touch! | |||
Dano | |||
Secretary@redbrick.dcu.ie | |||
==== Friday Evening ==== | |||
The admins meet up in DCU about 5/6pm. Atlas also comes down. There is much drawing on a whiteboard about what our plans should be. Most of the night is spent taking backups. | |||
==== Saturday ==== | |||
We meet up again in DCU early on Saturday and start the re-install process. One of the red dells is setup to allow admins external access to the network so stuff can be done remotely. | |||
Minerva is reinstalled. Murphy continues to be shit. | |||
Cynic is re-installed with solaris, and DNS moves there. | |||
==== Sunday ==== | |||
Minerva is finished off on Sunday. We also started scripting people's web directories back to normal. Temporary ldap is setup on one of the red Dells. | |||
I attempt to send out an email to everyone with more information, but the web interface for mailman is down, and the thing refuses to let me authorise the mail without it. In the end I write a script to grab addresses from an ldif and just mail everyone. | |||
From: root | |||
Date: Mon, 28 Apr 2008 01:41:35+0100 | |||
Subject: [Webgroup] RedBrick Network Downtime | |||
Hi, | |||
By now most of you will be aware that we have been forced to take the entire | |||
Redbrick network offline on Friday following a compromise in security. | |||
There has been no loss of user data, email, websites, pr0n etc. - it is all | |||
safely backed up. However, we are sure that the root access was gained on | |||
both murphy and minerva as well ldap, so we are re-installing all systems | |||
cleanly before they are going back online. | |||
At present, the only service active is email - email which is autoforwarded | |||
will be operating as normal, other email will be waiting when you login | |||
again. Mailing lists are operational also, but the web interface is offline. | |||
I am hopeful that we will be able to offer basic login services and chat | |||
by Tuesday evening, but we can't make any guarantees - we are working as | |||
fast as possible to get services back online safely. Progress with murphy | |||
has been very slow, and website and database access will take longer to get | |||
up again. | |||
As ldap has been accessed by the intruders, all users will be issued a new | |||
password in the near future. Please keep this password safe, you will | |||
need it to log back in. | |||
I would like to stress again that all your data and mail is safe and waiting | |||
for you. If you need urgent file access or mail forwarding drop me a reply | |||
and I'll do whatever I can. | |||
We'll be in touch shortly when we have more online. We will also be keeping | |||
the website error page on www.redbrick.dcu.ie updated with current info. | |||
Thanks to everyone who sent us advice, encouragement, and steak. It is | |||
much appreciated. | |||
All the best, | |||
andrew. | |||
==== Monday ==== | |||
Halenger builds ircd on minerva. Ubuntu starts working on carbon on Monday evening, after we discover that the ubuntu-server kernel hates the disk controller, and get a different kernel. | |||
The new passwords went out on Monday. This required some epic hack scripting, since useradm was on one of the shite reddells, but that couldn't send email. | |||
==== Tuesday ==== | |||
Phaxx gets IMAP working on minerva on Tuesday afternoon. | |||
Murphy still refuses to work | |||
{0} ok WORK DAMN YOU | |||
WORK ? | |||
At the end of Tuesday the network looked like: | |||
Primary Login: minerva is ready | |||
Primary Services: carbon is installed. Needs work! | |||
Hey: Installed | |||
IRC: Done | |||
LDAP-secondary: Done | |||
WWW-temp: Done | |||
New ldap passwords: Done | |||
Reset changes to webtree: Done | |||
DNS-secondary: Working on cynic | |||
At this point, Deathray hadn't been touched yet, since mail needed to be migrated before that could happen, and Murphy was generally being a cunt, and wasn't working at all. | |||
==== Saturday ==== | |||
The system comes back at lunchtime on Saturday. | |||
From: Andrew Harford | |||
Sent: Saturday, May 3, 2008 3:02:11 PM | |||
Subject: RedBrick Online | |||
Hi, | |||
We are pleased to announce that (parts of) the network are now back online :) | |||
Minerva has been completly re-installed, and should be operating (relatively) | |||
normally, with chat, boards and mail all working. | |||
Carbon has also been re-installed, and it excepting logins, but there's | |||
no configuration there, so don't expect it to do much. Mail, mailing | |||
lists & boards are currently running from carbon. | |||
Murphy was finally re-installed on Thursday, and we will begin work on | |||
restoring web services and databases tomorrow, we hope to have them | |||
operational shortly. | |||
Severus & Obelisk haven't been re-installed yet, so at present there is | |||
no nightly backups, or access to anyterm.redbrick.dcu.ie | |||
Other services such as bitlbee, jakarta, ircproxy etc. will be coming | |||
back after deathray has been re-installed. MUD is also offline. | |||
All users should have received a new password to login, if you haven't please | |||
contact admins@redbrick.dcu.ie | |||
Despite our best efforts, we expect there to be many bugs & things missing. | |||
If you find something wrong, please mail admins@redbrick.dcu.ie | |||
All the best, | |||
andrew. | |||
.. | |||