Great Hack of 2008

From RedBrick Wiki
Jump to: navigation, search

Hax.png


;_;


Redbrick died a death 25 April 2008, following the immortal words of lil_cain only a day before, on 24 April:
<lil_cain>: Thankfully in my time there have been no root exploits

Timeline of Events

In an attempt to procrastinate from real work I've tried to document what happened on the week of no sleep, and insane amounts of coke.

To give a bit of background, the hack happened just a few weeks after myself and johan were elected, lil_cain was the third admin, and werdz had recently moved from admin to webmaster. All of us were working full time on Intra, except for cian, who had a real job. Murphy was the main web server, running Ubuntu 6.06. Minerva was the main login server, also running Ubuntu.

Mostly, I've pieced this together from memory, email and irc logs. Some other people who were there may be able to fill in more of the details.

--receive (May 2009)


Friday Morning

  • Around 8.30am (ish) - People start to notice something is up. Most people are on the way to work and stuff. Pretty sure I got a text on the bus, can't remember who from.
  • 09:19 - people have posted to redbrick.computing.www about the problem.
  • 09:27 - people in #lobby start to notice.
 09:27      cambo anyone know what teh deal with the website is ?
 09:27     marvin singer is afk
 09:27     marvin tbh
 09:28      fatwa cambo: haxed.  lil_cain quit his admin position, that's his 'goodbye' revenge
 09:28    tbolger entire site hax0rd ?
 09:28      cambo fatwa: srsly... 
 09:28      fatwa tbolger: check www.redbrick.dcu.ie/~tbolger
 09:28      fatwa cambo: ya rly
 09:28      fatwa tbolger: it only seems to affect home directorys
  • 09:31 - ssh access to murphy is shut down.
  • 09:40 - apache is shutdown on murphy.
  • 09:42 - access is traced to the hurling club account, which is disusered.
  • 09:46 - checks on other servers reveal that auth.log is missing from minerva.
  • 10:00 - the decision is made to shutdown all access
 10:01            receive changed the topic of #lobby to: RedBrick access is about to be shut off, and will remain off until at least late tonight. 
  • 10:30 - final mass hey goes out before shutdown
  • 10:32 - access to the network is cut off. A few root holders take refuge in #blueblock on irc.linux.ie to discuss meeting later on.


Friday Afternoon

D_fens, svan & drag0n go down to the server room and start tracing through the rest of the logs. The rest of us can't do much with no access.

Dano sends out a mail to the DCU RedBrick list

 From: Damien Rathigan                                                                                  
 Date: Fri, 25 Apr 2008 14:54:58 +0100                                                                               
 To: redbricksoc@list.dcu.ie                                                                          
 Subject: [Redbricksoc] downtime                                                                                     
                                                                                                                   
 hey all its me your beloved secretary,                                                                              
                                                                                                                   
 There will be down time for unplanned maintenance of the network. We'll keep                                        
 you up to date with a dramatic blow by blow dialog to let you all know whats                                        
 happening! We're working on it and hopefully will not take too long! In the                                         
 meanwhile stretch out your legs get some coffee, go for a walk, smell a flower                                      
 or something,get high on life! And we'll be in touch!                                                               
                                                                                                                   
 Dano                                                                                                                
 Secretary@redbrick.dcu.ie


Friday Evening

The admins meet up in DCU about 5/6pm. Atlas also comes down. There is much drawing on a whiteboard about what our plans should be. Most of the night is spent taking backups. Getting some sort of mail processing in place is a priority, as we're not sure how long HEAnet/DCU will store our incoming mail for before giving up. Minerva's OS (and feck all else) is wiped and reinstalled by around midnight (after some extremely odd keyboard controller-related difficulties), and by the time we leave at around 1am, Deathray (not quite reinstalled, but isolated as much as possible from everything) is routing mail to inboxes.


Saturday

We meet up again in DCU early on Saturday and start the re-install process. One of the red dells is setup to allow admins external access to the network so stuff can be done remotely.

Murphy continues to be shit.

Cynic is re-installed with solaris, and DNS moves there.


Sunday

Minerva is finished off on Sunday. We also started scripting people's web directories back to normal. Temporary ldap is setup on one of the red Dells.

I attempt to send out an email to everyone with more information, but the web interface for mailman is down, and the thing refuses to let me authorise the mail without it. In the end I write a script to grab addresses from an ldif and just mail everyone.

 From: root 
 Date: Mon, 28 Apr 2008 01:41:35+0100           
 Subject: [Webgroup] RedBrick Network Downtime                                                                       
                                                                                                                   
 Hi,                                                                                                                 
                                                                                                                   
 By now most of you will be aware that we have been forced to take the entire                                        
 Redbrick network offline on Friday following a compromise in security.                                              
                                                                                                                   
 There has been no loss of user data, email, websites, pr0n etc. - it is all                                         
 safely backed up. However, we are sure that the root access was gained on                                           
 both murphy and minerva as well ldap, so we are re-installing all systems                                           
 cleanly before they are going back online.                                                                          
                                                                                                                   
 At present, the only service active is email - email which is autoforwarded                                         
 will be operating as normal, other email will be waiting when you login                                             
 again. Mailing lists are operational also, but the web interface is offline.                                        
                                                                                                                   
 I am hopeful that we will be able to offer basic login services and chat                                            
 by Tuesday evening, but we can't make any guarantees - we are working as                                            
 fast as possible to get services back online safely. Progress with murphy                                           
 has been very slow, and website and database access will take longer to get      
 up again.                                                                                                           
                                                                                                                   
 As ldap has been accessed by the intruders, all users will be issued a new                                          
 password in the near future. Please keep this password safe, you will                                               
 need it to log back in.                                                                                             
                                                                                                                   
 I would like to stress again that all your data and mail is safe and waiting                                        
 for you. If you need urgent file access or mail forwarding drop me a reply                                          
 and I'll do whatever I can.                                                                                         
                                                                                                                     
 We'll be in touch shortly when we have more online. We will also be keeping                                         
 the website error page on www.redbrick.dcu.ie updated with current info.                                            
                                                                                                                   
 Thanks to everyone who sent us advice, encouragement, and steak. It is                                              
 much appreciated.                                                                                                   
                                                                                                                   
 All the best,                                                                                                       
 andrew.


Monday

Halenger builds ircd on minerva. Ubuntu starts working on carbon on Monday evening, after we discover that the ubuntu-server kernel hates the disk controller, and manually install a different kernel using a rescue shell.

The new passwords went out on Monday. This required some epic hack scripting, since useradm was on one of the shite reddells, but that couldn't send email.

Tuesday

Phaxx gets IMAP working on minerva on Tuesday afternoon.

I fell asleep on the bus back from work, and woke up at Ballymun shopping center.

Murphy still refuses to work. The Ubuntu installer CD refuses to talk to the upgraded version of the T2000 firmware (which we were told to upgrade by the installation instructions).

 {0} ok WORK DAMN YOU
 WORK ?


At the end of Tuesday the network looked like:

 Primary Login: minerva is ready                                                                                     
 Primary Services: carbon is installed. Needs work!                                                                  
 Hey: Installed                                                                                                      
 IRC: Done                                                                                                           
 LDAP-secondary: Done                                                                                                
 WWW-temp: Done                                                                                                      
 New ldap passwords: Done                                                                                            
 Reset changes to webtree: Done                                                                                      
 DNS-secondary: Working on cynic           

At this point, Deathray hadn't been touched yet, since mail needed to be migrated before that could happen, and Murphy was generally being a cunt, and wasn't working at all.


Wednesday

I make some progress beating boards into working over the afternoon, it finally starts working properly about 7pm

Phaxx & atlas come to DCU to help make exim and mailman work. We also eat chinese.

I (werdz) think it was around now (possibly Thursday or Friday) that I got ubuntu installed on murphy. Manually. After downgrading the firmware to version Poxy.ancient. The installer didn't seem to want to work at all (can't remember the exact list of problems), so it was installed using a rescue shell to create partitions, dpkg-bootstrap the system and install frivolous things like the kernel manually. Once this was done, Ubuntu booted, and things generally worked, but the serial console (well... what you got by typing console -f into the ALOM) randomly locked up. And some other things randomly broke. We decided this level of reliability wasn't what we were looking for in an OS. See Murphy for the rest of this epic tale, and how it eventually ended up with Solaris 10, a full year later.

Thursday

Svan gets fap working, and and syslog-ng doing remote logging.


Friday

Get mailman working about 1am. Continue testing mail and stuff until about 2/3am


Saturday

Go down to dcu about 10am to put all the cables back to normal, and let users login again. Security finally let us in at about 12.30

SSH access to minerva is allowed again about 8pm, and the mail goes out shortly after.

 From: Andrew Harford
 Sent: Saturday, May 3, 2008 23:10:06 +0100 
 Subject: RedBrick Online                                                                                            
                                                                                                                   
 Hi,                                                                                                                 
                                                                                                                   
 We are pleased to announce that (parts of) the network are now back online :)                                       
                                                                                                                   
 Minerva has been completly re-installed, and should be operating (relatively)                                       
 normally, with chat, boards and mail all working.                                                                   
                                                                                                                   
 Carbon has also been re-installed, and it excepting logins, but there's                                             
 no configuration there, so don't expect it to do much. Mail, mailing                                                
 lists & boards are currently running from carbon.                                                                   
                                                                                                                   
 Murphy was finally re-installed on Thursday, and we will begin work on                                              
 restoring web services and databases tomorrow, we hope to have them                                                 
 operational shortly.                                                                                                
                                                                                                                   
 Severus & Obelisk haven't been re-installed yet, so at present there is                                             
 no nightly backups, or access to anyterm.redbrick.dcu.ie                                                            
                                                                                                                   
 Other services such as bitlbee, jakarta, ircproxy etc. will be coming    
 back after deathray has been re-installed. MUD is also offline.                                                     
                                                                                                                   
 All users should have received a new password to login, if you haven't please                                       
 contact admins@redbrick.dcu.ie                                                                                      
                                                                                                                   
 Despite our best efforts, we expect there to be many bugs & things missing.                                         
 If you find something wrong, please mail admins@redbrick.dcu.ie                                                     
                                                                                                                   
 All the best,                                                                                                       
 andrew.        


Sunday

Welmar is backed up, and re-installed, followed by Obelisk. Anyterm starts working again late Sunday night.


Monday

Monday was a bank-holiday, so we got into DCU about midday.

We backed-up the rest of the stuff on Deathray, and re-installed that on Monday afternoon. Bitlbee was also setup late Monday night.


The simplified version

GreatHack.jpg