Everything back up

All services are now fully restored and all email that was queued during the outage has been delivered.

The reason the outage took so long is that the technicians at NYI had an electrician in and replaced all our power ciruitry for the primary cabinet with much higher rating cables and circuit breakers. This should ensure that we don’t have another power outage like the on which just occured.

Our deepest apologies to all who were affected by this outage. We can only repeat that we are moving towards an infrastructure layout that will mitigate the effect of future outages like this, but we aren’t there yet.

Bron.

Yes, we’re down

For those who know to look here and can read this – we’re totally down. Everything out.

This is so that NYI can put much beefier power supplies to the cabinet containing out DB server. The goal being to make sure this never happens again. They gave us the option of doing it at a different time, but I decided that with all the outages and stuff having trouble coming back up it was better to swallow the pill now and just do it.

Meanwhile, I’ve made myself a coffee in anticipation of the current 2:22am just being the start of it :( :( :(

Am waiting on a call from NYI. See this ongoing forum thread for gory details of everything: http://www.emailaddresses.com/forum/showthread.php?s=&threadid=41899

Bron.

All services down

Our database server has rebooted itself for some reason. We don’t know why yet.

I will update as soon as we’re back up.

edit: everything is back up now. I had to bring services back up in stages to avoid overloading the database as everything tried to connect at once, which is why it took about 20 minutes

Follow

Get every new post delivered to your Inbox.

Join 2,294 other followers