server 1 is back online
Full network connectivity has been restored.
There is currently some network problem affecting access to our machines. We’re looking into this.
OK, the outage is complete now. The problem was that the database server did not exit cleanly when we rebooted. It’s never happened before – we’re going to talk to the database developer to make sure we understand what happened and stop it from happening again.
We now have an updated version of the Linux kernel in place which has a fix for the problem that has caused some outages over the last few weeks. The problem was caused by a ‘dead-lock’ where two processes would try to access the same thing at the same time, lock each other out in the process, and get stuck. The cause of the deadlock has been identified by the Linux kernel developers and has been resolved (well, since we can’t replicate the problem, we can’t be 100% sure – but the logic of the bug-fix looks sound, and it passes our stress tests).
We have monitored the database recovery speed and based on the rate so far, completion of the database recovery will take another 90 minutes.
The database has to run a recovery process, which could a couple of hours. There will be an outage during this time. More information will be posted as we have it.
We have to reboot our servers to clear a problem. This will take around 5 minutes. The good news is that we have installed a kernel patch which should resolve the problem in the future – I’ll most more about that after we’re back up.
We have to extend our database file size. This will take around 5 minutes. Sorry about this outage – we will add to our monitoring systems something to track this in the future, so no more outages of this kind will be required.
