Power has been restored to all servers. There is a mail queue backlog which should be delivered over the next hour.
Update: One of the backend servers had problems and was restarted again. All servers appear to be functioning normally now. The mail queue is almost empty, only a few more to go.
While we’re waiting to find out why power to the web servers has died, we’re routing all traffic to the beta server so email can still be accessed
All our web server units have just lost power. IMAP servers are still up and working.
One of the backend servers (server 2) is currently very slow/down. We’re investigating.
I’m updating the indexes on the database server to make undelete work in a reasonable time, but it’s freezing all other actions while it happens.
I estimate filestorage change functions will timeout for about the next 10 minutes, including any file uploads.
Bron.
Outgoing email was delayed for up to about 1 hour earlier today. The queue has now been flushed and email is being delivered normally again
There has been a very short outage followed by some slowdowns on all servers.
I estimate approximately 10 minutes before everything settles down again.
Bron.
edit: there was a filestorage issue. Fixed now
There has been an issue which caused web services to go down briefly, and cascading overloads on the backend IMAP servers meant I needed to shut everything down and bring services back one at a time.
The outage will last approximately 10 minutes. I apologise for the inconvenience caused.
Bron.
edit: everything except filestorage should be back up now
edit2: filestorage is back up as well
