Status (by robm at Sat Dec 31 22:01 UTC)
Users may have experienced a 5-15 minute outage. This was due to an imap server failure and some cascade effects.
All services have been restored and should be running normally again.
Status (by robm at Sat Dec 31 22:01 UTC)
Users may have experienced a 5-15 minute outage. This was due to an imap server failure and some cascade effects.
All services have been restored and should be running normally again.
Status (by brong at Wed Dec 21 17:06 UTC)
One IMAP server is down – restarting it now
Update (by brong at Wed Dec 21 17:12 UTC)
Everything is running normally again now
Status (by brong at Tue Dec 20 17:32 UTC)
We have had an outage for some users, where a monitoring tool which was out of date made it appear that no mailboxes existed temporarily. This is now being fixed.
Update (by brong at Tue Dec 20 17:51 UTC)
This might be worse than first thought – for safety I have disabled all services on the affected machine while I investigate. Again, this is isolated to one machine, so many users are not affected.
Update (by brong at Tue Dec 20 18:44 UTC)
All users are fully operational again now
Update (by brong at Tue Dec 20 18:45 UTC)
I should probably also clarify that only 27 users were affected, and we don’t believe any email was lost, though some users may have seen confusing messages along the way! It may take a little time for all email to arrive as the queues clear now – on the order of 5-15 minutes.
Status (by brong at Tue Dec 20 11:30 UTC)
One of our imap servers has rebooted itself without warning. Users on that server will experience a short outage as we restart services.
Update (by brong at Tue Dec 20 11:40 UTC)
The server is back up running correctly – everybody’s email should be accessible again.
Status (by brong at Thu Dec 15 12:01 UTC)
We are investigating. This will only affect some users.
Update (by brong at Thu Dec 15 12:11 UTC)
Everything is back up again now – it was a kernel crash on one of the IMAP servers. We suspect it’s due to a fault with old firmware in one of the network cards, and will update it once replication is caught up.
Status (by brong at Thu Dec 1 10:28 UTC)
Some users may have noticed error pages over the past 2-3 minutes as we restarted all IMAP servers.
The issues introduced a couple of days ago needed one final restart to clean everything up. Searches should now be reliable again, and everything else should be much faster if nothing has changed. We have managed to enable some optimisations to the "detect unchanged" case which weren’t possible before.
Update (by brong at Thu Dec 1 12:13 UTC)
It turns out a couple of restarts were required. Each one would have lasted under 30 seconds for each server, but apologies to those who saw them. The restarts should be finished now, and we’re back running normally.
