We appear to have lost external network again. I am contacting NYI about it now.
Update (by brong at Thu Jan 26 23:01 UTC)
We are back online again.
NYI assure me that they have a good idea what the root cause was, and will update me shortly.
We appear to have lost external network again. I am contacting NYI about it now.
Update (by brong at Thu Jan 26 23:01 UTC)
We are back online again.
NYI assure me that they have a good idea what the root cause was, and will update me shortly.
Status (by brong at Thu Jan 26 20:28 UTC)
Our external network in New York is experiencing high packet loss – we’re not sure if the cause is an attack or just an equipment fault. We’re working with the techs in New York now.
All users will be experiencing slow connections or timeout errors.
Update (by brong at Thu Jan 26 20:54 UTC)
There was a processor spike on our uplink switch. NYI are investigating the root cause. Meanwhile, they have manually reset it, and everything is working again.
Some email may be slightly delayed. New emails are arriving immediately. Everything else is online now.
Status (by brong at Wed Jan 18 10:25 UTC)
One IMAP server has crashed – some users are offline. Restarting now.
Update (by brong at Wed Jan 18 10:35 UTC)
Everything is running normally again now.
Status (by brong at Sun Jan 15 12:08 UTC)
We’ve just had an approx 10 minute outage of the entire site when a machine went crazy and flooded the internal network after its hard disks both failed. We remote killed the machine, and everything is working again now.
Status (by brong at Mon Jan 9 17:51 UTC)
We just had a short IMAP/POP outage for all users – all fixed now.
Status (by brong at Mon Jan 9 08:35 UTC)
A test failure has shut down all IMAP servers. We are investigating the cause.
Update (by brong at Mon Jan 9 08:44 UTC)
Everything is running again – it was caused by our internal testing user hitting the new rate-limits!
Status (by robm at Sat Dec 31 22:01 UTC)
Users may have experienced a 5-15 minute outage. This was due to an imap server failure and some cascade effects.
All services have been restored and should be running normally again.
Status (by brong at Wed Dec 21 17:06 UTC)
One IMAP server is down – restarting it now
Update (by brong at Wed Dec 21 17:12 UTC)
Everything is running normally again now
Status (by brong at Tue Dec 20 17:32 UTC)
We have had an outage for some users, where a monitoring tool which was out of date made it appear that no mailboxes existed temporarily. This is now being fixed.
Update (by brong at Tue Dec 20 17:51 UTC)
This might be worse than first thought – for safety I have disabled all services on the affected machine while I investigate. Again, this is isolated to one machine, so many users are not affected.
Update (by brong at Tue Dec 20 18:44 UTC)
All users are fully operational again now
Update (by brong at Tue Dec 20 18:45 UTC)
I should probably also clarify that only 27 users were affected, and we don’t believe any email was lost, though some users may have seen confusing messages along the way! It may take a little time for all email to arrive as the queues clear now – on the order of 5-15 minutes.
Status (by brong at Tue Dec 20 11:30 UTC)
One of our imap servers has rebooted itself without warning. Users on that server will experience a short outage as we restart services.
Update (by brong at Tue Dec 20 11:40 UTC)
The server is back up running correctly – everybody’s email should be accessible again.
