2 minute network outage

Status (by brong at Wed Mar 16 13:54 UTC)
We just had a 2 minute network outage. Everything should be back up and running again now.

Posted in Status. Comments Off

Resolved: One IMAP server down

Status (by robm at Sat Mar 12 02:05 UTC)
One IMAP server is down. We’re investigating.

Update (by robm at Sat Mar 12 02:31 UTC)
IMAP server is restored, all services are working again

Posted in Status. Comments Off

High load on all servers

Status (by brong at Fri Mar 11 09:25 UTC)
An index upgrade to our IMAP servers is causing considerably higher load than anticipated, which is causing some outage messages from the web servers. Working on bringing it under control now.

Update (by brong at Fri Mar 11 10:08 UTC)
The web interface is working again, but slow. Incoming emails are being delayed temporarily, but will start again soon.

Update (by brong at Fri Mar 11 10:26 UTC)
Everything is back running again – the last emails are being delivered and will be finished within 5 minutes.

Posted in Status. Comments Off

Slow or intermittent access for the past 30 minutes

Status (by brong at Mon Mar 7 14:55 UTC)
Wow, it’s been a bad week. We got caught in a serious DOS that interrupted our internet connectivity briefly. The DOS has now been blocked.

We will keep monitoring, but everything seems to be running smoothly again now.

Posted in Status. Comments Off

Resolved: Some IMAP servers reporting problems

Status (by robm at Mon Mar 7 03:04 UTC)
Two of our IMAP servers are currently reporting errors, causing access problems for some users. We’re investigating.

Update (by robm at Mon Mar 7 03:26 UTC)
We’ve identified the problem, and are working on it. Two IMAP servers are currently down and will be for about 15 more minutes

Update (by robm at Mon Mar 7 03:58 UTC)
Ok, taking a bit longer than expected to restore, but it’s in progress.

One machine should be 15 mins, the other might be more 30 mins

Update (by robm at Mon Mar 7 04:17 UTC)
One server is back up, still one server to go

Update (by robm at Mon Mar 7 05:52 UTC)
The final server is restored and runing again.

Posted in Status. Comments Off

Resolved: Over quota issues for some users

Status (by brong at Wed Mar 2 16:46 UTC)
Many users have been affected by a bug which causes an over-quota reeport. We are looking in to it.

Update (by brong at Wed Mar 2 17:22 UTC)
No new emails are coming in currently – but mail is not lost. It will be queuing on remote systems.

We are fixing all affected accounts. Estimated time is within an hour.

Update (by brong at Wed Mar 2 17:54 UTC)
Emails to non-affected accounts have been re-enabled. Affected accounts are being fixed as fast as the disks will process them. Estimated finishing time is slightly over an hour, but many accounts are fixed already or will be sooner than that.

Update (by brong at Wed Mar 2 18:28 UTC)
In the rush to get the bugfix out, and old bug crept back in – greyed out folders in Thunderbird. This bug is now fixed as well.

Restore update: over half way done, many users are fully fixed now. Some are still in progress.

Update (by brong at Wed Mar 2 19:28 UTC)
All but a small handfull of users are now fixed. There was a yet another bug involving directory names with a lot of digits (i.e. eBay item number) that meant one store didn’t finish. I’m working on that now.

Will update again once that’s fixed, and deepest apologies for this botched rollout. At least we’re back faster than gmail!

Update (by brong at Wed Mar 2 21:30 UTC)
Finally – all users are fully recovered. There should be no more quota issues, and all delayed emails will be delivered eventually (the queue times are at remote sites, so out of our control)

Posted in Status. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 3,975 other followers