IMAP server issues – all users

Status (by brong at Tue Nov 29 19:10 UTC)
A bug fix to a very old bug broke some databases.

We’re reverting the bug fix now, and restarting all servers.

Technical details for those who care: it was due to signed ‘char’ comparisons on messageids and 8 bit characters being present in some messageids, which they shouldn’t

Update (by robm at Tue Nov 29 22:20 UTC)
The visible effect of this bug for users would have been:

  1. Some messages would have been delivered multiple times
  2. Pop Links would have been reporting delivery failures

The bug has been fixed.

Posted in Status. Comments Off

Short outage on IMAP servesr

Status (by brong at Fri Nov 18 21:04 UTC)
We’re having a short outage on our IMAP servers caused by a configuration bug which needs an immediate restart.

Update (by brong at Fri Nov 18 21:21 UTC)
All servers are now back online

Posted in Status. Comments Off

IMAP connection failures

Status (by brong at Thu Nov 10 13:32 UTC)
There have been some IMAP connections failing with "Server Error" messages due to a connection limit being reached on one of our frontends – just for the past 10 minutes or so. This is now fixed.

Posted in Status. Comments Off

Short outage

Status (by robm at Tue Nov 8 02:32 UTC)
We’ve had two short outages for different sets of users in the last 24 hours.

Both of these are due to kernel crashes on an imap server, though different servers. Interestingly both servers had an uptime that was similar, so we’re wondering if there’s some "uptime" bug that both managed to trigger. We’ll check other servers and reboot after failover as needed.

Posted in Status. Comments Off

One server down

Status (by brong at Mon Nov 7 18:48 UTC)
One of our IMAP servers is down, so some users are offline. It should be back within 10 minutes

Update (by brong at Mon Nov 7 19:15 UTC)
The IMAP server has been back up for a bit now, and everything looks stable.

Posted in Status. Comments Off

Short outages

Status (by brong at Mon Nov 7 11:21 UTC)
There have been a couple of very short outages (a few seconds to about a minute) caused by updates to our frontend management software – they should be finished soon.

Posted in Status. Comments Off

System Outage

Status (by brong at Sun Nov 6 17:09 UTC)
I’ve just been paged by a whole lot of things. Investigating.

Update (by brong at Sun Nov 6 17:18 UTC)
Oh good – only one spam scanning machine down. You may have noticed a 10 minute delay in some incoming emails. They should come through in the next minute or so as the queues clear.

Posted in Status. Comments Off

Outage for some users

Status (by brong at Thu Nov 3 16:02 UTC)
It appears that the entire site has been offline for some users for a few hours due to incorrect caching of IP addresses in one of our switches.

The worst part is, none of our notification systems could tell, so we didn’t know until we saw users reports. My apologies for the outage – it was me working on a new "reliability" system which caused the IP address to be moved. So much for reliability systems…

Bron.

Posted in Status. Comments Off
Follow

Get every new post delivered to your Inbox.

Join 3,975 other followers