FastMail.FM Status

Status (by brong at Tue Nov 29 19:10 UTC)
A bug fix to a very old bug broke some databases.

We’re reverting the bug fix now, and restarting all servers.

Technical details for those who care: it was due to signed ‘char’ comparisons on messageids and 8 bit characters being present in some messageids, which they shouldn’t

Update (by robm at Tue Nov 29 22:20 UTC)
The visible effect of this bug for users would have been:

1. Some messages would have been delivered multiple times
2. Pop Links would have been reporting delivery failures

The bug has been fixed.

Status (by brong at Fri Nov 18 21:04 UTC)
We’re having a short outage on our IMAP servers caused by a configuration bug which needs an immediate restart.

Update (by brong at Fri Nov 18 21:21 UTC)
All servers are now back online

Status (by brong at Thu Nov 10 13:32 UTC)
There have been some IMAP connections failing with "Server Error" messages due to a connection limit being reached on one of our frontends – just for the past 10 minutes or so. This is now fixed.

Status (by robm at Tue Nov 8 02:32 UTC)
We’ve had two short outages for different sets of users in the last 24 hours.

Both of these are due to kernel crashes on an imap server, though different servers. Interestingly both servers had an uptime that was similar, so we’re wondering if there’s some "uptime" bug that both managed to trigger. We’ll check other servers and reboot after failover as needed.

Status (by brong at Mon Nov 7 18:48 UTC)
One of our IMAP servers is down, so some users are offline. It should be back within 10 minutes

Update (by brong at Mon Nov 7 19:15 UTC)
The IMAP server has been back up for a bit now, and everything looks stable.

Status (by brong at Mon Nov 7 11:21 UTC)
There have been a couple of very short outages (a few seconds to about a minute) caused by updates to our frontend management software – they should be finished soon.

Status (by brong at Sun Nov 6 17:09 UTC)
I’ve just been paged by a whole lot of things. Investigating.

Update (by brong at Sun Nov 6 17:18 UTC)
Oh good – only one spam scanning machine down. You may have noticed a 10 minute delay in some incoming emails. They should come through in the next minute or so as the queues clear.

Status (by brong at Thu Nov 3 16:02 UTC)
It appears that the entire site has been offline for some users for a few hours due to incorrect caching of IP addresses in one of our switches.

The worst part is, none of our notification systems could tell, so we didn’t know until we saw users reports. My apologies for the outage – it was me working on a new "reliability" system which caused the IP address to be moved. So much for reliability systems…

Bron.