Status (by robm at Wed Jul 8 13:31 UTC)
For a few hours earlier today, we had user reports that all email being sent to Yahoo was being filed in the users Junk Mail folder at Yahoo.
When we heard about this, we switched outgoing IP addresses, and that appears to have fixed the problem.
Unfortunately we’re not sure why this happened in the first place, but are currently trying to contact Yahoo to find out more, and prevent this in the future.
Status (by robm at Mon Jul 6 01:37 UTC)
A few users are reporting that their personal bayes databases have been reset. This only appears to have affected a few users, and we’re investigating to see what’s going on.
Status (by robm at Sun Jul 5 02:44 UTC)
Annoyingly, two web servers crashed about 6 hours ago in a really peculiar way. The failure mode was such that our 2 minute checks didn’t detect the failures, and the frontend routing servers didn’t automatically route to the alternate web servers.
The net result of this is that for the last 6 hours or so, the web interface has been randomly "flaky", working fine for some people, not working for others, and intermittently working for some.
We’ve now got the servers back up, and will be adding new code to our regular two minute checks to handle this new failure mode, so the servers are correctly taken out of service if it happens again.
Update (by robm at Sun Jul 5 03:14 UTC)
The problem only affected web services. All other services (IMAP, POP, LDAP, etc) have been running fine.
As mentioned, we’ll make this a high priority item this week to make sure this doesn’t happen again.
