All queues cleared

All mail queues have cleared and theres no backlog or delayed mail any more

5-10 minute email delay

One of our incoming mail servers experienced some problems, causing some mail delivery to backup by 5-10 minutes. The queues are now flushing and should be back to normal in a little while.

Some services down

Some servers are down. We’re investigating.

Update: All services should be restored now

Email delays

Our servers are currently being hit by a huge number of spam delivery attempts. We’re seeing over 200 a second on some machines coming from 100,000′s of different hosts scattered around the world, probably all compromised zombie machines. This is causing some of the incoming servers to be overloaded and delaying email delivery by 10-15 minutes for some emails.

database server down.

Our main database server has gone down – everything is stopped. Investigating…

Update: IMAP and POP connections are working again. Just need to get the web sessions database fixed. Should be back soon.

Update: Everything should be up and working now. Things might be a bit slow as they catch up with queued emails and requests

imap7 outage

Imap7 has had a failure. I’ve move all the imap stores over to their replica servers (yay replication) and am investigating.

Some users will have noticed a short outage.

Bron.

update: a while later (still no easy timestamp support and I’m too tired to translate… um, 4:11pm UTC) imap7 has been back up for a while, I’ve been checking that everything is OK, and then I re-enabled replication.  Everything is functioning normally now

.FM Domain Name Issues

The company that runs the .fm domain, and hence is responsible for directing the name “fastmail.fm” at us managed to break some of their name servers for a couple of hours such that they returned “NXDOMAIN” (No Such Domain) for requests for fastmail.fm.\
What this means for you

There are three possible issues here that could affect you:

1) many people would have been unable to log in to the website http://www.fastmail.fm/ or even view this status blog – likelihood: high, effect minor.
2) some outbound mail may have been rejecteded because “sender domain doesn’t exist” – likelihood: low, effect minor (you will get a bounce message and can resend the mail)

3) inbound emails for any users with the fastmail.fm domain may have been bounced before reaching our servers – likelihood: high, effect major.

We’re understandably very upset at the administrators of .fm. We spoke to them once on the phone at the start of this issue but have been unable to get through since, which is understandable since this issue affected all of their customers.

We apologise for any lost emails. They never reached our servers so the only way you can find out about them is if the senders resend them.

As far as we can tell, the issue is now completely fixed, though some systems may cache the “does not exist” response for up to an hour yet, and .fm could of course break it again – but we’re sort of stuck with them. We will try to contact them during Californian business hours and convince them to put a better system into place for avoiding this happening.

Restore of missing emails started

The volume check on the original server3 drive has completed. We’ve now started delivering the messages that were missing from the 12 hours window (see here) to users accounts. This will take a number of hours to run. The messages will appear correctly in place (that is, with the correct original delivery date, flags, etc). We’ll update this entry when this process is complete.

Once the deliveries are complete, we’ll also notify all users with more details via email.

Update: The restores have all been completed and everyone has been emailed with details of all messages that were restored

Follow

Get every new post delivered to your Inbox.

Join 3,975 other followers