Logins/actions failing for some users for the last hour

Status (by robm at Mon Apr 14 03:13 UTC)
Due to an error while rolling out a change, some web servers were incorrectly rejecting all actions/logins for the last hour. This would have affected about 15% of users trying to use the web interface. This has now been fixed.

Posted in Status. Comments Off

FastMail changelog update

The following changes have been rolled out to production:

  • New keyboard shortcuts in the rich text editor:
    * Ctrl-Shift-8 – make bullet list
    * Ctrl-Shift-9 – make numbered list
    * Ctrl-[ - decrease quote level
    * Ctrl-] – increase quote level
  • New! Use multiple levels of bullet points and numbers to structure your message. Just hit ‘tab’ on a blank list item to increase the indentation, or ‘enter’ on a blank list item to decrease the indentation.
Posted in Status. Comments Off

SSL connections failing for some users

Status (by brong at Thu Apr 10 04:52 UTC)
A failed update has caused missing intermediate certificates on our servers.

We’re working on a fix now.

Update (by brong at Thu Apr 10 04:59 UTC)
All servers are now updated and serving the intermediate certificates correctly again

Posted in Status. Comments Off

One server down

Status (by brong at Mon Mar 31 22:03 UTC)
One IMAP server is having difficulty. Some users will see slow responses or an outage.

Update (by brong at Mon Mar 31 22:14 UTC)
Update: everything is running fine again

Posted in Status. Comments Off

XMPP server unreliability

Status (by robn at Fri Mar 14 04:59 UTC)
The XMPP server has been crashing and restarting every few minutes for a few hours now. The problem has been traced to unusual data received from a client. A workaround has been implemented while we work to understand and fix the problem properly.

Posted in Status. Comments Off

FastMail unavailable

It looks like there’s some network problems at our primary data centre in New York, rendering FastMail unavailable to most users. We’re investigating further.

Update 13:34 AEDT:

The problem was caused by a denial-of-service attack against our network in New York. The data centre staff have implemented workarounds to get us back online and we’re working with them to restore services properly.

Posted in Status. Comments Off

One server running slowly

Status (by brong at Fri Feb 28 22:16 UTC)
After the server failure a couple of days ago, we are re-replicating data to bring every server back to fully replicated status. Unfortunately, this has caused high load on one server, which is running slowly. We are slowing down the replication rate to improve performance again.

Posted in Status. Comments Off

One backend mail server down

Status (by robn at Wed Feb 26 17:58 UTC)
One backend (IMAP) server has crashed. Services are being moved to alternate machines. Everything will be back to normal in a few minutes.

Update (by robn at Wed Feb 26 22:10 UTC)
It seems that the failover did not work correctly, despite all monitoring reporting that all is well, so mail service has been unavailable for a some users for three hours now. We’re working to restore the failed server and should have something to report soon.

Update (by robn at Wed Feb 26 23:37 UTC)
We’ve managed to restore the dead server (looks like a failed disk). The missing users/mail has been traced to a bug that caused replication to be backed up for several days, in a way that was not properly detected by our monitoring tools. We’re replaying the replica now and things are slowly catching up. We don’t have an ETA on when this will be finished, but hopefully won’t take too long. I’ll post here again when its done.

Posted in Status. Comments Off

Potentially more email than normal marked as spam the last few days

Status (by robm at Sun Feb 23 23:18 UTC)
So after accidentally rolling out a rule in error last week (http://status.fastmail.fm/2014/02/20/increased-spam-due-to-spamassassin-error/), that caused some very likely spam to have a low score, the maintainers of SpamAssassin made another change to the rule scoring the other day (http://www.gossamer-threads.com/lists/spamassassin/users/183590#183590).

Unfortunately this change was incompatible with the previous change we made to fix their error. This caused messages that were deemed to be very likely to be spam by the bayes db to have a very, very high score, even in cases where sites have a whitelist score that would normally bring them under the spam threshold.

The net result, it’s possible that some people will see quite a bit more email incorrectly marked as spam in the Spam/Junk Mail folders over the weekend. We recommend you check this folder and "Report non-spam" on these items to move them back to your Inbox.

We’ve reverted our change now, which should fix this issue, but hopefully the SpamAssasin maintainers are a bit more careful in the future.

Posted in Status. Comments Off

Increased spam due to SpamAssassin error

Status (by robm at Thu Feb 20 03:02 UTC)
A recent automatic update to SpamAssassin rules contained an error that caused a significant increase in the amount of spam getting through for a number of users (details in this mailing list post: <http://www.gossamer-threads.com/lists/spamassassin/users/183404>)

A fix for this has been rolled out now which should bring things back to normal again. We apologise for the inconvenience.

Posted in Status. Comments Off

Get every new post delivered to your Inbox.

Join 3,975 other followers