Status (by robm at Thu Sep 27 23:27 UTC)
A change just rolled out unfortunately broke almost all web login attempts. Fortunately our regular tests picked this up very quickly, and we rolled back the change. Now to work out why it worked in testing but not in production
Status (by robm at Mon Sep 24 13:29 UTC)
We had a few problems with slowness on one of our web servers affecting some users. We’ve found the problem and services should be running normally again
Status (by robm at Mon Sep 17 08:33 UTC)
One of our IMAP servers is currently acting very slow. We’re trying to work out what’s causing it to be overloaded and hope to resolve it soon. This should only be affecting a few users as there are currently not many users on this server
Update (by robm at Mon Sep 17 08:48 UTC)
We’ve identified what was causing the server to become overloaded and stopped the process causing the problem. Response times should return to normal soon
Ok, this one was entirely my fault! I have a script which is automatically moving users around to balance the load across all our servers evenly. I managed to get one of our “pingusers” into the move list, and the automated move software promptly moved it to another server. The automated tests on that server then failed and shut down the offending service. Oops.
I’ve moved the user back and cleaned up the moves database so no other pingusers are accidentally moved. My apologies to all those affected by the brief outage.
Bron.
