We’ve been hit with another mail flood, twice in 2 weeks. It’s causing some mail queues to build up and for email to be delayed. We’re working on clearing the queues but it may take a few hours at this stage
Update: We’ve now tracked down the main cause of the flooding that was causing queues to build. The queues are clearing, but it will take a while. It’s also causing high load on some of the backend servers as the backlog of mail is delivered.
We had to restart imap processes on one server because a runaway program ate up all resources and nobody could log in until everything was restarted.
Estimated downtime – back up again already!
Some users may not slower than usual performance during the day today. Unfortunately this is due to a planned moving of data over the weekend taking longer than expected. We hope to complete the move of data during the next 24 hours or so.
It seems the kernel we upgraded some of our servers to has some different behaviour under load in production compared to the old kernel and what we saw in our testing. This is causing some mail to back up in the queue and some slower performance on some of the IMAP servers. We’ve downgraded to the previous kernel on the mail processing servers to help clear the queue which may take a couple of hours. We will return the IMAP servers to the previous version tonight or on the weekend until we’ve worked out what’s causing the problems
We have discovered we need to apply an urgent patch to the drivers on some of our machines which requires a reboot. There will be an approximately 5 minute outage for users on each server as we move through them, starting now. I will update this entry when all servers are done.
mostly done – but I stopped when I realised that a bug in our server status detection meant the web interface was freezing up! Will do the rest once that’s fixed.
imap5 server is down. We’re investigating.
Update: imap5 is back again, all services are up. Will take a little while to flush and deliver all email
