server3 status update

One of the filesystems on one of our servers (server3) failed last night due to a filesystem bug. After a bit of work we were able to bring up the users on other partitions on server3. However, mail delivery was suspended while we were doing this and so there was a period (most of the US day) where incoming mail was being delayed by around an hour. The queues have all emptied back out now, and delivery should be happening normally.

Incoming mail for users on the affected partition is still being queued, but these users are currently unable to log in via the web interface or IMAP.

We’re currently running a filesystem check on the affected disk. Unfortunately, there is no faster option to get that set of mailboxes back online. We apologise for the inconvenience and are working hard to restore service.

I’ll post more here as it becomes available.

Update: 02:00 on the 2nd of September GMT

As of now, I estimate between 17 and 24 hours remain before the filesystem check is complete.

Update: 08:30, 2nd of September GMT

My latest estimate is somewhere between 10 and 14 hours remaining on the disk check

server3 has been down for about 1/2 hour

Server3 died about 1/2 hour ago. I am looking at it now.

Bron.

UPDATE: it was a filesystem bug, most frustrating. I needed to reboot the server, which takes about 8 minutes – so probably another 20 minutes down all together while I check and start everything

UPDATE 2: seems that the filesystem will need a full check, which will take a couple of hours.

Follow

Get every new post delivered to your Inbox.

Join 2,294 other followers