SAN issue affecting several VMs

A hardware issue is affecting several Virtual Servers currently. We are working to restore full service for you as soon as possible.

Please accept our apologies for any inconvenience.

Systems Operations

Update 07:20 15/11/16:

The issue is unfortunately due to one member of the SAN (Storage Area Network) suffering drive failures. We are currently reconstructing the array and are doing our best to restore normal service as soon as possible. We apologise again for any inconvenience.

Systems Operations

Update 11:22 15/11/16

This issue is ongoing and we have not yet been able to recover the SAN data.  We are working with our vendors to try to find a solution.

Update 16:46 15/11/16

We have managed to reconstruct what appears to be a working RAID set.  A rebuild is presently in progress.  If this succeeds, we should be able to start bringing volumes online in approximately 3 hours.  If it fails, we will need to look at other options.

Update 21:57 15/11/16

The rebuild completed successfully.  We need to carry out one more drive swap before we attempt to bring the volumes online.  This is presently at 25%.

Update 01:30 16/11/16

This process completed but we have detected one further drive with read errors and are currently replacing this.  We will then attempt to bring the data online and start servers.

Update 07:27 16/11/16

The volumes are now back online.  We are now gradually booting VMs but please note filesystem checks will be needed on most servers so there will be some delay in starting them.

Update 18:33 16/11/16

All VMs should be online.  If your VM is not working please contact support at helpdesk@daily.co.uk.