The System Crashed Hard

 

The Disaster 

A server containing multiple disks managed by a vendor for the State of Hawaii’s driver's licensing programs had multiple hard disks crash. The storage media is encrypted and secured with some data not readable. There was no security or data breach. It will take a couple of months to determine if the data can be recovered.

The Vendor

The spokesperson gives a briefing at a press conference similar to "the dog ate my homework".

1) There was a backup system in place that's supposed to protect the data when the hard drives crash, it was not properly configured.

2) They were not aware that certain documents or images were not getting backed up properly.

3) The backups are checked to make sure everything is working properly. Don't  know what the details are and what the plan was on checking the data.

Summary

When implementing a new system, I have few suggestions on how to make your system bulletproof. I used to be a Systems Administrator and I never lost any data on my watch.

1) The Project Manager in charge needs to have good Subject Matter Experts to recommend the technology, security, and procedures to be in place to ensure the system has the redundancy to withstand a disaster and prevent a security breach.

2) The Statement of Work needs to always include clear expectations.

3) Service Level Agreements need to be in place for the system.

4) Disaster Avoidance plan and Recovery Strategies to meet Service Level Agreements.

5) Test your systems periodically to ensure that the data is being replicated or backed up properly to a media that will go offsite.

6) Make sure you have a good hardware / software support contract in place.

7) Never ever trust the vendor. Make a checklist of your key deliverables to be reviewed and demonstrated when you begin the sign-off stage in the project. Any missing checks will result in a big missing check for that vendor to cash.