Lesson of the day: redundancy

One day my company bough a big multi rack black box that was supposed to provide disk space. It was supposed to be faster, better, stronger. Better than direct attached space they said. Wiz bangier they said. Never to lose data neither!

Well all hardware is subject to the all powerful murphy, including this one. Moving parts failed, bits were lost, chaos ensued. People were sad and over all not having good days because of too much trust in the black box.

Here's the thing. Where I work we have n+2 redunancy almost everywthere, n+1 at the very least. This was one place where this did not exist. Sure we were using raid5, but the data itself was not n+1 in that particular environment, and for that reason bad things were bound to happen.

Moral of the story:

"if anything can go wrong, it will"

Corollary:

"Always keep a spare for the event above"

This is probably the single most valuable lesson I've learned is systems administration. There's a reason why us sysadmins use "Single Point of Failure" as an acronym.

The takehome question is this. What are you using right now that if it failed you'd go nuts? Is there a backup plan if this were the case?