During the day I am an ops sys-admin. During the night I am a husband, father of two, and a CEO of a bootstrapped start-up. After launch, my first project was to schedule regular backups of user data and archive off-site. My goal was to create backups but never need them. Boy was I lucky …
Yes, leave it to me to inadvertently delete the VPS root disk. One of the major cloud providers places the “rename” and “remove” disk buttons right next to each other and I learned a nasty habit of clearing pop-ups without reading them (thanks Windows).
“Honey! I just deleted LinkPeek.com”
The horror… My stomach felt like I took a tumble in a roller-coaster. Instantly I tossed off my developer hat and put on my operations hat. I checked the off-site backups. I had nightly dumps of MongoDB and weekly tar backups of the /etc partition. The user data was in MongoDB and most of the system configuration information was in the tar. I used the tar to recover 2 upstart scripts, 2 supervisord scripts, 2 complex nginx confs, an ssl cert, and the pyramid production.ini.
I set out to stand up a new server, re-install the needed packages, recover the user data, and restore the service. After 1.5 hours of feverishly typing LinkPeek was back online.
What I learned and my plan going forward
If you are a small team or a start-up, you must have somebody dedicated to operations. Without backups I would not have been able to gracefully recover. Most likely I would have reimbursed the existing members and shuttered the doors.
This experience was eye-opening. In my next couple of posts I will explain how I create and maintain backups and my next project will implement a configuration management and provisioning system.
This system will allow me to:
- take out the human element of recovery
- significantly reduce the time-to-recover from a catastrophic failure
- test disaster recovery procedures before needing them
- provision development and production environments without effort
- have a reproducible blueprint of “how to build a LinkPeek server”