Honey! I just DELETED LinkPeek.com

During the day I am an ops sys-admin. During the night I am a husband, father of two, and a CEO of a bootstrapped start-up. After launch, my first project was to schedule regular backups of user data and archive off-site. My goal was to create backups but never need them. Boy was I lucky …

Yes, leave it to me to inadvertently delete the VPS root disk. One of the major cloud providers places the “rename” and “remove” disk buttons right next to each other and I learned a nasty habit of clearing pop-ups without reading them (thanks Windows).

“Honey! I just deleted LinkPeek.com”

The horror… My stomach felt like I took a tumble in a roller-coaster. Instantly I tossed off my developer hat and put on my operations hat. I checked the off-site backups. I had nightly dumps of MongoDB and weekly tar backups of the /etc partition. The user data was in MongoDB and most of the system configuration information was in the tar. I used the tar to recover 2 upstart scripts, 2 supervisord scripts, 2 complex nginx confs, an ssl cert, and the pyramid production.ini.

I set out to stand up a new server, re-install the needed packages, recover the user data, and restore the service. After 1.5 hours of feverishly typing LinkPeek was back online.

What I learned and my plan going forward

If you are a small team or a start-up, you must have somebody dedicated to operations. Without backups I would not have been able to gracefully recover. Most likely I would have reimbursed the existing members and shuttered the doors.

This experience was eye-opening. In my next couple of posts I will explain how I create and maintain backups and my next project will implement a configuration management and provisioning system.

This system will allow me to:

  1. take out the human element of recovery
  2. significantly reduce the time-to-recover from a catastrophic failure
  3. test disaster recovery procedures before needing them
  4. provision development and production environments without effort
  5. have a reproducible blueprint of “how to build a LinkPeek server”

Leave a Reply

Your email address will not be published. Required fields are marked *