My journey to perfect backups

There and backup again.

Over the years, the more digital photos I’ve gotten, the more I’ve begun to worry about about backups. Now with my music and home videos all located on my computer, it’s grown from a worry into a paranoia. Last year when I bought a 1 terabyte drive I combined all by previous hard drives onto it but within the month I was jittery after hearing about other people whose new terabyte drives had died. I realized that I had balanced all my memories and documents on the tip of a single hard drive spindle. That is when I knew that I needed to resolve this once and for all.

Attempt 1- Subversion

Several years ago, fresh out of college I was familiar with the code version control system I’d been using. At that time Dreamhost, my web host, began providing complimentary Subversion (aka SVN) repositories. I setup a SVN repository and decided to check in all my personal documents and photos.

The theory was sound. My documents were being copied to an offsite location and supposedly recoverable. Wrong.

Its was painful and slow. It wasn’t automated so I had to routinely check in changes. (I realize I could have scripted the check in). It also littered .svn directories with junk all over the place. Finally, if I moved files to different directories the file had to be uploaded again.

I’ll admit that I’m pretty green when it comes the the powers of SVN but there is no doubt that I was subverting its purpose.

Attempt 2- Rsync and DeltaCopy

Soon afterwards I became acquainted with rsync. Dreamhost doesn’t let me backup files I can’t redistribute to to my web directory that I’m not allowed to redistribute (my MP3s). Even if I password protected it and blocked access from the world I wasn’t allowed.  So as a solution I setup my home server with rsync and used DeltaCopy on my windows machines to publish to the rsync server.

Sure, later Dreamhost came out with a personal backup option, but 50 gig was too confining and insufficient for my needs.

Vista for some reason wasn’t allowing DeltaCopy access to all my own files (even when run as admin) so my backup never felt complete. Also the backup wasn’t offsite and file recovery was completely manual. Also no versioning of previous files anymore.

Attempt 3- Rsync + SSH + TrueCrypt to a remote friend

One day at work I came up with the idea that I could just backup to my friend’s computer. We both had home servers, unfortunately his was a Windows server, so I had to make my solution work on his windows server while mine was a Linux server.

So I chose several open source pieces. The plan was to have an SSH server running on each end.  I’d run a script on my end the connected to the other server, mount a TrueCrypt volume to make sure the data was encrypted on the other end, run rsync to push the data across. Then unmounting it. Eventually I might have configured a virtual machine to simplying configuration in the future. However, 2 weeks into my tinkering the motherboard died on one of my test servers and I just couldn’t get motivated to try to reconfigure it all over again.

If I could have gotten it to work it would have been quick with rsync’s incremental updates, secure with the SSH and TrueCrypt combo, and off site. Unfortunately it was so incredibly difficult to setup that I knew my next step would have been impossible. There was no way I could get my family backups.

Attempt 4- SyncBack

So, finally I just bought an external hard drive and used SyncBack to duplicate my files to the drive. It was simple and it worked. It actually was simple enough that I actually maintained this system for a while.

The only 2 drawbacks were that the backup wasn’t offsite and no way to get previous versions. I wasn’t too concerned though, because every 6 months I planned on just swapping the external drive to my in-laws when I’d visit.

Final Solution- CrashPlan

I was satisfied with SyncBack and probably would have upgraded to the paid for version, but then I saw on LifeHacker a post for CrashPlan. It presented CrashPlan as a solution that would let you back up to a friend’s computer over the internet and the end result was encrypted on the remote computer.  Its even cross platform. To top the whole deal off: its free!

Its seems too good to be true.  So next I’ll present my testing experience and related observations.

Summary Chart

Fast Encrypted Off Site Easy Versioning Free
SVN check check
rsync check check check
rsync + SSH + TrueCrypt check check check check
SyncBack check check check
CrashPlan check check check check check check

3 thoughts on “My journey to perfect backups

  1. Wow – Thanks for the great writeup. It’s great how you explain the journey. You’re not alone, the developers of CrashPlan had a similar journey.. so not surprising you like where you’re ending up.

    One thing CrashPlan does that the others don’t is automatically (and asynchronously) test your archive at the destination. It doesn’t trust that wherever your data is stored is actually working! What this means is it’ll detect if your drive looses a block and heal around it automatically. It’ll also notify you.

    Another cool feature we added was de-duplication. All that data of yours, there is bound to be duplicate data in there.. so don’t be surprised when the incompressible actually compresses a bit. We had one user with nearly 1TB of data that dropped down to 400GB.

    Anyway, after you’ve kicked the tires a bit more – hop on our forums and tell us which features you’d like to see.

    Best regards,

    CrashPlan Team.

  2. My followup post definitely mentions the de-duplication.

    I’ve already been enjoying the forums too. This morning I commented on the need for a tweak for netbooks and already gotten a response. Very impressed.

  3. CrashPlan looks nice. I ended up getting a Windows Home Server and then backing that up to Amazon Web Services. Took a bit more money, but I can backup my PC, and Emily’s Mac.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.