Tools 26 Apr 2007 02:14:02

Incremental Backup

A long time ago when I started wanting good incremental backups for my data, I looked around the net and found many good tools, but none that did exactly what I wanted. Either the process to restore from the backup was slow, or they didn’t support SSH, or any number of other small annoyances.

So I wrote my own wrapper around rsync to do what I wanted: incremental.sh. And now I’m releasing that for anyone else to use. Pretty much as long as the source and target machine can run rsync, they can use the wrapper. Instructions are in the script itself.

It makes use of rsync’s –link-dest option to minimize space usage for increments by hard linking unchanged files to the previous increment’s version. I run it nightly, and have between 100x to 1000x speed improvements over doing a full backup, plus even though I keep several days’ worth of changes, the used space overhead is limited to only what has changed between current and oldest version.

Another plus is how restoring from any of the increments is as easy as copying the file out of that folder. And a full restore from that point in time is an rsync away. I have done several full restores from backups made with this script, and I so far haven’t found a better format for backups.

One minus is that currently this is only a pull script. I have no push version, nor need of one, as I prefer running a read-only rsync daemon firewalled down to only the IPs that will be pulling data from it.

URLs

Comments (5)

5 Responses to “Incremental Backup”

  1. on 04 Sep 2007 at 22:39:19 1.Anders said …

    Have you tried rdiff-backup?

  2. on 21 Sep 2007 at 19:24:23 2.Tino Didriksen said …

    rdiff-backup essentially does all that rsync does and even uses the rsync protocol, but I needed something that would run without modification on Cygwin, RedHat 8, Mac OS X, as well as Fedora 7. rdiff-backup had such varying versions (RH8’s 0.12.7 vs. Fedora’s 1.0.5) that I felt better using rsync.

    Another thing is rdiff-backup’s increment format. Reverse diffs will certainly result in less disk space eaten by increments, but I prefer having the actual files in the folder tree. For larger backups I may have to use rdiff-backup simply due to space constraints, but I am not quite there yet.

  3. on 14 Oct 2008 at 03:46:56 3.Daniel said …

    Thanks a lot!
    With some adaptation and a crontab, this was what I was trying to do with my little-to-nothing knowledge of scripting.

    Thanks a lot!

  4. on 27 Aug 2010 at 16:20:47 4.Hendy said …

    I’m not knowledgeable on scripts… how might this be modified to create a backup list like this (as opposed to just *.0, *.1, *.2…):

    backup.hour
    backup.day
    backup.week
    backup.month

    Doest that make sense? Some way to keep one for a reasonably far back time but build on top of it more recently as time moves on?

  5. on 27 Aug 2010 at 16:44:49 5.Tino Didriksen said …

    I’d say go with http://www.rsnapshot.org/ or http://rdiff-backup.nongnu.org/ – they will do what you want, so no need to reinvent that wheel.