Trouble Free Backups, Part One - rsync and Strongspace
If you’ve been using computers for a while, chances are you’ve lost valuable data more than once. Most computers nowadays come with CD writers (and many with DVD writers) which is great for the occasional manual backup - there is even the nice OSX-specific “iSync”:http://www.apple.com/macosx/features/isync/ tool which remind you when to perform the backup, and can manage the files being backed up too. However what would be better would be an automatic solution.
The “rsync”:http://samba.anu.edu.au/rsync/ tool ships with OSX, Linux, and even “Windows(Installing ssh and rsync on a Windows machine: minimalist approach)”:http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html. Rather than copying and overwriting files on a remote location @rsyc@ performs incremental transfer of files - only those files not already present will be copied - this drastically reduces the time taken to perform backups.
For my purposes, I wanted files to be backed up off site. I decided to sign up for a “Strongspace”:http://www.Strongspace.com/ account. Strongspace provide external, secure storage - you can access your files either via the web interface, SFTP, or (handy for our purposes) via @rsync@. While the rest of the article (and following parts) assume you’ll also be backing up to a Strongpsace account, it is a small matter to instead copy your files to an external disk drive or even another computer on your network. Note that Strongpsace supports @rsync@ by running an @rsync@ server - so these instructions should work equally well for backing up to any other @rsync@ server.
Assuming you have your Strongspace account setup already, lets get started. To start with, I simply want to backup the contents of my @Documents@ folder:
rsync -azv /Users/johnsmith/Documents \ johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac
When run, this will copy the contents of @Documents@ to the remote directory @/home/johnsmith/backups/mac@. Unfortunately, since the release of Tiger my @Documents@ folder has become stuffed with lots of Dashboard widgets I never use. To ignore these, I specify the @exclude@ option:
rsync -azv --exclude "Widgets" \ /Users/johnsmith/Documents \ johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac
The @exclude@ option can take a regular expression, so you can specify multiple directories if you want to. If you want to backup files from multiple locations, then consider having a single @backup@ directory, then create symlinks to the directories you want backed up. You’ll need to add the @L@ option so that @rsync@ copies the referenced directory rather than simply copying the symlink itself:
rsync -azvL \ /Users/johnsmith/backup \ johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac
Notice that when you run the command, you get prompted every time for your strongspace password. In “part two(magpiebrain - Trouble free backups, Part Two - SSH Keys for trouble free authentication)”:http://www.magpiebrain.com/archives/2005/10/31/strongspace_and_ssh we’ll start using an SSH key to avoid this, and we’ll be well on the way to providing an automatic backup solution.
Category: howto 9 comments »
November 3rd, 2005 at 6:07 am
You should probably mention that the built in rsync in Mac OS X doesn’t copy the resource fork which means that the files are pretty useless when restored from backup. You need to get RSyncX from here:
http://archive.macosxlabs.org/rsyncx/rsyncx.html
November 3rd, 2005 at 9:10 am
Ah - I wasn’t aware of that, thanks Jon (although I have been using the Fink build of rsync rather than the built in one). I’ll get round to updating the tutorial in a day or so.
November 3rd, 2005 at 9:24 am
Google’d a bit, and found that RsyncX won’t work perfectly if the destination server isn’t also OSX (apparently suome of the extra metadata can get lost). There is a “patch(rsync+hfsmode patch)”:http://www.quesera.com/reynhout/misc/rsync+hfsmode/ for the OSX rsyc which enables the copying of HFS+ resource forks and finder metadata.
Some more digging turns up at least one reference to rsync supporting resource forks in Tiger - I’ll try and test this tonight - so from the sounds of it you should only need rsyncx/the HFS+ patch if running Panther or earlier.
November 3rd, 2005 at 10:32 am
More googling implies rsync has a bug with resource forks in Tiger:
bq. on Tiger, rsync has a critical bug when handling files with resource forks. The bug makes rsync generate a new copy of files and directories with resource forks in each increment.
(from “this topic(AppleTalk forum)”:http://forums.appletalk.com.au/index.php?s=ba6dc11860726cb374cc170f466d6464&showtopic=11327&st=0&#entry93258)
*sigh* - for me this is probably OK though, I really only backup code and my documents to strongpsace. The bulk of my stuff gets backed up to an external drive, so I guess I can use RsyncX (or even Backup 3) for that.
November 3rd, 2005 at 10:34 pm
Yeah, it’s a bit of pain in the backside those resource forks. Please do blog if you come up with a good solution for using rsync to a non-OSX machine that preserves resource forks.
(Sometimes I just wish Apple would’ve dropped resource forks when they switched to Unix.)
November 4th, 2005 at 9:17 am
They _are_ deprecated according to Apple - they’ve been trying to get people to move away from them for a while, but I guess they had to keep them for backwards compatibility.
For the moment it looks like I’ll have to use a local full backup to either an external drive or another mac, and use my strongspace stuff for things like my documents, code and the like.
November 9th, 2005 at 5:27 pm
Apparantly, the version of rsync that ships with Mac OS X 10.4 includes an -E option to include “extended attributes” of files (such as resource forks). There are issues with Spotlight though: see this hint: http://www.macosxhints.com/article.php?story=20051104185525439&lsrc=osxh
March 2nd, 2006 at 1:08 pm
What I’ve done to back up several machines on a staff network is:
1. Create a central server which uses rsync to back up each Mac
2. That central server nfs-mounts a Unix box (but you could do it with sftp virtual drives if nfs is not available or desirable)
3. The central server then saves the data to a disk image using hdiutil on the unix server. Voila: resource forks preserved, and all you have to do is download a handy disk image when you want to restore :)
March 29th, 2007 at 10:48 pm
So, I still have not had a consistent answer on what happens if you *do* use rsync to try and backup to Strongspace (which I have by dint of my Joyent textdrive account).
Basically, an rsync backup of my Mac would kick a** (particularly as my flat was burgled and drive stolen few months back).
So, let’s say I rsync my entire Users folder to Strongspace and then need to restore the entire thing back to a new machine because of a complete hard drive failure on the PB.
Has anyone actually tried to see what would happen ? with or without the -E flag ? (in particular, mail archives, documents, photos, sites and sources directories and perhaps a compressed automated mysql dump)
Basically, I just want to know that I’d be able to reconstruct critical data in the event of catastrophic loss.
let me know,
Daryl.