Efficiently Move rsnapshot Repository Between FuseCompress Volumes

cody — Mon, 02 Jan 2012 07:28:48 +0000

As explained in my last post, I’m moving from a Fusecompress 2.0 backend to the more stable 0.9.x branch. I’m using the following code to move the repository, incrementally (and in recoverable way). The disk is too small to maintain both copies at once. Because the normal tools (mv/cp/rsync) work depth-first and daily.0 is nearly identical to daily.1 (and daily.2, etc.), only a small amount of space is freed after moving daily.0 and the disk eventually fills.

As with everything posted here, use this at your own risk and ALWAYS backup!

dest=/home/backup.new/snapshot
source=/home/backup/snapshot

# Create the directories
find . -type d -print0 | while read -r -d '' file
  do
  if test ! -d "$dest/$file"
    then
      mkdir -vp "$dest/$file"
      touch --reference="$file" "$dest/$file"
      chown --reference="$file" "$dest/$file"
  fi
done

# Move the files in, link any duplicates, and remove the original.
for ssnap in `ls $source`
  do
  cd $ssnap
  find . ! -type d -print0 | while read -r -d '' file
    do
    if test ! -e "$dest/$ssnap/$file" -o -h "$dest/$ssnap/$file"
      then
        cp -adnvp "$file" "$dest/$ssnap/$file"
      else
        echo "$dest/$ssnap/$file" exists.
    fi
    for dsnap in `ls $source`
      do
      if test "$ssnap" = "$dsnap"
        then continue
      fi
      echo "Checking $dsnap - ../$dsnap/$file"
        if test "$file" -ef "../$dsnap/$file";
        then
          ln -v "$dest/$ssnap/$file" "$dest/$dsnap/$file"
          rm -vf "../$dsnap/$file"
        fi
    done
    rm -vf "$file"
  done
done

Updated 2012-01-03: Filenames/Directories names with new line character caused mischief. Fixed by null separating. Also included symlinks, char devices, and other special filetypes in the transfer.

Verify Hardlink Integrity of RSnapshot Copy

cody — Wed, 28 Dec 2011 18:17:08 +0000

I use fusecompress as a backing file system for rsnapshot. Unfortunately I’ve been having crashes with the 2.0 C++ fork, so I’ve tried converting back to the 0.9.x C implementation. That’s been a tricky process because my disks are only just a bit too small for two concurrent copies of the rsnapshot tree. I’ve been using rsync to duplicate the tree, but because of the disk overfill it’s failed a few times. RSnapshot uses hard links to deduplicate files between snapshots (daily.0, daily.1, etc). Since rsync had failed a few times I wasn’t sure that it would maintain hard links consistently between the yet to sync’ed portion and the previously sync’ed portion after being restarted. This script compares the hardlink structure between the two disk stores and generates a script to remedy any inconsistencies. That said, rsync seems to behave properly because the script found no inconsistencies.

find . -type f | while read file
  do if (test "$file" -ef "../daily.2/$file");
    then
      if ! (test "/home/backup.new.fusecompress.backing/snapshot/daily.1/$file" -ef "/home/backup.new.fusecompress.backing/snapshot/daily.2/$file")
      then
        echo rm -f "/home/backup.new.fusecompress.backing/snapshot/daily.2/$file" >> /root/linked.daily.1
        echo ln "/home/backup.new.fusecompress.backing/snapshot/daily.1/$file" "/home/backup.new.fusecompress.backing/snapshot/daily.2/$file" >> /root/linked.daily.1
    fi
  fi
done

Cody Baker

Efficiently Move rsnapshot Repository Between FuseCompress Volumes

Verify Hardlink Integrity of RSnapshot Copy