wamblee.org

Table of contents

Overview

Snapshot is a perl script for creating backups based on Linux Logical Volume Management. Actually multiple modules are used as well but it is one script. When executed the script preforms the following tasks:

  • Check pre-backup requirements: Before the backup takes place a complete check for available disk space is done. This includes a check for available snapshot space as well as space for backups. The aim of these checks is to ensure that nothing can go wrong while doing a backup (read the NOTE below though).
  • Disable services: Each script in the backup.d directory is invoked with suspend
  • Create snapshot volumes: Creates snapshots using LVM.
  • Enable services: Each script in the backup.d directory is invoked with resume
  • Perform backup: After each backup, the snapshot volume that was backed up is removed.

Some interesting features of the snapshot script are:

  • Configurable snapshot volume: Depending on the size and activity of the volume a a different snapshot size can be used. It is also possible to use a different snapshot size per volume
  • Almost consistent snapshots: Multiple volumes are backed up in one go so there is a matter of seconds between the time snapshots are taken. In practice I have been able to backup my whole system and run it completely from a backup disk.
  • Throttling: To reduce system load the number of MB/s is controlled. There is a default max. bandwidth but the max. bandwidth can be changed on the command-line.
  • Progress indication: It will show progress with estimated time left per volume.
  • Backup in dd style: The backup is equivalent to what you would get using linux dd, it does not use dd however because of the detailed progress logging and throttling features.
  • Verification: Verification of the backup using checksums.

The scripts are developed and tested on centos 6.4 and have been used for a number of years already on a weekly basis on an opensuse 11.1 in the beginning, since late 2010 on 11.3, and since 2012 on opensuse 11.3, centos 6.2, 6.3, and 6.4.

Usage

The mksnapshot shell script wrapper

The main entry point for backups is the mksnapshot script and its system configuration file /etc/sysconfig/snapshot. You should edit this configuration file to indicate which volumes you want to backup. Type mksnapshot without arguments for help and also see the comments in the configuration file.

The backup action of the mksnapshot script triggers the aforementioned snapshot perl script which is in /opt/snapshot directory. This script is usually not executed directly because it has an extensive command line. Instead configure the backup using /etc/sysconfig/snapshot and use mksnapshot.

Running mksnapshot requires you to specify one or more lifecycle phases or all of them using all. The lifecycle phases are:

  • login: Login to an iSCSI target.
  • activate: Import and activate backup logical volumes.
  • remove: Remove backup logical volumes.
  • backup: Execute the backup as described in the overview.
  • deactivate: Deactivate and export backup logical volumes.
  • logout: Logout from an iSCSI target

Logging

The script produces a lot of logging output which is useful for troubleshooting and seeing the progress in real time. If you want to see a summary of the backup then use the script tailbackup which provides a much more condensed output showing you the results of the backup for each logical volume.

iSCSI

The iSCSI functionality is optional, login and logout simply do nothing if no iSCSI target is specified. I used iSCSI to be able to dynamically access a disk from a virtual machine. In that case USB is too slow and runtime plugging disks is not supported (yet) by Kernel Virtual Machine.

Snapshot volume sizes

It is important to choose correct snapshot volume sizes. The volumes must be big enough to hold all changes to the snapshotted volumes for the duration of the backup. Typically, put the fastest changing volumes first. In my case this is the VCR directory where I am recording TV. This volume can have at most 4GB/hour of traffic so I make sure I back this up first snd also give it a larger snapshot volume.

Don't screw things up yourself by doing huge data transfers while the backup is running. If you modify data on the original volume, then the snapshot volume will use up more space (copy on write semantics).

Run the backup when the system is not heavily loaded. (In my case with the server running at home this is almost always :-)). This backup will impact system performance somewhat. You can reduce the load by throttling but that will make the backup take longer and may require larger snapshot volume sizes.

Taking consistent snapshots: suspend/resume

When taking a snapshot, some system activity must be frozen. This involves in particular databases. The backup script does this by invoking the scripts in the /etc/snapshot.d directory in alphabetical order with suspend before the snapshot takes place, and invoking these scripts in reverse order with resume after the snapshot volumes have been created.

Investigate all of these scripts to see whether they meet your needs and add, remove, or modify them as needed.

Throttling

Use throttling to reduce the load on your system. Lower bandwidth does however come with the price of longer backup times and therefore possibly bigger required snapshot sizes.

NOTE: I did not verify what happens when a snapshot volume becomes full during a backup.

NOTE: When mounting an XFS backup at the same time the original XFS volume is still mounted you will get an error because of a UUID conflict. Solve this by mounting the XFS backup with the nouuid option of mount or by changing the UUID.

Releases

The scripts are packaged in RPM format for Opensuse 11.3. The RPMs might work on other RPM based systems as well. The URL for the repository is http://wamblee.org/yum/public/. The RPM is snapshot

Download

The subversion URL for read-only access is https://wamblee.org/svn/public/snapshot/trunk .

Disclaimer

Use it at your own risk. I have not seen any instability of the system in the past year because of these backups (performed every week). However, on my previous server, throttling was absolutely necessary as the system could become unstable. I have not seen this issue on my newest server (Intel Xeon L5630, 24GB), but you have been warned.