virt-back: a python libvirt backup utility for kvm xen virtualbox

This entry is part 1 of 3 in the series Virt-back

Over the weekend I wrote virt-back, a backup utility for QEMU, KVM, XEN, or Virtualbox guests.

virt-back is a python application that uses the libvirt API to safely shutdown, gzip, and restart guests.

The backup process logs to syslog for auditing and virt-back works great with cron for scheduling outages. Virt-back is in active development so feel free to give suggestions or branch the source.

virt-back has been placed in the public domain and the latest version may be downloaded here: https://bitbucket.org/russellballestrini/virt-back

Installation:

The fastest way to install virt-back is to use pip or setuptools.

Try sudo pip install virt-back or sudo easy_install virt-back

Otherwise you may manually install virt-back
sudo wget https://bitbucket.org/russellballestrini/virt-back/raw/tip/virt-back -O  /usr/local/bin/virt-back
sudo chmod 755 /usr/local/bin/virt-back
Test installation:
virt-back --help
Example cronjob:
15  2  *  *  1  /usr/local/bin/virt-back --quiet --backup sagat
15  23 *  *  5  /usr/local/bin/virt-back --quiet --backup mbison
Manual:
russell@host:~$ virt-back --help
Usage: virt-back [options]
Options:
  -h, --help            show this help message and exit
  -q, --quiet           prevent output to stdout
  -d, --date            append date to tar filename [default: no date]
  -g, --no-gzip         do not gzip or tar the resulting files
  -a amount, --retention=amount
                        backups to retain [default: 3]
  -p 'PATH', --path='PATH'
                        backup path [default: '/KVMBACK']
  -u 'URI', --uri='URI'
                        optional hypervisor uri

  Actions for info testing:
    These options display info or test a list of guests.

    -i, --info          info/test a list of guests (space delimited dom names)
    --info-all          attempt to show info on ALL guests

  Actions for a list of dom names:
    WARNING:  These options WILL bring down guests!

    -b, --backup        backup a list of guests (space delimited dom names)
    -r, --reboot        reboot a list of guests (space delimited dom names)
    -s, --shutdown      shutdown a list of guests (space delimited dom names)
    -c, --create        start a list of guests (space delimited dom names)

  Actions for all doms:
    WARNING:  These options WILL bring down ALL guests!

    --backup-all        attempt to shutdown, backup, and start ALL guests
    --reboot-all        attempt to shutdown and then start ALL guests
    --shutdown-all      attempt to shutdown ALL guests
    --create-all        attempt to start ALL guests

Series Navigationvirt-back: restoring from backups

36 thoughts on “virt-back: a python libvirt backup utility for kvm xen virtualbox

  1. Hello,
    i’m trying your script to backup guest.
    I thought when the backup was done the script restarted the guest but it is not the case.
    could you give me information?
    thanks for your script
    bye

  2. Hey Pascal,

    The script has intelligence and keeps track of the guests state before backup.

    If the guest was running when the backup was initiated then the guest will be started when the procedure completes.

    However, if the guest was shut off when the backup was initiated, then the guest will not be started when the backup is finished.

    Also make sure that the guest is defined in libvirt:

    syntax:
    virsh define /path/to/dom/xml

    example:
    virsh define /etc/libvirt/qemu/mbison.xml

    Let me know if you have any other questions and I’ll be glad to troubleshoot further.

  3. hey Russel,
    first thanks for your answer.
    i create guest with the “virt-install” command. does this command define guest in libvirt ?
    what is the way to see if a guest is defined? when i do a virsh list –all i can see all my guest.
    when i use your script on a running guest, the backup works fine, but the guest is not re started.
    thanks again for your script and your answer.

    1. Could you try a test for me?

      Could you shut off your guest and then try to start it using virt-back?

      # shutdown
      virsh shutdown

      # start
      virt-back –verbose –create

      If this doesn’t work there might be something wrong…

  4. Hey Russel,
    it was a mistake from me.
    i let the script in my home directory.
    after i put it in /usr/local/bin, it works
    thanks for your answers and your script

  5. Hey Russel,

    something seems to be wrong with your script.

    ——————————————-
    virt-back -p /root/backup/ -b vm132
    Traceback (most recent call last):
    File “/usr/local/bin/virt-back”, line 242, in
    if options.backup or options.backupall: backup( domList )
    File “/usr/local/bin/virt-back”, line 87, in backup
    tar.add( disklist[0] ) # we could loop and tar each disk in disklist
    IndexError: list index out of range
    ——————————————-

    Creating the VM with “virt-back -v –create vm132″ works like a charme. Also see

    root@host ~ # virsh ‘list'”

    Id Name State
    ———————————-
    5 vm132 running

    root@host ~ # virt-back –info-all –verbose

    name: invoking name on vm132
    name: name returned vm132 on vm132
    info: invoking info on vm132
    info: info returned [1, 524288L, 524288L, 1, 11680000000L] on vm132

  6. Hi Russell,

    I was able to successfully backup my VM using your script, but I can’t figure out how to restore it. I just rebuild my system and need your help!

    Thx

  7. ok, i see: “A snapshot volume only stores the delta (changes) since it was created. This is convenient for a point-in-time, well, snapshot, but if your original data is lost, you can’t restore from a snapshot. ”
    What about first question?

      1. I do many work on your script before :
        – I already implement multi-disk support
        – support for qcow2 snapshot to minimise the downtime.
        – usage “suspend” instead of “shutdown”
        – add option to restore a (previously backup) VM
        – add ability to migrate a VM (supend, transfer and re-up on another server)

        have a look here if you want this full feature version : https://bitbucket.org/guilhemfr/virt-back

        (I do pull-request for this…)

    1. himuraken, Today is your lucky day. I have implemented a retention flag for setting the amount of backups to keep when rotating. Please use the latest version of virt-back.

      -a amount, –retention=amount
      backups to retain [default: 3]

  8. In fact, I did a vim /usr/local/bin/virt-back and found:
    if path.isfile( tarpath ): # if file exists, rotate 3 versions

    Can I just increment rotate 3 to the number of revisions desired?

    Thanks!

  9. Hello,

    Thanks for this utility. It is really useful.

    I am using the script and have not had any success in limiting the number of retained backups. This ends up filling up my backup disk (a NAS drive mounted via NFS), so I need to figure out how to resolve it. Here is an example of the command I am using via cron job:

    30  1  *  *  *  /usr/local/bin/virt-back --quiet --backup --date --retention=2 -p /vmimages1/backups scre-ubuntu-log
    

    The retention flag is being ignored. (Currently there are 4 backups stored, and if I don’t delete them manually, they will stay there.) I am not seeing anything about the rotation in the logfile/syslog:

    Sep  3 01:30:01 535-1201a-vm9 CRON[4486]: (root) CMD (/usr/local/bin/virt-back --quiet --backup --date --retention=2 -p /vmimages1/backups scre-ubuntu-log)
    Sep  3 01:30:02 535-1201a-vm9 virt-back: invoking shutdown on scre-ubuntu-log
    Sep  3 01:30:02 535-1201a-vm9 virt-back: waited 0 seconds for scre-ubuntu-log to shut off
    Sep  3 01:30:12 535-1201a-vm9 virt-back: waited 10 seconds for scre-ubuntu-log to shut off
    Sep  3 01:30:22 535-1201a-vm9 virt-back: waited 20 seconds for scre-ubuntu-log to shut off
    Sep  3 01:30:32 535-1201a-vm9 virt-back: waited 30 seconds for scre-ubuntu-log to shut off
    Sep  3 01:30:42 535-1201a-vm9 virt-back: waited 40 seconds for scre-ubuntu-log to shut off
    Sep  3 01:30:52 535-1201a-vm9 virt-back: waited 50 seconds for scre-ubuntu-log to shut off
    Sep  3 01:31:02 535-1201a-vm9 virt-back: invoking backup for scre-ubuntu-log
    Sep  3 01:31:02 535-1201a-vm9 virt-back: copying /vmimages1/img/scre-ubuntu-log.vmdk to /vmimages1/backups/scre-ubuntu-log.vmdk for scre-ubuntu-log
    Sep  3 01:32:46 535-1201a-vm9 virt-back: invoking create on scre-ubuntu-log
    Sep  3 01:32:50 535-1201a-vm9 virt-back: archiving files for scre-ubuntu-log to /vmimages1/backups/scre-ubuntu-log-2014-09-03.tar.gz
    Sep  3 01:32:50 535-1201a-vm9 virt-back: archiving /vmimages1/backups/scre-ubuntu-log.xml for scre-ubuntu-log
    Sep  3 01:32:50 535-1201a-vm9 virt-back: archiving /vmimages1/backups/scre-ubuntu-log.vmdk for scre-ubuntu-log
    Sep  3 01:52:58 535-1201a-vm9 virt-back: finished backup for scre-ubuntu-log
    

    I would appreciate your help in resolving this. Thanks for this great utility.

    1. Hey Jennifer,

      Just a guess but I don’t think the way I wrote the retention will work with the --date option.
      This is because the filenames are always unique. You must choose either to put dates into your filenames and find a different way to perform rotations, or use the --retention flag and let virt-back rotate the files.

      1. OK, thanks, I’ll try removing the date option and see if retention works then. Thanks for the quick reply. Will let you know if that works.

  10. Hello, getting this error:

    root@cypdell1:/home/modias# ./virt-back –backup-all
    shutdown: invoking shutdown on oxygen
    shutdown: waited 0 seconds for oxygen to shut off
    shutdown: waited 10 seconds for oxygen to shut off
    backup: Create XML file with configuration of oxygen
    backup: invoking backup for oxygen
    backup: copy of /var/lib/libvirt/images/oxygen.img to /home/modias/oxygen.img
    Traceback (most recent call last):
    File “./virt-back”, line 531, in
    files = backup(dom)
    File “./virt-back”, line 134, in backup
    copytmpFile = join(options.backpath, basename(tmpfile))
    File “/usr/lib/python2.7/posixpath.py”, line 121, in basename
    i = p.rfind(‘/’) + 1
    AttributeError: ‘NoneType’ object has no attribute ‘rfind’

    i am running Ubuntu 14.04 LTS – the guest was running with a .img file under /lib/libvirt/images/os.img and a storage drive from a LVM

  11. Hello Russell,
    i try your backup script, it seems to be very nice. But one thing is not very well. It use the normal gzip single threaded, that is very slow. I search your script, but python and I don’t be friends ;) I don’t find where you call gzip. Where can I change your script to use pigz?
    Thank You

  12. Hello – is it possible to exclude one (or more) of the disk images associated with a VM? For instance, I have a VM that has two disk images associated with it – one is the OS/Applications and the other is Data. I only want to back up the Data image.

    If this is not an option currently, please consider this a feature request. Thank you!

    1. Just following up on this – not sure if you saw my initial comment regarding excluding certain disk images from backup. Thanks!

        1. Thanks for the answer on that. I would appreciate if the ability to exclude specific disk images would be supported in a future version. Thanks for this great utility.

Leave a Reply

Your email address will not be published. Required fields are marked *