Well, it finally happened. We keep a secure login to our home network open just in case we need files that aren’t on the laptop, or, to use as a relay point when locked behind a customer’s firewall, or, well, because we can. While on travel, our gateway machine at home ceased responding. Oh-oh. When we got home, our fears were confirmed: the power supply had failed. It just so happens that the gateway machine hosted some virtual machines that we need for vital business, so we had to get it recovered quickly. It’s not a good plan to have valuable data on your gateway machine: that also has been corrected–we moved the gateway to another box.
In most of these cases, we might be faced with installing the software (which, unfortunately, was a Microsoft Windows application, so there are licensing and compatibility issues) on another machine, then restoring the data from backup, and so on. Furthermore, we discovered that, while the user files and configurations were backed up, the volume containing the virtual machine images (along with the data) was not. Oops, again. Although we do manually backup the data from time to time, it was a bit overdue. But, the disk was still good, so we popped the hard drive out of the failed unit, opened up another machine, plugged it in, and transferred the virtual machine image and configuration files to the second machine.
Viola! in the time it took to copy a 20GB file (we build virtual machines with as small hard drives as necessary, as they are usually special-purpose machines anyway), we had recovered the application and the data. However, it wasn’t convenient to run it from the new host, so a few days later, we made room on another system and transferred the virtual machine image once again, this time across the network. Of course, we immediately included the virtual disk images in the backup scheme, and changed our procedures so we shut down the system when not in use so we get a clean backup image.
One of the reasons we hadn’t been backing up the virtual disk image is that, when the system is running, the image is inconsistent and might not be recoverable anyway, unless we can snapshot it (i.e., capture a static image that can be backed up and recovered). With most systems, we simply run a backup client (which for rsnapshot, is simply the SSH daemon that is usually on anyway) on the virtual machine. But, we don’t usually run an SSH service on Windows, so a different backup system needs to be implemented for Windows. There are a number of operating-system-agnostic backup software systems available, even several open-source, but they aren’t as convenient as what we use. However, losing valuable data is extremely inconvenient, so we need a different approach.
On our portable systems, we use Oracle’s VirtualBox to run our virtual machines. VirtualBox is intended for desktop use: the VM is the property of a single user and can be easily migrated from system to system or hosted on networked storage and launched from one of several different workstations. The most frequent use in this case is to virtualize Microsoft Windows systems for running those few applications for which we do not have an equivalent Linux application or which will not run under the WINdows Emulator (WINE). For training, we often use VMWare appliances, which are also easy to install and migrate. Within our network, the network services–such as DNS, web servers, and file servers–and development for multiple Linux, BSD, and Solaris distributions are virtualized on Citrix Xenserver, running on a server-class machine dedicated to hosting virtual machines.
Recently, we attended a seminar on Ganeti, which is Google’s answer to keeping virtual machines running all the time. We are thinking of migrating our systems to Ganeti, a cluster management system for virtual machines that keeps mirrored copies of virtual machines on multiple servers in a cluster, so that the VM is always available, even if any single node fails. And, if hosted on three or more machines, any two nodes can fail without loss of data or incurring downtime. This will solve the issue of backing up non-Unix VMs and the several hours of downtime needed to restore a backup to another system.
Virtualization is the future of computing, where we depend on having all our data available all the time, or need multiple systems but have desktop space for only one. There are some performance and technical issues, such as enabling audio and accessing optical and flash drives, but using local virtualization like VirtualBox and VMware appliances on a workstation helps solve that problem, as long as the VMs get regular snapshots or are shut down during system backup times.