All things Unix, Linux, Uncategorized

Drivers Wanted — Call 1-888-GOT-UNIX*

*not a real phone number

The English language has been characterized as one that not so much borrows from other languages, but outright steals, remakes, and repurposes words to suit one or many purposes. So, the “Drivers Wanted” signs on the back of 18-wheeler trailers has different meaning to Unix system administrators. But, just as the long-haul trucker operates the tractor that gets your load hauled to where it is needed, a Unix device driver is a piece of software that gets your data to and from an external device, like a hard disk, dvd, printer.

Recently, Chaos Central experienced several close encounters of the frustrating kind with device drivers. Unix device drivers have always been a challenge, but the growth of open source software has made them even more so. Device drivers have traditionally been provided by the manufacturers of computer equipment, either through iron-clad non-compete agreements with commerical Unix vendors (like Oracle and Apple) for source code or operational specification for the device or in binary form in the case of Windows computers. The problem arises with Linux: in order to be distributed with the GNU open-source licensing, all components must be open source and freely redistributable, and many device drivers are not.  At best, drivers come as object files and header files that can be linked with the kernel version at the current patch level.

Device manufacturers develop and deliver driver software in binary form for Microsoft Windows, because they sell most of their products for use with computers running Windows. However, since the hardware is proprietary and the software required to interface with it reveals much of the inner workings and design, vendors are reluctant to release the source code to Linux developers. Also, because the Linux market is relatively small, they are also reluctant to produce binary versions compatible with Linux. In cases where there are binary drivers for Linux, the drivers are specific to kernel revision. Since the Linux kernel is frequently patched and has at least a minor revision release every six months, keeping a complete configuration inventory is problematic, even for systems that are supported, which brings us to the first pothole in the information highway…  And then, there is the problem of keeping drivers current with firmware changes in the hardware.  The devices themselves are often programmable circuitry, with flash memory.  Changes to firmware usually also require changes in the host driver.

Fiber Optic/Multipath Disk Drivers

It is no secret by now that the majority of Internet servers and almost all high-performance computing systems run Linux. High-end server hardware is most often attached to high-end storage systems as well, most of which achieve high throughput by using fiber optic connections. So, most fiber channel Host Bus Adapters do have device driver support for Linux, either available directly from the vendor or, in the case of supported Linux distributions, as part of a paid service package from the Linux provider. But, these choices also present what is basically a fork in the configuration management of the drivers: for instance, the Red Hat version does not always match the QLogic version. The reason for this is complex, but usually the hardware vendor wants to keep the drivers and firmware as up-to-date as possible to correct problems and support newer platforms and storage devices. The operating system vendor wants to keep older systems running smoothly, as well as supporting newer systems. Both need to balance the cost of maintenance against the number of installed systems. Inevitably, combinations of hardware and software exist that are not supportable, or that require some additional work by the system administrator to maintain a working system.

Thus, it came to pass that, during a routine patch-level upgrade to bring a pair of systems to the same Red Hat 5.8 level, a warning popped out on the console, announcing components of the fiber-optic driver were missing. Fortunately, Linux systems are built in such a way that the system software libraries maintain compatibility with all patch levels of a particular kernel release: Red Hat does not update the kernel release through a major distribution release, so it is possible to update the system software without updating the kernel patch level, so that the drivers do not need to be rebuilt. It was a simple matter of editing the grub.conf file to reboot using the old kernel version (which is why Linux retains the older kernels in the Grub menu). Even though most device drivers are now loadable at run time, if they are needed during the boot process, they must be integrated into the boot image. In the case of these machines, the boot disk appears to be local, but the driver is needed to mount the data drives for the server application. Being able to build a boot image with the driver is usually a pretty good indication that the driver will load when needed.  Also, many high-end systems use multiple paths between the host bus adapter and the storage array, for both fail-over and to increase throughput.  Multipathing needs to know what devices support it, so even if the system will boot without the device driver, multipathing may not work.  In most Linux systems, the device mapping will differ between multipath and non-multipath configurations, so the correct devices will not be available if the drivers are loaded in the wrong order.

But, it is desirable to have the kernel patches installed as well, so the problem of building a boot image with the driver installed had to be solved. There was a utility package for rebuilding the current image, but that presupposes that the system would actually reboot without the driver installed. A bit more research turned up a manual procedure for building a boot image from any kernel, which appeared to work. To complicate issues, this upgrade was for a remote client, so standing at the console to select a fall-back kernel in case the reboot was unsuccessful was not an option. In these cases, it is necessary to have someone standing by at the physical machine to intervene if necessary during the reboot. So, final resolution is awaiting the next maintenance availability for the system in question.

Printing

Meanwhile, in our own office at Chaos Central, we have been struggling with printing problems for months. Our aging Xerox color laser printer has been printing slowly or simply hanging in mid-print for some time. Canceling the job, then turning the printer off for 30 seconds to clear the queue and back on again worked, sometimes. And, when the printer went into “Power Save” mode, it wouldn’t wake up when we sent it a print job. We had recently replaced the expensive print cartridges, so the budget wouldn’t support a new printer right now. But, the symptoms seemed to indicate a driver problem more than anything else, since the problems seemed to crop up with a recent system upgrade.

Printing from Unix has always been an issue. From early on, Postscript has been the de facto common printer language, so we were faced with buying Postscript-capable printers in the 1990s. With the advent of CUPS, the Common Unix Printing System, it is possible to use almost any printer for which a CUPS PPD file is available, and to use job control features in various Postscript printers. Again, vendors defer to Windows for their proprietary print drivers, and send the Linux user elsewhere for Linux “drivers.” However, all PPD files are not created equal, and they change from time to time.

Before throwing out the old printer, we decided to try bypassing CUPS entirely. Most network printers support multiple printing protocols, one of which is FTP, the venerable File Transfer Protocol, used since the days of dial-up modems, but deprecated in the Internet age due to lack of encrypted security measures. Normally, the Network Police (i.e., the network security administrators) insist that this feature be turned off. But, with it turned on, it is a simple matter of uploading a Postscript file to the printer, using anonymous FTP. Which we did; and, voila!, it printed.

So, a search for a new printer driver ensued. We did find one at the Xerox site, with a PPD file several times larger than the generic one we were using that came with the latest CUPS package. And, the printer now wakes up from a sound sleep and prints! Yes, it is very slow yet, but it does print. The slow printing seems to be associated with different pieces of software that render the print selection to Postscript, but we can at least produce output more or less on demand.

Network

It has been some time now since we’ve had network problems, but getting connected “on the road” was a major problem until recently, as Linux drivers were not available for a lot of wireless cards bundled with laptops.  The choices for a while were to run a utility that mapped the system calls in Windows drivers to Linux systems calls, which had variable results.  Fortunately, many vendors now provide support for Linux and it is possible to buy laptops now made to run Linux.  But, as recently as 2011, using our old (ex-Windows) systems, we had a problem with the connecting layer between the driver and the I/O that reset the network connection every couple of minutes in an attempt to keep connected with the strongest signal.  In hotels or congested areas where there were multiple access points broadcasting on the same channels, this effectively precluded any kind of meaningful network experience and certainly made persistent connections (like file transfers or remote administration) impossible.  This issue seems to have been corrected in later versions of the wireless software, but there is always the next surprise waiting in the world of Linux device drivers.

Bicycling, Uncategorized

Warm Showers 2011

Gallery

The gallery of Warm Showers travelers hosted in 2011.  Warm Showers is an international on-line community of bicycle tourists who share lodging with fellow tourists.  Since we live on the Adventure Cycling Pacific Coast Route and popular routes for tours of the Olympic Peninsula an Hood Canal, we have the opportunity to host bicycle tourists almost every week that we are home during the season, which lasts roughly from early May through late October.

Todd – Vancouver, BC – Imperial Beach, CA
Heather and Adam – Lacey, WA to San Juan Islands to Seattle

Stefaan and Mark: Vancouver, BC – Imperial Beach, CA
Ray – www.biketouringtips.com – Olympic tour, Washington/Oregon
Peter : Yukon to Tierra del Fuego
Laura and Gabi – Seattle to San Francisco
Rueben, Heidi, Eden, and Harper – Toronto to Cheyenne, Vancouver, BC to Panama City, Panama, Richmond to Toronto: one year, 8000 miles.
All things Unix, Linux

Virtues of Virtualization

Well, it finally happened.  We keep a secure login to our home network open just in case we need files that aren’t on the laptop, or, to use as a relay point when locked behind a customer’s firewall, or, well, because we can.  While on travel, our gateway machine at home ceased responding.  Oh-oh.  When we got home, our fears were confirmed: the power supply had failed.  It just so happens that the gateway machine hosted some virtual machines that we need for vital business, so we had to get it recovered quickly.  It’s not a good plan to have valuable data on your gateway machine: that also has been corrected–we moved the gateway to another box.

In most of these cases, we might be faced with installing the software (which, unfortunately, was a Microsoft Windows application, so there are licensing and compatibility issues) on another machine, then restoring the data from backup, and so on.  Furthermore, we discovered that, while the user files and configurations were backed up, the volume containing the virtual machine images (along with the data) was not.  Oops, again. Although we do manually backup the data from time to time, it was a bit overdue.  But, the disk was still good, so we popped the hard drive out of the failed unit, opened up another machine, plugged it in, and transferred the virtual machine image and configuration files to the second machine.

Viola! in the time it took to copy a 20GB file (we build virtual machines with as small hard drives as necessary, as they are usually special-purpose machines anyway), we had recovered the application and the data.  However, it wasn’t convenient to run it from the new host, so a few days later, we made room on another system and transferred the virtual machine image once again, this time across the network.  Of course, we immediately included the virtual disk images in the backup scheme, and changed our procedures so we shut down the system when not in use so we get a clean backup image.

One of the reasons we hadn’t been backing up the virtual disk image is that, when the system is running, the image is inconsistent and might not be recoverable anyway, unless we can snapshot it (i.e., capture a static image that can be backed up and recovered). With most systems, we simply run a backup client (which for rsnapshot, is simply the SSH daemon that is usually on anyway) on the virtual machine.  But, we don’t usually run an SSH service on Windows, so a different backup system needs to be implemented for Windows.  There are a number of operating-system-agnostic backup software systems available, even several open-source, but they aren’t as convenient as what we use.  However, losing valuable data is extremely inconvenient, so we need a different approach.

On our portable systems, we use Oracle’s VirtualBox to run our virtual machines. VirtualBox is intended for desktop use: the VM is the property of a single user and can be easily migrated from system to system or hosted on networked storage and launched from one of several different workstations.  The most frequent use in this case is to virtualize Microsoft Windows systems for running those few applications for which we do not have an equivalent Linux application or which will not run under the WINdows Emulator (WINE).  For training, we often use VMWare appliances, which are also easy to install and migrate.  Within our network, the network services–such as DNS, web servers, and file servers–and development for multiple Linux, BSD, and Solaris distributions are virtualized on Citrix Xenserver, running on a server-class machine dedicated to hosting virtual machines.

Recently, we attended a seminar on Ganeti, which is Google’s answer to keeping virtual machines running all the time.  We are thinking of migrating our systems to Ganeti, a cluster management system for virtual machines that keeps mirrored copies of virtual machines on multiple servers in a cluster, so that the VM is always available, even if any single node fails.  And, if hosted on three or more machines, any two nodes can fail without loss of data or incurring downtime.  This will solve the issue of backing up non-Unix VMs and the several hours of downtime needed to restore a backup to another system.

Virtualization is the future of computing, where we depend on having all our data available all the time, or need multiple systems but have desktop space for only one.  There are some performance and technical issues, such as enabling audio and accessing optical and flash drives, but using local virtualization like VirtualBox and VMware appliances on a workstation helps solve that problem, as long as the VMs get regular snapshots or are shut down during system backup times.

Uncategorized

Multi-modal Transport: Re-thinking Bicycle Season

As we reach Memorial Day weekend, the traditional start of summer activities, we are reminded that we so far have failed to get a start on bicycle season 2012.  Not that we haven’t planned: it just hasn’t materialized.  After we returned from Florida, early in December, I repaired the broken spoke on the Green Machine and applied anti-corrosion treatment to the parts savaged by the Florida mangrove swamps.  But, the bike sat, unridden, all winter and into the spring.  Recently, we realized we weren’t going to be “competitive” in the touring category this season, so we canceled a tour planned for Colorado in August, forfeiting our deposit in the process.  But, we can’t not ride at all: we just need to re-evaluate our goals and plan accordingly.

We elected not to take our bike with us to Alberta in early May, as it was still snowy there. Oh, there were bicyclists out the weekend we arrived, but the weather, predictably, turned cold and wet.  We did get in a few short hikes between snow and rain showers, though.  On our return, the fact we were much belated in our spring training was realized with the arrival of our first WarmShowers bike tourist.  Sarah, from Seattle, was starting on a fast trip down the coast to southern California.  We convinced her to take the quieter backroads to the coast instead of the busy highways, and offered to guide her part way.  We elected not to make a test run with the repaired but untried Green Machine, so I dusted off “Rocky,” my trusty old Specialized Hard Rock commuter machine, and accompanied her as far as Cloquallum, a bit more than halfway to Elma and more well-traveled roads, turning around at the Bucks Prairie store for a 27-mile inaugural spring ride.

Tour-season sendoff at the Bucks Prairie Store, Cloquallum

The next day, we did get the Green Machine out for a shakedown test on our favorite local short loop, a 10-mile rolling ride through Shelton Valley, dodging the inevitable loose dog, but otherwise running fairly smoothly until reaching the furthest point from home, when our SRAM X7 rear derailleur suddenly ate the internal springs, leaving us with no chain tension. We stopped and manually shifted into the second-largest freewheel sprocket, giving enough tension to stay in gear and use the three hub gears to get us home.

When a good derailleur goes bad

The good folks at Green Gear Cycling, makers of Bike Friday, promptly sent out a new RD, though there were literally minutes left on the warranty. Meanwhile, our second long-distance cyclist of the season arrived, Lang, from Korea, via Alaska and an over-winter stay in Vancouver. He was headed south to Los Angeles, and wanted to pass through Olympia rather than take the rural route through Elma, so I once again cranked up the commuter to lead the way out to Highway 3 South, a twisty maze of dips and climbs through our neighborhood to avoid the drop to sea level and heavy traffic through town.

Lang, continuing his trip from Alaska to Los Angeles

The new RD arrived and I installed it before we were due to depart for Bend, Oregon on a short outing with friends, but no time to test drive the new unit. The weather forecast called for rain, so we elected not to dismantle the bike and left it behind. However, we had been discussing the need to be able to take the Green Machine to an alternate starting point without the ordeal of packing and unpacking the machine when we were not taking public transit or traveling multiple days. Or, since we had been through a bikeless period waiting on parts, a way to transport our old Santana tandem, “Leviathan,” to either ride it on rougher trails or deliver it to a shop or prospective buyer, should we try to sell it again. At this point, it seems prudent to keep it, since the load capacity of the packable Bike Friday is considerably less, making it less suitable for touring on roads where the two-wheel trailer is not practical or it is subject to greater stress.

Front fork secured, ready to load the bike by swinging it by the rear rack onto the mid-tube cradle.

For a number of years, we had been weary of the need to clean and jerk the 60-pound Santana overhead and then maneuver it no-handed onto the front fork clamp and rear bottom bracket support of the 24-year-old Yakima tandem rack, which did not fit our new Jeep, in order to transport it anywhere. We had decided that, should the opportunity arise, we would replace that rack system with a swing-out system to allow the bike to be attached one-half at a time, for easier loading. Since we had decided to provide a means of transporting the Bike Friday more or less intact, and REI had one of the side-loading types on sale, we ordered one before leaving for Oregon, along with a new set of rack clamps to attach to the factory roof rails, to replace the old rain-gutter clamps.

The Green Machine, racked for transport with minimal disassembly

The intention was to re-use the 18-year-old crossbars from the old rack, purchased for the old 1994 Jeep we replaced last year. However, the protective end caps on one of the bars had been somewhat abused over the years, and 18 years of Pacific Northwest and Montana weather and 286,000 miles of highway had taken their toll. I had to saw off the rusted ends of the bar, and force the old clamps off one end, over the bulging plastic coating. The cut ends revealed solid metal with just surface rust under the outer coating, and the bars were more than long enough for the new Jeep, so it went together without too much incident, and appears to be sound. Of course, we now drive with that characteristic buffeting drone of the slipstream tumbling over the round bars, that we have missed for the past year and a half.

What we have learned from our own recent touring experience and from our long-distance touring guests is this: it is all about the journey, the places and people you encounter, not about riding X miles per day or getting there entirely on your wheels. This is why we bought the Bike Friday–to have the flexibility of one-way bike trips, starting and ending anywhere accessible by public transportation. Re-instating the roof-top car carrier system also means we have the freedom of making short loop trips easily without starting from home or having to take the time to pack and unpack the bicycle, like driving to the children’s house in Olympia and riding the rail trails that pass by the end of their street, or engaging in out-and-back trips from base camps, then moving on the next day.

Travel, Uncategorized

Seasonal Affect

Upper Kananaskis Lake

No seasonal affective disorders here, just a brief return to winter.  On this last day of April, a quiet Monday, when most of the skiers have gone home and the golfers, mountain bikers, hikers, and fly fishers have not yet arrived, we took advantage of the solitude to venture up into the Kananaskis Country, south of Canmore, climbing to an elevation of 1724 meters (5600 ft), where winter was in full force yet at the Kananaskis Lakes, which won’t begin to fill with snow melt for many weeks yet.

Lower Kananaskis Lake

We chose not to bring our bicycle on this trip, figuring the bike trails would not yet be open. Well, we were partly right. Although there is an extensive trail system throughout Kananaskis Country, some trails are closed to bicycles, as seen here, and some are just closed, anyway, along with many roads, services, and all the campgrounds.

No Bikes Allowed on this trail, even if you can find it...

We had set out for the day up Alberta Highway 40, which was still closed for the winter about 50Km up the valley. After a short hike for views of the lakes, we headed back, following the GPS, which pointed out the shortest route back was via the Spray Lakes, most of which was unpaved, but, for the first 50 Km or so, was very wide and relatively smooth and empty of traffic.

Spray Trail, along the Spray Lakes Reservoir

The scenery was great, even though visibility was low.

Spray Lakes Reservoir

Despite having dropped quite a bit in altitude, the lakes were still frozen, and we were getting quite close to town. At the dam at the head of the main reservoir, the road narrowed and deteriorated, and, more ominously, quit descending, until we came to Whitemans Pond, perched on the edge of a cliff, 400 meters (1300 ft) directly above our resort. The narrow gravel road turned sharply left across the face of the cliff and descended at a double-digit grade. Suffice it to say that, had we chosen this route outbound to the high country, we would not have attempted the journey. As it was, we were within 5Km of town and no turn-arounds, so we plummeted down the slope, transmission screaming in low gear (but, to her credit, the Nice Person, white-knuckled in the infelicitous downward-view seat, was not–screaming, that is).

Spray Trail (diagonal cut above the trees in the forground)

Meanwhile, between adventures in the wilderness, work goes on for the Unix Curmudgeon. Email correspondence tended, files uploaded, advice rendered, and life goes on. Hmmm, wonder where to go tomorrow on coffee break?

Musings on Unix, Bicycling, Quilting, Weaving, Old Houses, and other diversions

%d bloggers like this: