*not a real phone number
The English language has been characterized as one that not so much borrows from other languages, but outright steals, remakes, and repurposes words to suit one or many purposes. So, the “Drivers Wanted” signs on the back of 18-wheeler trailers has different meaning to Unix system administrators. But, just as the long-haul trucker operates the tractor that gets your load hauled to where it is needed, a Unix device driver is a piece of software that gets your data to and from an external device, like a hard disk, dvd, printer.
Recently, Chaos Central experienced several close encounters of the frustrating kind with device drivers. Unix device drivers have always been a challenge, but the growth of open source software has made them even more so. Device drivers have traditionally been provided by the manufacturers of computer equipment, either through iron-clad non-compete agreements with commerical Unix vendors (like Oracle and Apple) for source code or operational specification for the device or in binary form in the case of Windows computers. The problem arises with Linux: in order to be distributed with the GNU open-source licensing, all components must be open source and freely redistributable, and many device drivers are not. At best, drivers come as object files and header files that can be linked with the kernel version at the current patch level.
Device manufacturers develop and deliver driver software in binary form for Microsoft Windows, because they sell most of their products for use with computers running Windows. However, since the hardware is proprietary and the software required to interface with it reveals much of the inner workings and design, vendors are reluctant to release the source code to Linux developers. Also, because the Linux market is relatively small, they are also reluctant to produce binary versions compatible with Linux. In cases where there are binary drivers for Linux, the drivers are specific to kernel revision. Since the Linux kernel is frequently patched and has at least a minor revision release every six months, keeping a complete configuration inventory is problematic, even for systems that are supported, which brings us to the first pothole in the information highway… And then, there is the problem of keeping drivers current with firmware changes in the hardware. The devices themselves are often programmable circuitry, with flash memory. Changes to firmware usually also require changes in the host driver.
Fiber Optic/Multipath Disk Drivers
It is no secret by now that the majority of Internet servers and almost all high-performance computing systems run Linux. High-end server hardware is most often attached to high-end storage systems as well, most of which achieve high throughput by using fiber optic connections. So, most fiber channel Host Bus Adapters do have device driver support for Linux, either available directly from the vendor or, in the case of supported Linux distributions, as part of a paid service package from the Linux provider. But, these choices also present what is basically a fork in the configuration management of the drivers: for instance, the Red Hat version does not always match the QLogic version. The reason for this is complex, but usually the hardware vendor wants to keep the drivers and firmware as up-to-date as possible to correct problems and support newer platforms and storage devices. The operating system vendor wants to keep older systems running smoothly, as well as supporting newer systems. Both need to balance the cost of maintenance against the number of installed systems. Inevitably, combinations of hardware and software exist that are not supportable, or that require some additional work by the system administrator to maintain a working system.
Thus, it came to pass that, during a routine patch-level upgrade to bring a pair of systems to the same Red Hat 5.8 level, a warning popped out on the console, announcing components of the fiber-optic driver were missing. Fortunately, Linux systems are built in such a way that the system software libraries maintain compatibility with all patch levels of a particular kernel release: Red Hat does not update the kernel release through a major distribution release, so it is possible to update the system software without updating the kernel patch level, so that the drivers do not need to be rebuilt. It was a simple matter of editing the grub.conf file to reboot using the old kernel version (which is why Linux retains the older kernels in the Grub menu). Even though most device drivers are now loadable at run time, if they are needed during the boot process, they must be integrated into the boot image. In the case of these machines, the boot disk appears to be local, but the driver is needed to mount the data drives for the server application. Being able to build a boot image with the driver is usually a pretty good indication that the driver will load when needed. Also, many high-end systems use multiple paths between the host bus adapter and the storage array, for both fail-over and to increase throughput. Multipathing needs to know what devices support it, so even if the system will boot without the device driver, multipathing may not work. In most Linux systems, the device mapping will differ between multipath and non-multipath configurations, so the correct devices will not be available if the drivers are loaded in the wrong order.
But, it is desirable to have the kernel patches installed as well, so the problem of building a boot image with the driver installed had to be solved. There was a utility package for rebuilding the current image, but that presupposes that the system would actually reboot without the driver installed. A bit more research turned up a manual procedure for building a boot image from any kernel, which appeared to work. To complicate issues, this upgrade was for a remote client, so standing at the console to select a fall-back kernel in case the reboot was unsuccessful was not an option. In these cases, it is necessary to have someone standing by at the physical machine to intervene if necessary during the reboot. So, final resolution is awaiting the next maintenance availability for the system in question.
Meanwhile, in our own office at Chaos Central, we have been struggling with printing problems for months. Our aging Xerox color laser printer has been printing slowly or simply hanging in mid-print for some time. Canceling the job, then turning the printer off for 30 seconds to clear the queue and back on again worked, sometimes. And, when the printer went into “Power Save” mode, it wouldn’t wake up when we sent it a print job. We had recently replaced the expensive print cartridges, so the budget wouldn’t support a new printer right now. But, the symptoms seemed to indicate a driver problem more than anything else, since the problems seemed to crop up with a recent system upgrade.
Printing from Unix has always been an issue. From early on, Postscript has been the de facto common printer language, so we were faced with buying Postscript-capable printers in the 1990s. With the advent of CUPS, the Common Unix Printing System, it is possible to use almost any printer for which a CUPS PPD file is available, and to use job control features in various Postscript printers. Again, vendors defer to Windows for their proprietary print drivers, and send the Linux user elsewhere for Linux “drivers.” However, all PPD files are not created equal, and they change from time to time.
Before throwing out the old printer, we decided to try bypassing CUPS entirely. Most network printers support multiple printing protocols, one of which is FTP, the venerable File Transfer Protocol, used since the days of dial-up modems, but deprecated in the Internet age due to lack of encrypted security measures. Normally, the Network Police (i.e., the network security administrators) insist that this feature be turned off. But, with it turned on, it is a simple matter of uploading a Postscript file to the printer, using anonymous FTP. Which we did; and, voila!, it printed.
So, a search for a new printer driver ensued. We did find one at the Xerox site, with a PPD file several times larger than the generic one we were using that came with the latest CUPS package. And, the printer now wakes up from a sound sleep and prints! Yes, it is very slow yet, but it does print. The slow printing seems to be associated with different pieces of software that render the print selection to Postscript, but we can at least produce output more or less on demand.
It has been some time now since we’ve had network problems, but getting connected “on the road” was a major problem until recently, as Linux drivers were not available for a lot of wireless cards bundled with laptops. The choices for a while were to run a utility that mapped the system calls in Windows drivers to Linux systems calls, which had variable results. Fortunately, many vendors now provide support for Linux and it is possible to buy laptops now made to run Linux. But, as recently as 2011, using our old (ex-Windows) systems, we had a problem with the connecting layer between the driver and the I/O that reset the network connection every couple of minutes in an attempt to keep connected with the strongest signal. In hotels or congested areas where there were multiple access points broadcasting on the same channels, this effectively precluded any kind of meaningful network experience and certainly made persistent connections (like file transfers or remote administration) impossible. This issue seems to have been corrected in later versions of the wireless software, but there is always the next surprise waiting in the world of Linux device drivers.