The last few months, our articles have focused on our bicycle adventures, notably, the preparation for, launch of, and, ultimately, termination of our planned four-month expedition from Florida up the east coast.  We arrived home just less than two months after departing, and just in time to perform some much-needed maintenance on the Chaos Central computer network.

As the title of this blog indicates, we have, for the last 25 years or so, depended on Unix, Solaris, and Linux for both our livelihood and, of course, to operate our in-house network.  The majority of our systems run GNU/Linux, in various distributions: Ubuntu and Mint Linux on desktops and laptops, CentOS on the server and virtual machines, and Raspian on the collection of Raspberry Pi micro-machines (and the above CHIP nano-computer) that are rapidly becoming the backbone of the home network.

GNU/Linux is very stable: we have, in the past, run systems for up to two years without a reboot–and then only because we suffered a major power outage.  But, with a collection of systems, something is bound to go wrong.  First, less than a month after we left on our trip, a power surge that made it through the power conditioner/battery on the server took out the virtual machine that renders the timelapse videos from our driveway surveillance system.  We actually didn’t notice this until I went to review the current day’s timelapse progress and found the video was an hour out of date.  Ah, this was because I had programmed a failover plan into the system: the videos were now being rendered by the Raspberry Pi cluster in the basement, much, much more slowly on a 32-bit ARM single-core processor with 512 MB RAM than on the Intel Xeon quad-core processor with 8GB RAM in the virtual machine host.

The server rebooted without incident when we got home: it actually didn’t reboot when the power hit came, but had an error that locked up the processor, an unusual condition.  Had we not been headed for home at the time we discovered it, we could have instructed our house-sitter in how to cycle power and bring up the system.

Then, a couple of weeks after we returned, the surveillance system, which is also the remote login gateway, simply stopped, which would have been a show-stopper had we not been at home.0  We happened to have a spare Raspberry Pi, one that had seen duty as a print server and scanner server before we got a new WiFi-enabled printer/scanner.  It took a couple of hours to add the necessary software packages to run the camera and web server and configure the machine to perform all of the necessary duties of the old one, including limiting access to specific machines and login accounts, and we were back in business–for a while.  A few days later, the external disk drive that we use to store the camera output had an unrecoverable error.  The files affected could not be erased due to the error, but renaming the folder and creating a new one kept the system running until we could get a new disk and copy the rest of the files onto it.  The old disk had been re-purposed from use as a portable backup for travel, and is about six years old, so it’s time to replace it, anyway.

After taking care of the disk issues, I revisited the Raspberry Pi failure: it turned out that the SD card that the Pi uses as the internal system drive had simply expired of  natural causes.  SD flash memory chips have a finite lifetime, and can be rewritten only so many times before becoming useless.  The culprit here was the surveillance system software (which I wrote, so I only have myself to blame)–even though the camera photos, taken every 10 seconds, are written to the external hard drive, my program copied the latest one to the system disk, in the web space.  Every 10 seconds, 8 to 18 hours a day, for a year and a half. That’s about 2 million writes, all in the same location, in addition to logging system activity.   So, a simple fix to preserve the new system: put the web file on the external drive.

The discovery of the worn-out SD card meant that the old Raspberry Pi was still OK, it just needed a new system drive.  About this time, I replaced my 3-year-old Android phone with an iPhone.  I had installed an SD card in the old phone for photos, so I removed that, backed up the files, reformatted it, and built a new Raspian “Jessie” operating system on it (the rest run the older “Wheezy” version), and booted up the once-dead Pi.  Yeah!

This uses up nearly the last of the 8GB cards around the house, though I still have a 2GB card in an old Kodak camera that I use to document our Warm Showers bicycle visits.  We have a few 16GB cards yet:  the smallest cards on the general market (Costco, Best Buy, etc) are 32GB.  I have one 64GB card, installed in a GoPro camera, which required installing an additional set of packages to handle the exFAT (no, not skinny: it’s an acronym for EXtended File Allocation Table) file system when copying files to Linux systems.  New purchases tend to be the micro-SD footprint, since most new devices, plus phones and POV cameras, take those, and the older devices use adapters that are supplied (for now) in the package.  Speed is important for high-resolution cameras and video devices.  But, when cost is a factor, we still look for the lowest capacity and speed, as older devices have a size limit, and won’t operate with the new cards.  In the age of mass-production, the devices themselves become obsolete while still functional because the supply of suitable storage media dries up.

So it goes–it has been said that Linux is free, but only if your time is worth nothing.  It takes a lot of time to build a custom system, but the flexibility is enormous.  Each machine takes on a personality of its own, as it develops different capabilities, selecting from among the many different distributions available and the thousands of software packages downloadable for free in addition to the basic system.  Plus, the machines acquire a collection of custom scripts over time, that don’t exist anywhere else.  As a software and web developer, having instant and free access to database engines, web servers, and many different programming systems is priceless.

When I need to run several different software systems or distributions, I can use virtual machines, running all versions at the same time, on the same physical machine.  And, there are choices, with no buyer’s remorse penalties with free software. I’ve tried three different non-linear video editors, and stick with an older version of one (the new version isn’t compatible with the old project files…).  The stock desktop systems come with web browsers and office productivity software, and several different graphical desktop systems, which can be chosen at login time.

The latest addition to our Linux/Unix obsession is the CHIP computer, by Next Thing, which I pre-ordered for $8 back in January and which arrived direct from the factory (in China) a couple of days ago.  The CHIP is a bit smaller than the Rpi, with no HDMI (TV output), only one USB port, but a built-in 4GB flash drive, WiFi, and Bluetooth, which are all not included in the Rpi.  The CHIP is low-power, has a battery connector (3.7v rechargeable battery not included), and can be  programmed via the micro-USB power cable if connected to a laptop.  This device is more suitable to mobile (read: robotic) applications, as, like the Pi, also includes a number of digital/analog input/output circuits.  And, being a full-featured Linux computer, is more versatile than the Arduino micro-controller popular for hobby embedded applications.   Unlike tablets and phones, which are powerful miniature computers in their own right, and microcontroller-based devices like thermostats and security systems, these small experimenter’s devices are completely programmable and physically extensible, becoming whatever tool your imagination can envision.

So it goes: in our 21st-century cottage (built in the early 20th), computing devices are as ubiquitous as light bulbs, with Windows becoming as irrelevant and obsolete as incandescent lights.  But, “some assembly required” becomes “a lot of assembly, some compiling, and a bit of fabrication essential.”  And, you may have to write your own documentation, operations manual, and maintenance plan, as well as some software.