Category Archives: All things Unix

Tinkers and Scriveners, 2015

Tinkering is, historically, the repair of tinware, usually by itinerant craftsmen.  Modern usage extends the term to any kind of more or less informal modification or adaptation of something.  If the object is intended to be useful or repurposed, we say it has been “tinkered with.”  If the purpose is to make an art statement, we resort to French, and call it “bricolage.”

A scrivener is traditionally a transcriptionist, someone who writes down what an illiterate person needs to have in print for legal or financial purposes.  Scriveners are still fairly common in Third-World countries, and in the First World, we use professionals to help us with legal and other formal paperwork.   In the modern age, reading and writing is more or less a universal skill, but computer coding is not, so we commonly also employ a scrivener to write down what we want a computer to do.   I think the term “scrivener” is more descriptive than “programmer.” As someone who revises more code than writing original code, we often use the combination, “Tinkers and Scriveners” as our informal motto.

In our last episode, we described a “just for fun” practice, describing (a word derived from “scribe”)  to our computers how to take pictures of our driveway, turn them into a timelapse video, and make them available on the Web.  To extend and refine this concept, we tinkered with the scripts this week, to automate the process and add an audio track, plus fix a few issues.

Try it out:  http://www.parkins.org/webcam

As part of the allure of having a video surveillance system, we had included a weather report along with the “real time” view.  The process is simple:  “Go get the weather report, take a picture, and display the picture, time, and current weather report in a web page.”   That worked fine when it was just me running it.  But, as soon as I posted the URL (Uniform Resource Locater, or web page name) to Facebook, that World-wide Eater of Time, I immediately got a “nastygram” from the weather service informing me that I had abused my (free) automated query privilege by asking for many weather reports all at once.   I don’t have all that many “fans” eagerly awaiting my next post, so I presume it was the WordPress.org  and Facebook robots burrowing into the link to see where it went–many, many times.  The search engines do this to score your site/post and the spammers also are looking for paths that lead to posts with open comment access.

So, the first order of business was to rethink the weather report.  The service only updates their reports a few times an hour, so there is no need to ask for a fresh report every time the page is accessed.  So, we told the computer, “Remember the weather report, and only get a new one if the one you have is more than 10 minutes old.”  We don’t expect a lot of traffic, but this ensures at least 10 minutes between weather updates, no matter how many visitors we get at once.

Here’s what that revision looks like, in patch form:

9a10
> import os.path
37,39c38,46
< 
< f = urllib2.urlopen('http://api.wunderground.com/api/...key.../geolookup/conditions/q/WA/Shelton.json')
< json_string = f.read()
---
> stale = time.time() - 600
> if ( os.path.getmtime('weather.dat') < stale ):
> f = urllib2.urlopen('http://api.wunderground.com/api/...key.../geolookup/conditions/q/WA/Shelton.json')
> json_string = f.read()
> w = open('weather.dat','w')
> w.write(json_string)
> w.close()
> w = open('weather.dat','r')
> json_string = w.read()

A minor change, but one that prevents breaking the application programming interface (API) agreement with the weather site. We can’t control how many hits we get on our web site, but we can control how often we hit theirs.

Since we are now taking a picture every 15 seconds, and the process runs all day, taking exclusive control over the camera hardware, we no longer can take snapshots in “real time” like we did originally. However, 15 seconds isn’t a long time, so we simply copy the picture into the “current” view every time we take one, and skip the “snap the shutter” part of the weather report page.  That part we had in the last post.  What we did this time, though, was add some code to update the time displayed on the screen and refresh the picture every 15 seconds.  This requires some browser code, in Javascript,  We also added some code to show when the weather report was last updated by the weather service.  The patch file looks something like this (we also cleaned up some other code that is no longer necessary):

28c30
< <h1>The view from Chaos Central %s</h1>
---
> <h1>The view from Chaos Central <span id="nowTime">%s</span></h1>

47a48
> updated = parsed_json['current_observation']['observation_time']
49,50c50,72
< print "<img src=\"/images/webcam.jpg\" />"
< print "<p>The current temperature in %s is: %s F / %s C<br />" % (location, temp_f, temp_c)
---
> print """
> <table><tr>
> <td>
> <img src="/images/current.png" id="currentImage"/>
> <script language="Javascript">
> setInterval(function() {
> var currentImageElement = document.getElementById('currentImage');
> currentImageElement.src = '/images/current.png?rand=' + Math.random();
> var now = new Date();
> var h = now.getHours();
> var m = now.getMinutes();
> var s = now.getSeconds();
> formatnum = function( num ) {
> if ( num < 10 ) { return '0' + num; }
> else { return '' + num; }
> }
> var time = formatnum(h) + ':' + formatnum(m) + ':' + formatnum(s);
> var nowTimeElement = document.getElementById("nowTime");
> nowTimeElement.innerHTML = time;
> }, 15000);
> </script>
> """
> print "<p>The temperature in %s is: %s F / %s C<br />" % (location, temp_f, temp_c)
52a75,76
> print "Weather Data %s<br />" % updated 
> print "Reload page to update weather: image updates automatically every 15 seconds<br />"
54a79,91
> print """
> </td><td>
> <ul>
> <li><a href="/timelapse0.html">Timelapse video of Monday</a></li>
> <li><a href="/timelapse1.html">Timelapse video of Tuesday</a></li>
> <li><a href="/timelapse2.html">Timelapse video of Wednesday</a></li>
> <li><a href="/timelapse3.html">Timelapse video of Thursday</a></li>
> <li><a href="/timelapse4.html">Timelapse video of Friday</a></li>
> <li><a href="/timelapse5.html">Timelapse video of Saturday</a></li>
> <li><a href="/timelapse6.html">Timelapse video of Sunday</a></li>
> </ul>
> </td></tr></table>
> """

Those of you familiar with HTML and web page layout will also notice that we converted the page into a table and added a column with a list of links to the timelapse videos, which weren’t really visible before, since we were testing them.

Now, to make all this useful and automated, we told the computers (plural, remember–the video production takes place in another, bigger machine) to make a new video every hour on the hour and add background music to the video.  For the first part, making a video of the images captured so far, we simply add the makevideo.sh script to a cron job.  “cron” is a daemon in the Unix/Linux system that runs programs at specified times, periodically. (A daemon is not to be confused with ‘demon.’  A demon is an evil force, a daemon is a helper or servant –the term comes from the Greek, adopted by the geek culture in the 1970s.)  Because our little Raspberry Pi runs on Universal Time (UTC, what used to be called Greenwich Mean Time), and daylight on the North American West Coast, eight time zones later, spans two days under UTC,  we have to put in two jobs for each time lapse image set, one for the morning and early afternoon, and one for late afternoon and evening:

0 16,17,18,19,20,21,22,23 * * 0 /home/larye/TIMELAPSE/makevideo.sh 6
0 0,1,2,3 * * 1 /home/larye/TIMELAPSE/makevideo.sh 6
0 16,17,18,19,20,21,22,23 * * 1 /home/larye/TIMELAPSE/makevideo.sh 0
0 0,1,2,3 * * 2 /home/larye/TIMELAPSE/makevideo.sh 0
0 16,17,18,19,20,21,22,23 * * 2 /home/larye/TIMELAPSE/makevideo.sh 1
0 0,1,2,3 * * 3 /home/larye/TIMELAPSE/makevideo.sh 1
0 16,17,18,19,20,21,22,23 * * 3 /home/larye/TIMELAPSE/makevideo.sh 2
0 0,1,2,3 * * 4 /home/larye/TIMELAPSE/makevideo.sh 2
0 16,17,18,19,20,21,22,23 * * 4 /home/larye/TIMELAPSE/makevideo.sh 3
0 0,1,2,3 * * 5 /home/larye/TIMELAPSE/makevideo.sh 3
0 16,17,18,19,20,21,22,23 * * 5 /home/larye/TIMELAPSE/makevideo.sh 4
0 0,1,2,3 * * 6 /home/larye/TIMELAPSE/makevideo.sh 4
0 16,17,18,19,20,21,22,23 * * 6 /home/larye/TIMELAPSE/makevideo.sh 5
0 0,1,2,3 * * 0 /home/larye/TIMELAPSE/makevideo.sh 5

Now, the web server owns the scripts that generate the web page and take the photos, but we share the folder on the external disk drive that holds the videos and I own the video building scripts on the CentOS server, so I run this set of  cron jobs under my own user account.  The timelapse show for the current day starts at 8:00am PST (1600 UTC), and runs until well after dark ( i.e., 0300 UTC, or 7:00pm PST).  As the days grow longer, if I am still running this experiment, I will need to extend the times, or use the astronomical data to control whether or not the video needs to be generated.  At the current display rate, the timelapse video covers an hour every 30 seconds.  In mid-winter, daylight lasts less than 8 hours at this latitude, but nearly 18 hours at the summer solstice, so the video will stretch out from the current 4-minute running time to nearly 10 minutes.  The camera program starts at 4:00am and waits for daylight to start recording, shutting off automatically at dark.  The programs are written so that it would be trivially easy to change the frame rate to keep the length of the video shorter by advancing the time scale faster later in the day, as the days get longer.

The last bit of code added was for an audio track.  For this, I told the computer, “Find a song in the music library that is about the same length as the video, and add a sound track using it.”  I have enough tunes of varying length in the library so that there is a different one every hour, and hopefully enough in between so we get different ones each week as the days get longer.

Part of this exercise was to get some practice writing in the Python language.  However, the song-picker, I wrote in my old standby, Perl, just to get it done quickly.

#!/usr/bin/perl -w

%list = (
 108 => "Sintel/Jan_Morgenstern_-_01_-_Snow_Fight.mp3",
.
.  # lots more songs here, deleted for clarity...
.
 225 => "Kai_Engel/Kai_Engel_-_09_-_Embracing_The_Sunrise.mp3");

$frames = $ARGV[0];
$rate = $ARGV[1];
$runtime = $frames / $rate;

foreach $key (sort { $a <=> $b } keys %list) {
 print STDERR "check: " . $key . " against " . $runtime . "\n";
 if ( $key >= $runtime ) {
 print $list{$key};
 exit();
 }
}
print $list{$key};

This program is called by the shell script that runs the video:

#!/bin/bash

cd ~/TIMELAPSE
rate=8
frames=`ls ./images/${1}/img*png | wc -l`
# find a music track that is about the same length as the video
song=`./songs.pl $frames $rate`
rm surveil${1}.mp4
~/bin/ffmpeg -framerate $rate -i ./images/${1}/img%04d.png -i ./music/${song}
-c:a aac -strict experimental -strict -2 -r 30 -pix_fmt yuv420p surveil${1}.mp4
echo $song > surveil${1}.txt

This also records the name of the artist and the song (for the credits) in a file, the contents of which are displayed by the Javascript code on the web page that plays the video clip.  This is still crude, using a frame.  The other part of this learning exercise is to hone my Javascript skills and learn to use AJAX (Asynchronous Javascript And XML–actually more often JSON rather than XML)  to pull data from the server and format it so the page is well-formatted and updates seamlessly.  The time updates on the “real-time” page also needs work to format it the same as the original time statement.  One oddity in that is that the initial timestamp is the server time, but the updates are for the timezone of the browser, so the hour will change if you are in a different timezone.  This violates the “least surprise” principle of good web and program design.

Finally, here is one of the video display pages, showing the mechanism to update the current video file and song title.  I should rewrite this as a Python script, so it accepts the index of the video and creates the proper page, instead of repeating the HTML code seven times and having to edit all of them for each code change, which leads to errors and inconsistency.  The random number function added on to the file URL in the Javascript code is a hack to make the browser reload the file–if the URL doesn’t change, the browser will use the cached (old) version of the file, so that the video and song title (and photo in the ‘real-time’ page)  wouldn’t update until the cache expired (usually a couple of weeks–not very effective for an hourly or 15-second update).  That’s always been an issue when fixing customer web sites–they don’t always see the change unless they clear their browser cache.  If they are working through a proxy server, they don’t have control over the proxy caching.  A cache is  a neat way to speed up the Internet for sites you visit a lot, but it can be annoying if the browser or proxy doesn’t check the file modification time against your cached copy.  Inaccurate time-keeping on the user end can also mess up cache coherence and browser cookie management.   Fortunately, most computers now get their time updates from the Internet automatically, so this is less of a problem than it was in the old modem/dial-up days when we couldn’t afford the overhead of a lot of background processing over the link.

<html>
<head><title>Webcam Timelapse</title></head>
<body>
<h2>Sunday</h2>
<table width="500px">
<tr><td align="left">
<a href="timelapse5.html">Saturday</a>
</td><td align="center">
<a href="/cgi-bin/webcam.cgi">Home</a>
</td><td align=right>
<a href="timelapse0.html">Monday</a>
</td></tr></table>
<video width="320" height="240" controls>
<source src="/TIMELAPSE/videos/surveil6.mp4" type="video/mp4" id="vidFile6">
</video>
<p>Music track: <iframe id="audFile6" src="/TIMELAPSE/videos/surveil6.txt" height="55px" width="300px" onload="upDate()"></iframe></p>
<script language="Javascript">
function upDate() {
 var currentVidFile6 = document.getElementById('vidFile6');
 currentVidFile6.src = '/TIMELAPSE/videos/surveil6.mp4?rand=' + Math.random();
}
</script>
</body>
</html>

So, there we have it: a project that is realized in a variety of programming languages and data formats: Python, Bash (Bourne-Again SHell), Perl (Practical Extraction and Report Language or Perniciously Eclectic Rubbish Lister or just short for ‘Pearl,’ depending who you ask), HTML5 (HyperText Markup Language version 5), and JSON (JavaScript Object Notation), MPEG 4 video and MPEG 3 audio, H264 video codec and AAC audio coding, and runs on several different Linux distributions, Raspian (Debian 7 on Atom CPU) and CentOS7 (Red Hat unlicensed version on 64-bit Intel CPU, as a virtual machine running on a CentOS6 host), using the Apache2 web server software configured to use CGI (Computer Gateway Interface, not Computer Generated Images, as used in the movies) and directories aliased on a shared disk drive. We use Secure Shell (SSH) with a password-caching agent to transfer files between the web server and the video processor, and a API (Application Programming Interface) to retrieve weather and astronomical data from a remote web server. We also use a script that queries an external “whats-my-address” server to track the variable IP address of the web server and link to it by a redirection statement on our public web server, routing the request through a firewall rule and port mapping to avoid a lot of hacker traffic on the public HTTP (HyperText Transport Protocol) port (80). All of the software used is freely available, shared for the benefit of all who use it and contribute to the software community: no proprietary, hidden code, no expensive training classes, just relatively inexpensive books or free on-line documentation and how-to-do-it forums. This is literacy in the 21st century–the ability to express yourself through ubiquitous and free technology, with the freedom to tinker–if you don’t like the way the software works, change it, or write something completely different. Expand your mind. Share your code and write about your experience.

Vacation as Mental Recreation

Vacation:

1. n. an extended period of recreation, especially one spent away from home or in traveling.

2. n. the action of leaving something one previously occupied.

The second definition just means “leaving,” without specific reference to reason or purpose…  As to the first, there are many different forms of recreation, which may or may not require extended travel.

Somehow, during the early 1990s, we talked ourselves into the time share mode of vacationing.  The scheme works like this:  you find a place you like to visit, with activities you like to do, and you buy a small share of a condominium at a resort near that location.  The purchase qualifies as a second home for tax purposes, so the interest bite isn’t quite so bad, and you own a tiny portion of [supposedly] prime real estate.  And, you get to use the property at designated times of the year, commensurate with the share you own.  Disregarding the purchase price, which is promoted as an “investment,” the cost of the vacation in an apartment with full kitchen, laundry, etc. and access to beach, tennis, pool, and proximity to other amenities such as golf and skiing (in season) is little more than or even less than a stay in a cramped and dark motel room nearby — provided, of course, that you use all the time allotted to you and that the property retains resale value.

Other than the fact that the amenities desirable to us, namely nice scenery and good places to ride our bicycle and hike, interesting shops, etc., make the venue less exclusive than the golfing, boating, skiing, etc., that attract most other resort guests,  our activities are exploratory, not conducive to repeat visits year after year, regardless of the season.

Not long after committing to this life-long enforced vacation plan, we found ourselves in jobs that

  1. didn’t have a lot of vacation time (as a troubleshooter and “cleaner” I tended, in the 1990s, to change jobs every year and a half, on average), and
  2. didn’t offer time off when we could schedule vacation, i.e., when not teaching night school, which was the other reason we became interested in the floating exchange system, besides the spirit of adventure: we could possibly coordinate our calendars, provided there was a vacancy in a place we wanted to visit at the time we desired.

The first reason made it difficult to use up the backlog of vacation credits, as we also needed time off to visit relatives, of which there were too many to have them visit us at the time share, and which option was impractical for a lot of reasons, such as wrong time of year for school vacations, travel cost, etc.

But, as often happened, I had jobs where I could telecommute from home, or, as it turned out, from anywhere with a phone connection (later, Internet connection).  Judy, with a more stable job, usually, had more vacation time accrued, but, due to commuting time, had little time for hobbies. Later, she was also self-employed, and vacation was a time to do her own sewing, weaving, and hand work, or portable projects for customers (we once pieced a guild raffle quilt at a time share condo, and were asked by the management to limit use of the sewing machine, as it vibrated the building).

We did have an option to get around the fixed vacation times, too:  for a few hundred dollars more a year, in membership fees and “exchange” fees, we could use our vacation credit to go to other comparable resorts instead of our own, if desired.  Which we did, for a number of years, a scheme that also allowed us to gain more vacation days by traveling exclusively in the “off season,” i.e., between the skiing and golfing/boating seasons, which are the best times for cycling and hiking, anyway.  The operative term being “comparable,” which means that all the other resorts have boating, skiing, and golfing as primary attractions, as well.

So, early on, we started using our resort time as a working vacation, hauling computers, sewing machines, looms, fax machines, and cell phones off to the resorts, spending part of the day working and the rest of the day exploring on foot or wheel.  This, then, has been our recreational plan for over 20 years.  And, the “24x7x365” work mentality has carried through even on non-resort outings, carrying on remote Internet work while traveling, from coffee shops, motels, and campgrounds, and even sometimes phone consultations pulled over at a freeway exchange or shopping center parking lot, or, on the bike, standing in the middle of a country road in the rain.

Now, in semi-retirement, with only a few clients in maintenance mode and some volunteer work, one would expect we would be free at last to have a “normal” vacation.  Well, old habits are hard to break.  On our most recent trip, back to the original time share condo that we actually own a piece of, we loaded the car with our computers and looms, knitting needles, and balls of yarn.  This time, though, the objective was, on the fiber side, perfecting new skills and making items for relatives, and the computer efforts aimed at also learning and perfecting new skills, and working on blog items.

Judy finishes tying on the warp, ready to start weaving on the small 1930s Structo Artcraft metal loom we recently acquired.
Judy finishes tying on the warp, ready to start weaving on the small 1930s Structo Artcraft metal loom we recently acquired.

This past week’s effort was aimed at setting up an experimental compute cluster, using the popular Raspberry Pi single-board tiny Linux computers.  We have a collection of them at home, pressed into service as Internet gateway, network services, and print server, respectively, but had acquired a couple more to experiment with home automation controls and sensors.  These, we wired up at the resort as a local area network, connected to the Internet via the resort WiFi, and using dnsmasq and IP forwarding to route our other computers to the Internet.  Setting up a compute cluster also involves sharing storage resources, so we were busy installing packages for the services needed, such as NFS (Network File System) for disk sharing and PostgreSQL for a database server, with the intention of building a distributed software application server that is extensible by adding more low-cost nodes to the system.  Fortunately, it was the slow season, with few other tenants in residence, so our bandwidth hogging went unnoticed, and the service was nearly as fast as at home.

pi_netDSCF0116
Two Raspberry Pi computers, plus power strip, Ethernet switch, WiFi dongle, USB hub, and smart phone comprise part of a makeshift local area network. Not shown: laptop computers connected to the switch and a USB hard drive added to the USB hub later. The extra cords are chargers for two iPads that complete the complement of “necessary” electronics.

This isn’t the first time we’ve wired up a router while on vacation — some resorts still charge for Internet, and only allow one or two devices on one account, so it is convenient to connect your own router and switch and run everyone’s personal devices off one login account.  In Canada, some hotels and resorts have wired Internet rather than WiFi, so having a router is the only way to share the connection among devices.  Almost all systems now have two network interfaces, ethernet and WiFi, so it is easy to make one device a router and connect the rest to it, either through the wired switch from one WiFi connection or reverse the flow (where there is a wired connection) to make the router a WiFi access point.

So it goes: you can take the system administrator out of the data center, but you can’t take the data center out of the sysadmin.  Because a lot of hotel WiFi systems have little or no security, we also use a web proxy server located in our home office–also on a Raspberry Pi and accessed through an encrypted “tunnel”–to browse sites that are also not secure.  This also hides our location from the Internet, so we don’t get a flood of local ads.  The home network gateway also provides access to a webcam to keep track of the house while we are gone.

As much as we take home with us just to have a different view out the window, the other side of time share vacationing is that we need to take time to visit relatives and time to explore and travel on our own, bicycle touring while we still can.  And, as retired folks, our income has declined (and, thanks to the 21st century economy and banking practices, so has our retirement nest egg:  we started out with nothing, and have very little of it left), leaving little discretionary income for unnecessary travel.  So, we’ve put the “fixed base” time share condo up for sale.  Never mind that it wasn’t a good investment: the current selling prices for our units are somewhat less than the real estate fees, and sales are slow, so we won’t get anything out of it, and after 20 years of declining value, our effective nightly cost has been much higher than we should be willing to pay, but we will cut the monthly maintenance fee expense.

We have a membership/owner share in another time share club that doesn’t involve ownership in a specific unit of a specific resort, but provides access to use any of their facilities, so we will still be able to enjoy condo vacationing (actually, obligated to go periodically, as the annual allocations expire within two years if not used).  The old type of time sharing a fixed location doesn’t quite work for us anymore, if it ever did, and it certainly doesn’t work for our children: none of them or even their extended families want to take on the responsibility, so we are letting it go–if it will sell. We had listed it once before, for two years, unsuccessfully, prior to converting it to the floating exchange program.  If it doesn’t sell this year, well,we will be back, toting bicycle, computers,looms, yarn, and whatever else we need to enjoy living at home away from home.

Yet Another Fruit-ful Computing Modality

Followers of the computing side of the Unix Curmudgeon blog will note that our 20-year dalliance with Linux has expanded from the server, workstation, and laptop incarnations to the appliance, namely Raspberry Pi, a tiny single-board card that runs Linux and uses an HDMI TV as a monitor. Of course, we’re familiar with the other fruity moniker, the Apple, and have even used Macs from time to time, since the advent of OS/X, the BSD-based operating environment introduced around the turn of the century:   use generally confined to command-line scripts in terminal windows, as opposed to the graphical desktop made popular with the original MacIntosh.

image

This summer, Judy–the Nice Person complement to the Unix Curmudgeon–who has patiently put up with the Linux-only network at Chaos Central until now–bought herself an iPad to replace the severely outclassed and underpowered Netbook road machine, which hadn’t fared well in the progressive upgrades over the years from Ubuntu 10.04 to Mint 17 (based on Ubuntu 14.04). Naturally, she has fallen in love with the tablet, the latest successor to “the computer for the rest of us.”

Meanwhile, the Curmudgeon has been making do between the Android phone and the Netbook during his limited-duty recovery from surgery earlier, akin to typing with boxing gloves while blindfolded.  So, the Nice Person, being a sentimental soul, designated a second iPad as a belated birthday present for the septuagenarian Curmudgeon.  OK, iOS, like Android, might have deep Unix roots, but, as “locked” appliances, aren’t multiuser and don’t have a command shell or root.  They also have “apps,” which aren’t “open,” though many of them are free (as in beer), and which, then aren’t customizable and it is less easy to roll your own custom apps.

So, the annoyances abound.  One of the first things of note is the way Apple deals with technologies they don’t like: they simply don’t support them.  Thus, none of the five dozen or so videos we’ve produced in the last couple of years will play on the iPad, with audio and/or video missing: we used the LAME MP3 coding for the audio track, and IOS only supports the newer Advanced Audio Coding format (OK, it’s been around for 17 years, but MP3 is still more common).  This in itself isn’t a huge problem, but it does mean re-generating all of the videos from project files, for which some of the source components have been moved, requiring hand-editing the text-based project files and searching for the missing component files.  Now, we do have some conversion utilities, but they inconveniently do not include the requisite audio coding.  The other issue is that we normally render videos at 25fps, even though the camera runs at 30, and Apple likes 30fps, period.  Fortunately, the video coding remains at H.264.  The real problem in all this was ferreting out the real cause of the problem, wading through the often less-than-helpful online help forums. The good news is that most other video software will accept the Apple set of protocols.

And so it goes: we have managed to find SSH apps that let us interact with the other *nix machines, with the exception of our bastion server, which requires host identification that we can’t generate or set on a rootless machine.  However, we can go through a third machine that is external to the network and registered with our server.

Some of the apps are just plain annoying or obtuse, with few clues as to how to get them to behave the way you expect ( no doubt some of the problem is unfamiliarity with the iOS gesture vocabulary that 4-year-olds seem to find intuitive).  As with all rapidly-changing technology today, the helpful hints found online only worked with the previous version of the system you just upgraded to.  But, yes, the slim tablet is much more portable, faster, brighter, and higher resolution than the old netbook, and the on-scene keyboard is much better than the tiny one on the phone… The apps put out by many of the sites we visit most make better use of the display than do the web browsers. I’d like to see some of the apps more generally available for Linux as well as the tablet OSes.

Oh, by the way, this post was composed, graphics and all, entirely on the iPad.  Who says old dogs can’t learn new tricks?

Spring Cleaning at Chaos Central

Once again, as winter draws to a close, it’s time to assess our systems, shovel out the office, and plan for the year ahead.  But, we seem to be a bit late, as everything catches up at once.  This has been one of those weeks/months where things fall apart when touched.

With tax time rapidly approaching, it’s that time of year to fire up the Windows XP virtual machine to run TurboTax.  Realizing that Microsoft is finally, in April of 2014, pulling the plug on the venerable platform first released in 2002, after three service packs, innumerable hotfixes, and an on-again, off-again sliding End-of-Life date, not to mention three successor (if not successful) systems.  Of course, many of us Unix professionals who don’t depend on Windows for our livelihood have nearly abandoned Microsoft altogether, but still are plagued with having to keep a working copy of Windows around “somewhere,” increasingly as a virtual desktop residing on a Linux or Apple workstation or server.

Vista was a complete failure: our copy–that come with “rover,” our 2007 Compaq laptop that has run Ubuntu Linux (and been updated at least every 2 years) since we unpacked it–spent its life unused but available as an alternate boot option until the hard drive failed and was replaced in 2012, with no hope (or desire) to revive the Windows installation. We had acquired a copy of Windows 7 with our HP Netbook ‘mini’ in 2010, which copy also lay dormant, minutes after unpacking it, taking up hard drive space until recently.  With the increasing dependence on Windows security (if there is such a thing) solutions for networking in the government, it looked like we might have to revive it, just to do business.  The first boot-up of Win7 in over three years took 3 days to complete, installing the updates, with multiple reboots.  Happily, in the meantime, the Unix/Linux support team, of which I am part, found a Linux solution to our immediate needs, so the NTFS partitions lapsed into dormancy once more–until the prospect of the demise of XP sent us shopping for an updated alternative for the Microsoft interoperability problem.  We do have some Windows applications that happily run under WINE, the Linux WINdows Emulator, but many others don’t.

Having migrated a friend’s home system from Apple (due to a serious case of narcolepsy in her Imac, a rare but aggravating problem that Apple seems to want to ignore) to a shiny new HP laptop, we had become painfully introduced to Windows 8, that ‘new idea’ from Microsoft that turns your desktop or laptop into a badly designed giant cell phone with no phone service.  A few minutes with that made me almost long for Vista.  But, realizing that Windows 7 seems to be Vista overlaid with the XP desktop, we decided the best option for the Chaos Central network would be to integrate Windows 7 into our stable of virtual machines, to be pulled up on demand, anywhere on the network it was needed.

Attempts to migrate the Windows 7 installation on the Netbook to a virtual appliance proved to be frustrating, as it appears to be difficult, if not impossible, to create a copy that doesn’t demand to see your genuine Microsoft license to boot up, which is difficult when you have a machine that doesn’t have an optical drive, and did not come with an install disk in the first place.  Thinking (wrongly, it turns out), that making a ‘recovery disk’ on a thumb drive would suffice, we proceeded to do so, which promptly destroyed the Ubuntu boot partition, rendering the machine totally useless, since that is also where the GRUB boot manager keeps the information on how to boot to all the systems, including Windows.  Erk.

So, time to reinstall Linux.  Having also been increasingly disenchanted with the Unity desktop (which turns your Linux desktop or laptop into a decently designed giant cell phone with no phone service), I decided to install Mint Linux (yet another Debian-based variant, similar to Ubuntu), with the XFCE desktop, a lightweight system that is annoyingly similar in appearance to the XP desktop, but nonetheless familiar and functional, as far as menu navigation goes, and with the ability to paste hot links all over your desktop instead of in a peek-a-boo toolbar with inscrutable icons instead of text labels.

Of course, once Grub was restored, we could boot to Windows, but, in process of trying to get around the Grub issue in order to export the Windows system, we ended up running the newly-created Windows Recovery disk, which restored the Windows 7 installation to Day 0 (no patches, no added software or files)–and promptly refused to boot again without the Genuine Microsoft Windows 7 installation disk, which we still don’t have, and for which the “Recovery Disk” is not a substitute, despite the fact that is the only thing we end users can create from the installed system we bought and paid for.  Have I mentioned lately how much I dislike Windows?

At this point, we are at the verge of simply continuing on with a static XP system, for as long as Quicken and Electric Quilt will support their applications on it.  Setting aside this issue for a while, having given up a chunk of our on-line and off-line storage to yet another unusable Microsoft product, we turned our focus back to the primary business of making and supporting Linux software, and the goal of organizing the accumulated piles of paperwork and other paraphernalia in the office.

Just then, my desktop workstation, ‘zara,’ which was recently converted to CentOS6 to be a bit more compatible with the customer development systems on the server, suddenly shut down in the middle of browsing the web.  That usually means overheating.  We had recently done a bit of mid-winter housecleaning on ‘strata,’ our big development laptop, which had been shutting down because of overheating, and also had thoroughly cleaned the interior of the virtualization server when we replaced the hard drives in it last month.  The laptop fan had stopped running, but a thorough cleaning and redressing the wire harness got it running again, solving that problem.

The desktop machine was another issue.  The CPU heat sink resembled the filter in the clothes dryer, and the fan was barely turning over.  After cleaning the heat sink fins of lint and giving a fan a few spins in an attempt to “loosen it up,” it still wouldn’t turn over more than a few turns.  So, we pulled it and peeled off the sticker over the shaft bearing. intending to re-lubricate the bearing.  Unfortunately, the bearing seal on this fan was plastic instead of rubber, and couldn’t be easily removed.

'armonk,' cannibalized only days after being removed from service
‘armonk,’ cannibalized only days after being removed from service

But, we had recently retired our Internet gateway server, ‘armonk,’ a 12-year-old IBM desktop machine running FreeBSD, replacing it with a Raspberry Pi, which is more than adequate for that use.  The IBM had been running smoothly, so I  went to the temporary holding area for dead computers (which has encroached on the studio area downstairs), popped the lid, and pulled the CPU fan.  It spun smoothly, so I bored out the mounting screw holes to countersink the shorter screws for the AMD CPU heat sink on zara, reassembled the system, and we’re back on the air again.

So, here we are, behind in our work, with the retired IBM now truly inoperative, no Windows 7 working copy, taxes undone, and the office and network still not completely reconfigured as we planned.  The fan fiasco occurred in the middle of replacing the printer table under the window with a storage cube system.  The old HP 1200 laser printer, which we haven’t used for several years, was finally taken off “standby” and retired along with the IBM server.  The storage cube now holds the ethernet switch, wireless router, Raspberry pi network servers and overflow books, in preparation for moving Judy’s desktop workstation, ‘giskard’ from her downstairs studio to the table formerly occupied by the IBM, and the old laptop, ‘rover,’ downstairs, to make space on the office table.

'zara' and 'rover,' with the network fixtures tucked away in the new storage unit, but clutter still in plain view...
‘zara’ and ‘rover,’ with the network fixtures tucked away in the new storage unit, but clutter still in plain view…

Oh, and the color laser printer suddenly decided it is out of cyan and yellow toner, printing everything with a magenta cast.  With receipts on the government contracts running behind (30-day due dates are simply ignored, by official policy) and a few months of lean billable hours behind us, the prospect of shelling out several hundred dollars for new toner cartridges just doesn’t fit the budget this month.  The ink-jet printer in Judy’s studio is out of black toner–we have a new cartridge, but waiting to move the workstation upstairs to service that one…

So it goes.  Spring cleaning continues: the goal here was to unclutter the office and upgrade the network services with fanless/solid-state low power devices that are reliable and recover automatically after power failures.  At the same time, it is hard to schedule a shutdown time to open the cases and blow out the accumulated dust that is the primary killer of computers, and to replenish supplies.  We don’t print a lot these days, and toner seems to have a finite shelf life that sometimes is longer than the useful life of the printer itself, so we tend not to order ahead.  A couple years of competitive bidding on contracts that trimmed upgrade/replacement budgets to the bone, plus the low ratio of billable to overhead time during contract turnover times means keeping vital systems running well past their reliable life, risking work disruption due to inevitable disk and fan failures.

'giskard's new home, next to the server.
‘giskard’s new home, next to the server.

On a final note, moving Judy’s old workstation upstairs seems to have killed the monitor, an old, early flat-panel model that had been cantankerous at best.  So, we’re up and running on our one remaining ancient and massive glass CRT monitor, until we can afford a new modern one to service both her workstation and the Dell virtualization server.  Right now, we have to switch the cable back and forth for the rare times we need a console on the big Dell server.  I did dig out an old KVM  switch (keyboard-video-mouse, not to be confused with Linux Kernel Virtual Machine, which is what we run on the server) to share one console between two machines, but the circuitry in that had failed either from lack of use or too much moving.   We’re due for another desktop machine, budget permitting, as ‘giskard,’ a Linux machine built from spare parts at least six years ago, isn’t modern enough or robust enough to run what we need, and the audio has never worked right, and we are way overdue for new monitors.

Happy Birthday World Wide Web

Today is the 25th anniversary of the birth of the World Wide Web, which is known today simply as “The Web,” or, even more generally, “The Internet.”  The “‘Net,” of course, is much more, encompassing email and communications other than HTML, but for most people, the Web is their main contact.

Tim Berner-Lee didn’t invent the Web out of thin air, though.   During the period of proliferation of personal computers in the early 1980s, the diversity of languages, applications, and data formats made it difficult to exchange data between different systems.  An early attempt to make a universally-readable data file was in the GEM (Graphical Environment Manager) system, which initially ran on top of CP/M, one of the first microcomputer operating systems.  GEM was based on research at the Xerox Palo Alto Research Center (PARC).

The “killer app” that resulted in GEM-like interfaces being ported to Apple and MS-DOS was Ventura Publisher, an early desktop-publishing software system that promoted WYSIWYG (What you see is what you get) editing.  The data files were plain text, “decorated” with markup language “tags” to identify elements like paragraphs, chapter and section headings, and all typographical marks that editors normally penciled in, hence “markup.”  The first standard in this was SGML, or Standard Generalized Markup Language, and style manuals, most notably the one from the Chicago Press, were published to promote standardized markup notation.   SGML really didn’t catch on, though, as popular word processors of the time, such as WordPerfect and Microsoft Word and desktop publishing software like Aldus Pagemaker (now owned by Adobe) retained their own binary document formats.  GEM and Ventura Publisher became buried in history with the demise of CP/M and the introduction of the GEM look-alike Apple Macintosh and the decidedly inferior Microsoft Windows graphical desktops that took center stage.

Meanwhile, in the Unix world, networking and inter-networking was the focus, along with a graphical desktop environment, the X Window System.  The X Window System was too cumbersome and tied to Unix to be used over the relatively slow inter-networks and with the now predominant IBM PC running MS-DOS and Microsoft Windows or the niche Apple environment.  Meanwhile, Berners-Lee was experimenting with a subset and extension of SGML called HyperText Markup Language (HTML).  Hypertext introduced not only markup for the appearance of rendered text, but the ability to cross-reference one section of text from another, even between documents.    Hyper-threaded text concepts were known at least since the 1940s, and used to create alternate endings or story lines in printed fiction, but computers made it possible to achieve smooth navigation between paths.

At the same time, networks were becoming mature, with the introduction of the Domain Name System and the TCP/IP network protocol in an internetworking scheme that provided unique names and addresses for every computer in the network, on a global scale.  Incorporation of DNS names and local paths in hypertext references made it possible to connect any document on any computer with references stored in different files on the same or any other computer in the network.  To implement these connections, a new network protocol, HyperText Transport Protocol (HTTP) was developed, and the World Wide Web was born in a flash of inspiration.

To complete the system required software that could speak HTTP and render HTML into formatted text and links on a computer screen, a server, and a client, which came to be known as a web browser.    The first web server was on a Next (Unix) machine, and early browsers were text-only, as the Internet was still largely based on dial-up modem access over ordinary phone lines.  But, with the increasing use of graphical displays, browsers also became graphics-capable, adding imaging and other visual effects to the HTML palette.

Today, 25 years later, most web servers are running some form of Unix or Unix-like operating system, mostly the open-source GNU/Linux system, though Microsoft Internet Information Service (IIS) runs many corporate internal services and public sites.  Browsers are now graphical, either Microsoft Internet Explorer or based on the Mozilla project, an open-source release of the original Netscape code derived from Mosaic, the first graphical browser.

The Web itself has evolved, with the addition of scripting capabilities on both the server and the client browser to create dynamic pages created at the moment, tailored to changing data, with the ability to refresh elements on a page as well as the whole page, and the ability to stream video and audio data as well as display text and images.   Indeed, HTML version 5 replaces many of the commonly-scripted constructs with simple markup tags that modern browsers know how to render.  The advent of social media sites to connect multiple people in real-time as well as  provide private messaging and real-time chat has largely replaced the spam-plagued email system for many people, and brought the promise of video-phone connections to reality.

The Web, in 25 years, has transformed society and our technology, nearly replacing newspapers, magazines, the telephone, and the television and video media player.  Even the personal computer has been transformed: the advent of software as a service (SAAS) means that users no longer have to purchase expensive applications to install on one computer, but can “rent” the use of an application on a distant server through the web browser, and rent storage space in “the cloud” that is available on any computer, even small hand-held devices like tablets or phones.  The web has also made possible the concept of wearable computers, such as Google Glass.  The World Wide Web not only covers the planet, but beyond (with live feed from the International Space Station and the Mars rovers), and has infused itself into our experience of reality.

I’ve evolved with the Web, starting with experimenting with SGML and HTML in the late 1980s, writing HTML in the mid-1990s, and in the late 1990s writing CGI (Common Gateway Interface) scripts in Perl, then server-side scripts in PHP,  progressing to Ruby CGI scripts and Javascript client-side scripting, some generated by the underlying server-side scripts to dynamically add images to slideshows.  Today, I still write CGI and server-side scripts, and create Cascading Style Sheets (CSS) for page layout and element design, but increasingly use content management systems like WordPress or CMSBuilder that make editing web sites no more difficult than writing an email or posting a status to Facebook.  Yet, I’d like to see the grandchildren and great-grandchildren–who have lived with the web either from the time they learned to read (for the older grandchildren)  or since they were born–learn to see “under the hood” and learn how the web works, so they can shape the future instead of being shaped by it.