Monthly Archives: March 2010

All things Unix, Webcraft

Backups Matter

1 Comment

Having been a Unix systems administrator for about 20 years and “in the IT business” for more than twice that time, I have always been aware that that tedious task of backing up your data can pay off in the end.  As we say, there are two types of computer users:  those who have lost valuable data and those who will lose valuable data.  But, backups can help.

The other day, while between crises at Chaos Central, our home/office/workshop, I was browsing through the web statistics for our public sites and actually paid attention to the list of ‘404’ (File Not Found) errors.  Now, a lot of these are the usual and customary web site attacks that go on all the time, “bad guys” probing for security holes that a) don’t exist because we are not using Microsoft Internet Information Server, or b) have long since been plugged in Apache, so I more or less ignore those, along with the blind probes for informational pages to gather email data to feed SPAMmer’s address lists.  But, since I have been using Google’s Webmaster Tools to aid in Search Engine Optimization (SEO) for my clients,  I’ve paid a bit more attention to little things, like having a ‘robots.txt’ file even if I don’t have any pages I want to hide from the search spiders.  Another one is the ‘favicon.ico’ file:  you don’t need one, but browsers always ask, so it is nice to have one.  Besides, it adds a distinctive icon to the address bar, tab, or ‘favorites’ list, so I have been creating those tiny graphics and adding them to sites as well.

But, this time, I also noticed missing pages where there shouldn’t have been any.  Oh-oh.  Until sometime last year, we had a number of links from our public server pages to our ‘extranet’ server located in our home office and port-mapped to our external router.   Some issues with our Internet service at home and our impending move made it imperative to move these pages onto our public servers, which I more or less did, in bulk, to a virtual server, and changed all the links to point to the new location.  I had thought I had tested all of the links, but here we were, many months after the transition, with a broken link.  An entire page and all of its images, gone.

Meanwhile, the old extranet computer had not survived the household move.  After sitting in a cold, damp basement for a month or so last fall, it refused to start.  But, we still have the last set of backups!  So, I extracted the files from the backup archive and uploaded them to the web site.

For the record, we are, of course, a Unix shop, running mostly Solaris and Linux, so we use Amanda, the open-source backup system from the University of Maryland, and backup to virtual tape on cheap USB drives.  The drives are large enough to keep several weeks’ worth of backups, and we archive the last backup set from “retired” machines.  For machines like my Linux laptop, that aren’t necessarily on or even in the network 100% of the time, I use rsync, freezing a checkpoint now and then, since we do upgrade Ubuntu versions, and we keep a pre-upgrade backup as well as current ones.  Because the Amanda backup runs daily, but we do a lot of work during the day, we also keep a snapshot backup (using rsync) every couple of hours during the workday on our main server/workstation.  For our one seldom-used physical Windows machine, we simply copy our working directories onto a file share on one of the Unix systems.

So, there are many ways to do backups, but it is important to do at least one of them, whether or not it seems to be a pain–it will pay off.  Oh, yes, I am in the group of users who have lost data–in my case,  I archived data, then failed to make a second copy before the orginal archive failed.  And, just because I’ve lost data once, doesn’t mean I won’t again, backups or no backups.  But, keeping good and current backups does reduce the chances that it will be soon or extensive.

Bicycling, Travel

Parking Meters are Evil

Yesterday, my tandem bike stoker (and life partner of the past 25 years) and I headed to the Big City by automobile to  run a few errands and check out the bike-path/walking trails around Capitol Lake.

Since the day had started cold and threatened to turn wet, we parked at metered parking on the downtown side of the lake, rather than out on the parkway on the far side of the lake.  Standing in the cold breeze from the lake and looking at the rain-laden clouds sweeping in from the Coast, and, with the closed-for-repairs causeway less than 500 meters down the shore, I thumbed enough of our dwindling supply of nickels into the meter for what seemed to be a generous 40 minutes of brisk walking.

As we  moved into the lee side of the hill on which Washington’s Capitol campus sits, the wind didn’t seem so cold, and we noted a switchback trail leading to the Capitol, so we diverted.  In the 30 years since I first came to Washington State,  I had never been to the Capitol itself, so it seemed like a good idea at the time to check it out.  After reaching the top and strolling past the domed seat of government, we noted that we only had eight minutes to get back to the car before the meter expired.

It wasn’t far as the raven flies, but ravens don’t walk down switchbacks, and the eight minutes expired by the time we reached the lake shore, where we noted, in the distance, the flashing red light of the  Parking Enforcement vehicle as it moved down the street–away from our car.  One more nickel would have saved us:  the time stamp on the $15.00 parking infraction was three minutes after the meter expired.  Officer Lisa must have sat in front of the car waiting for the LCD to flash all zips.  The car next to us also had a ticket, so it was obviously a good place to just hang out and watch the meters tick.

Needless to say, the afternoon gloom intensified inside the car as well as out on the drive home.   Parking meters are evil: you can’t add more time unless you are physically there, and they don’t record how long the meter has been expired, or know if the same car has been in the spot for more than the 3-hour limit.  In some municipalities, the parking enforcement folks make more than one pass to check for deliberate violators.  Obviously, there is no “grace period” in Olympia, especially during the legislative session when parking space is at a premium.

So, the moral of this story is:  be generous with your supply of nickels at the meter–if you return on time, pass your good fortune on to the next car, or consider it a tip to the city for having a convenient spot for you to have parked in.  Or, bring your bike and park the car in a free lot outside the city core.  Next time, we will.

All things Unix

Why We Don’t Use Windows (if we can avoid it) #537

Some software vendors haven’t caught on yet, and still make applications that run under Windows only.  Unfortunately for us, some of those apps are “must have,” so we still have to have at least one working copy of Windows around.  Our current one is a refurbed off-lease IBM desktop we picked up a few years ago, running XP.  It’s been up and running lately, but doesn’t do much other than installing so-called security updates from time to time.

We had left it alone for a couple of weeks while busy with other projects.  Today,  I tickled it a bit to wake it up, and, when the “select user” screen appeared, clicked on my username.  The machine whirred a bit, then promptly announced I was being logged out.  OK, idiosyncratic behavior sometimes happens, so I try again.  Same thing.  Different user, same thing.  Hmm.  Maybe something got corrupted, boot to Safe Mode.  Same thing…  Can’t logon, even as administrator (it’s XP Home, so, no difference, really).  Oh-oh.  We did have some fairly important stuff on this machine.  Not good.

So, off to the Web on one of our trusty Linux boxes, symptoms indicate the C:\\WINDOWS\system32\userinit.exe file is, what?  Missing?  How can that happen?  This is a vital system file in the middle of one of the most holy of holies in the whole operating system.  But, I rummage through my files and drag out the Ubuntu 9.10 CD I used to upgrade my laptop a couple weeks ago (which had it’s own issues with wireless drivers, but nothing like casually losing an essential file after running for three or four years).

I boot the “live CD” version of Ubuntu 9.10, mount the Windows partition, and, sure enough, no userinit.exe file.  Everything else seems to be there, so I boot up a newly-installed  XP virtual machine running under VirtualBox on our 64-bit Linux box–to which I was planning to migrate our Windows “have to” apps.  I upload the missing file to my Linux account using pscp.exe, then pull it over to the broken XP installation with scp, using the live-CD session.   Next, reboot to Windows XP, and voila! it comes up normally.

Going back over the long tech forum trail, I find that this is a not-uncommon event with XP, and, if you only have Windows to work with, is painful if not nearly impossible to recover from, usually involving extracting the file from an OEM CD.  We do have one (which is where our Virtual-Box installation came from), but it was much easier to do the deed by booting Linux.  Besides, taking the file from a running XP installation ensures it is the latest version–there is always that issue when restoring from the original CD.  Windows doesn’t have packages with dependency checking like Linux and other Unix-like systems do, so it would be easy to get out-of-spec by doing that.

Next, we did some more detective work to see why something like this would happen in the first place.  Now, there isn’t a smoking gun, because these forums are run by and frequented by folks who think that Windows is the only way to compute, and that whatever it does is “normal,” so nobody thinks it is strange that this fatal bug is actually a Stupid Microsoft Trick, used in a feeble attempt to emulate Unix’ system single-user-mode and ‘NOLOGIN’ environment settings, when installing system updates.  It appears that the Windows Updater actually modifies the registry under certain circumstances to perform this trick, and may replace the userinit.exe file.  After all, have you ever wondered how the system does the reboot thing during an update and bypasses the logon entirely?  If you are so unlucky as to try to logon to your machine during this time or–because the automated process may not go quite as smoothly due to network glitches, power events, or whatever–after the update has failed to complete correctly, you are now permanently in zero-user mode, and the file is just plain gone.

Sorry, Microsoft, a lot of us think this is just not the way to run a computer operating system.  Our preferred system architecture is well-thought-out, handles multiple security levels, multiple users, multitasking, and has done so nearly flawlessly for about 40 years, during which time it just gets better.  Yours has been built from a 16-bit port of an 8-bit microprocessor program loader that has been an endless series of compromises and risks to beat the clock and can’t get any better unless the legacy crutches are abandoned once and for all.  No matter how fine you try to close the gap on a race condition, it is still a race condition, and sooner or later, deadlocks happen.  A patched kludge is still a kludge.

Meanwhile, while working on the fix, we quietly backed up the entire hard drive onto one of the Linux servers. so our data is safe.  Now, if we could only persuade the software providers to write their applications in solid code for operating systems it could actually work on safely, and we wouldn’t have to endure these kinds of torments.

Caveat–this worked for me as written, but you might have to modify more files in the system32 directory to get the machine to boot so you can fix the Registry if the logon value for the executable file has been changed.  In my case, the userinit.exe file was just missing, the Registry seems to be normal.  No malicious malware suspected, just a missing file that gets unsafely manipulated by updates–on purpose.