Category Archives: Linux

All things Unix, Linux, Uncategorized

Memory Leaks: Programming in Retirement

What “retired” software engineers do: keep making software. A project I’ve been putting off for a long time, having been interrupted from time to time with medical issues and other projects, including learning Python, which I avoided for many years, writing in PHP, Perl, and Ruby instead. Python is the current “most popular” cross-platform programming language and has a lot of library support, so the likelihood of someone else [younger] being able to maintain this project in the future is good.
 
But making progress on it now: Starting with 619 photos of pages and items from “sample books” of handwoven patterns, containing notes and actual fabric samples made by weaving guild members up to 50 years ago. The books were too fragile to allow members to peruse them, so one of our members carefully photographed each item and the books put in storage. I’ve processed the photos to enhance the contrast from the faded and yellowed pages so the writing is legible, and put the binary images into a database, organizing them in 186 separate sets, with one to eight images per sample.
I’m now writing code to display the samples and provide forms to manually digitize the information in the notes, to make them searchable. In the screenshot, left to right: Python language reference (on-line), four “sandbox” terminal windows with, clockwise from top left, shell to launch programs, database “sandbox” to test query construction, Python “sandbox” to test methods and objects in code, and code from a demo script I wrote some time ago to explore methods of displaying images from the database, from which to copy code into the full project files. Yes, there are three monitors attached to my computer. The monitor on the right is the editor screen for four of the files I’m currently working on, that interact with each other to select, retrieve, and display the data.
All things Unix, Linux, Uncategorized

Evolution of the Personal Computer – a Remembered History

The following is an article I wrote on Quora in response to a question about the evolution of the personal computer.  Looking back at the 45-year history of the “personal confuser” as we sometimes call those devices we now can’t seem to live without.  Realizing that most of the population was born into a world where the PC has always been there, it is sometimes important to remember where we came from and how we got here.

Soon after the development of the microprocessor, in the early 1970s, the Altair 8000 appeared on the market in 1975, powered by the Intel 8080 8-bit CPU. Many more 8-bit machines followed, coupled with keyboards and CRT monitors, though many of the early machines intended for home use used television sets for monitors, connected through a video to TV converter outputting an NTSC signal on VHF channel 3 or 4.

Commodore introduced the PET computer, a complete unit with keyboard and monitor in one desktop case, which ran a version of BASIC as an operating system. In the early 1980s, a home version, the Commodore VIC-20, was introduced, with the computer built into an enlarged keyboard, with a cartridge tape drive for data, and video output to an optional CRT monitor or a TV converter. These machines had a whopping 5KB of memory.

A number of smaller, more portable machines appeared, all with BASIC as the operating system, stored in various incompatible byte-code forms. Tandy Corp. sold the Radio Shack TRS-80 (which stood for Tandy Radio Shack w/ 8080 CPU), which promptly got the moniker “Trash-80” from critics. Chip-maker Texas Instruments, which had earlier killed the mechanical slide rule market with their hand-held calculators, came out with the TI-99 computer. Another low-cost compact machine was from Timex. Most of these machines used the MOS 6502 8-bit microprocessor, or the Zilog Z-80.

Steve Jobs and Steve Wozniak came out with the Apple computer, which, in Jobs’ inimitable marketing instincts, was named Apple II in the production model, to give the impression that it was a new and improved version of an earlier model, actually a prototype built in a wooden box. Apple would become one of the most successful of the early personal computers.

About the time Jobs and Wozniak were tinkering, Gary Kildahl, also in California, was experimenting with uses for the 8-bit Intel microprocessor, came up with a loader and command shell for a Winchester hard disk, which he named CP/M, Control Program for Microprocessors, introducing the concept of a block-structured disk operating system and a command shell that wasn’t BASIC. Prior computers were capable of running machine-code programs directly in addition to BASIC, but CP/M was the first to emphasize using binary programs on disk as the primary operating mode.

Some of the first commercial CP/M machines were the Osborne I luggable computer, which featured a 360-KB 5–1/4″ floppy drive, a 16×64-character CRT terminal, and a detachable keyboard in the lid of a suit-case-sized case that weighed over 15 kg. Kaypro came out with a similar form factor CP/M machine that was more suitable for desktop use, with a larger display screen and keyboard.

With the introduction of the 16-bit 8086 CPU from Intel, IBM became interested in adding a microprocessor-based system to their business line of mainframes and terminals, first approaching Kildahl’s Digital Research, developers of CP/M, and Bill Gates’ fledgling Microsoft, which had gotten its start writing BASIC interpreters for the Altair and other hobbyist microprocessor-based systems, and had recently acquired a 16-bit independent rewrite of CP/M, which, with the addition of some concepts from Xenix, a micro-processor-based version of Unix that Microsoft had licensed, became MS-DOS, the Disk Operating System. IBM adopted MS-DOS as PC-DOS for its new personal computer, the IBM PC, which featured an open system design that allowed third-party vendors to develop add-on hardware and write device drivers for it, which, of course, led to the proliferation of “IBM clones” from many different manufacturers.

By this time, the successful marketing of the Osborne and Kaypro CP/M machines and the Apple II had spawned a flurry of software development companies building business software that would run on the desktop without requiring a mainframe back-end, including spreadsheets, project planning software, and word processors. Stand-alone word processing workstations had sprung up with the microprocessor revolution, but were single-purpose machines, relegated to the “typing pool” at large corporations and document-preparation companies. With the introduction of a desktop machine from IBM, then the largest computer company on the planet, those vendors quickly ported their products to the 16-bit MS-DOS platform, and CP/M faded from the market, in part due to the untimely death of its founder.

Microsoft solidified its grip on the personal computer operating system market with exclusive contracts with various PC manufacturers. Apple focused its market on education and the arts, moving to the wider word-size market with the Motorola 68000, a 32-bit CPU with a 16-bit data bus, and a graphical desktop system based on the Xerox Palo Alto Research Center (PARC) Star project. The Macintosh was introduced in 1984. Microsoft countered with the Windows desktop environment running on top of MS-DOS soon after.

IBM commissioned a new graphical operating system to replace the Windows environment as 32-bit CPUs came into wide use in the early 1990s, but disputes between IBM and Microsoft led to the split into OS-2 on the IBM side and Windows 95 on the Microsoft side. Microsoft’s grip on the generic PC market won out, with OS/2 gradually fading away with the incompatibility between OS/2 and Windows 95 and Windows NT.

On the Unix side, Microsoft sold Xenix rights to the Santa Cruz Operation early on, which, after porting to 32-bit, became SCO Unix, but never got our of the small niche as a multi-user solution for small to medium-sized businesses. The mainframe Unix of the 1980s ported to high-end 32-bit microprocessor workstations used in academic and scientific research.

In 1990, Linus Torvalds, a college student in Finland, desiring a 32-bit alternative to the 16-bit micro-kernel teaching tool, Minix, built a new Unix-like monolithic 32-bit kernel, around which he and developers all over the world wrapped the GNU software collection from the Free Software Foundation, creating the GNU/Linux operating system, which has, in the 21st century, captured the network server market from Unix and replaced the Unix workstations on desktops of developers and research scientists, as well as taken over the hobby market started with the Altair 8000 in 1975, with re-purposed former Windows machines and the proliferation of tiny single-board computers, led by the Raspberry Pi.

The adaptable Linux kernel also became the core of Google’s Android operating system, which, along with iOS, the embedded version of the now BSD-Unix-based operating system adopted by Apple at the turn of the century, drives all the world’s handheld computing devices, the ultimate personal computers most of us carry with us everywhere, disguised as telephones, pagers, cameras, and music and video entertainment devices, as well as portals to the World Wide Web. Microsoft’s Windows system remains the default OS for the desktop and laptop environment, and the core server operating system in corporations—for now.

All things Unix, Linux

GOTO Considered [Rarely] Necessary

I’ve been contributing to Quora.com lately, answering various computer-related questions on programming and Linux in particular.  Here’s one that came up at the end of January:

My answer is below, but, surprisingly enough, only a couple of days later, I found it expedient to use a GOTO in a PHP script, admittedly to fix a logic problem that wasn’t very well thought out.  In a while() loop, a test at the end needed to start a new output section, but the current data set needed to be put in the new section.  The expediency was to use the initialization code to start the new section, which was at the top of the loop, but allowing the loop to cycle would read a new data set, so the “quick-and-dirty” solution was to jump to the top of the loop without reading the new value.  Problem solved, and preserved the integrity of the code block, which was the main problem with GOTO in unstructured programs, back in 1968 when Edsgar Dykstra gave his infamous “GOTO Considered Harmful” proclamation.

The only time I have used GOTO has been to implement a decision construct not supported by the language. Of course, GOTO was necessary in linear programming languages which did not have syntactic constructs as we use in structured programming. All compiled programs have a GOTO in each and every looping or decision construct, but in the resulting output code. When it is necessary to use, it must be a construct defined in a safe manner, rigorously applied. Usually, this would be to stop execution in the middle of a code block by a GOTO to exit the block, something which most structured languages already have, in the form of a ‘break’ statement, or similar keyword. I could imagine a deeply nested decision tree construct where GOTO could be used for clarity, but there would probably be another way to write such a construct to be unambiguous.

One such example in C/C++ I could envision would be to construct a type of switch construct that accepted strings instead of integers as case arguments, where GOTO took the place of the break clause. I had used a similar construct in COBOL successfully, to build a switch-type operation, which doesn’t exist in COBOL, such that, once a test succeeded, the rest of the tests would be skipped. (Disclaimer: I took a job writing COBOL as a last resort, around 1990, in the middle of changing careers from systems engineering to software engineering—not a recommended choice: wrote COBOL by day at work and C by night at grad school).

Afterword:  Here is a snippet of the code block…

  
while ($fil = readdir($subdir)) {
newpage:
    if (is_dir($fil)) { continue;}
    if ( $row > 8) {
.
.
.
    if ($column >= 5 ) {
      $column = 1;
      $row++;
      print ("\n");
      goto newpage;
    }
.
.
.
All things Unix, Linux, Uncategorized

Raising Software: A Tale of Unruly Tools

One of my retirement hobbies has been shooting and editing videos, mostly of our bicycle adventures.  When I was young, back in the middle of the 20th century, one of my uncles had a movie camera and projector, and I learned to splice broken film and edit clips together with razor blades and transparent tape.  He moved away and I grew up, and the skill was filed away as just another data point in How The World Works.

Fast forward to the second decade of the 21st century, when I decided to strap a cheap digital camera to the handlebars of the bicycle.  Digital editing requires software, a much less messy and more forgiving (i.e., non-destructive to the original “film”) process.  Since we use Linux exclusively as our computing platform of choice, there were a number of choices for Open Source video editing projects, mostly attempts to clone commercial video editing software for Windows and Apple.

I looked at Cinelerra and Kdenlive, which are fairly complex tools with a steep learning curve, but settle on OpenShot, a simpler tool with a lot of attractive features and a cleaner, no frills user interface.  Openshot 1 was essentially a one-man project, a user interface written in Python with the Tk graphical toolkit, built on the Unix principle of lots of little programs working together, using the multimedia command-line editor ffmpeg and associated libraries, the Inkscape vector graphics editor, and the Blender animation tools and libraries.

Openshot 1.4.3

OpenShot made it possible to load up a project with multiple clips, arrange them on a timeline, trim the ends or snip out shorter clips, and add titles and audio tracks, like voice-over and musical scores, in multiple overlaid tracks, and turn video and audio on and off by track and clip.   For several years, this worked fine.  However, success is not always a good thing, and Openshot suffered from it.  Not in the usual riches-to-rags story of an entrepreneur or rock star who descends into excess and loses his way, but in the attempt to seek wider appeal.

OpenShot was originally a purely Linux product, as mentioned.  To port the project to Windows, it was necessary, for a project with limited manpower resources, to keep a common code base.  Openshot attempts to keep a smoothly flowing user interface through parallel processing, using OpenMP.  The Windows philosophy is based on a single-user, single-task model rather than the multi-user, multi-tasking model of Unix.  When Windows evolved into a multi-tasking system, it used the pre-emptive model, which is relatively inefficient for the pipelined processing flow in the Unix cooperative model.  So, Windows applications tend to be monolithic, with all resources loaded in one huge process.  Parallel processing in Windows monolithic applications is accomplished largely through threads, rather the inter-process communication protocols.  I’ve programmed with threads in Linux, which tends to be tricky at best, and takes a thorough knowledge of parallel processing and memory management to do successfully.

The move to Windows-compatible architecture necessitated rewriting a lot of the Unix-specific standard library code in C++, which introduces the danger of memory-management issues. Openshot began to get buggy, with newer versions crashing often.  The developers claim it is the fault of unstable system libraries, but I’m not buying that explanation.  Since the user interface was also getting a major overhaul, work on version 2 meant that no more bug fixes were forthcoming for the now-crippled version 1.4.3.  Alas, initial releases of Version 2, with the back end still largely 1.4 code base, was also prone to crashing as well as presenting an unfamiliar user experience.

Openshot 2.4.1

So, we stayed with version 1.4.3 for a while longer, with short auto-save intervals.  Finally, with crashes and deadlocks rampant, we just had to try version 2 again.  Yes, the crashes had been largely fixed, but the new version was a monolithic package wrapped in a “launcher,”  (AppImage), apparently a type of container tool including all of the library dependencies, which rapidly ate up all available memory and much of the swap space, becoming so slow as to be indistinguishable from deadlock. Memory leaks come to mind when seeing this type of behavior.  On top of that, some of our favorite tools for controlling video blanking and audio muting by track were missing, to be restored by popular demand in a future revision.   Back to 1.4.3 .

Kdenlive

The other alternative, kdenlive, based on the Konquerer Desktop Environment (kde, not native to Ubuntu, thus necessitating loading the complete KDE library support suite), is yet another learning curve, with many editing feature differences and rendering options.  We did use this for one video, as the internal de-shaking algorithm is a bit more efficient than reprocessing the clips with command-line utilities, and we had a bad experience with a new camera mount that was sensitive to shaking.  Kdenlive also crashes from time to time, lending some credence to the Openshot claim that the system libraries are at fault.

But, I continue on, putting up with slow response, freezes, and crashes, because I’m familiar with the features I like, and it produces acceptable videos.   I may spend some time to learn kdenlive, but, hopefully, Openshot 2 will improve over time.  The other alternative is to try to build a native Ubuntu version from source, which is a daunting task, since most open source software has a very specific support software version dependencies.   Despite the woes with Openshot’s growing pains, it is still faster than writing command-line scripts to use the Linux multi-media-processing utilities ffmpeg or avconv to trim and assemble video clips and sound files.  I use those command-line tools to assemble time-lapse videos from my home-built security camera system, but that is much simpler.

All things Unix, Linux

Affordable Linux: A Quest

As most readers know, we here at Chaos Central are a Linux establishment for our primary computing.  We use Apple iPhones and iPads for mobile convenience and casual surfing, but depend on Linux for everything else–with the exception of what has become known at CC as “The Screaming Season,” where we are dependent on Windows to run Turbotax to render unto Caesar.

As much as we’d like to be on a 3 to 5-year cycle for computer upgrade/replacement, retirement, and before that, negotiated fee schedules, have reduced that to “when it breaks, MacGyver it.”  The result has been dismal.  When Judy’s desktop Linux machine died last year, having already received a CPU cooling fan transplant several years ago, salvaged from an old machine before it went to the recyclers, we pressed our old 2007-vintage laptop back into service, restoring her files from backup.  That machine had had a new hard drive and more memory installed during its last incarnation, but the extra memory had faded away, and the CPU not up to the demands of the 2016 version of Ubuntu Linux, so it was running slowly with the reduced-capability desktop.

Since the current crop of machines with pre-installed Linux are aimed at developers rather than standard office use, we couldn’t justify a new laptop from one of the few vendors who supply them.  So, as in the “bad old days,” we went shopping for a cheap Windows laptop on which we could install Linux.  The “cheap” machines are now essentially Microsoft’s answer to the iPad and Android tablets they see as main competition–a light-weight, small-screen device with as small (32GB) flash memory (EMMC) instead of a hard drive, and no optical drive.  Not to worry, I loaded a light-weight version of Linux (Lubuntu) on a 64GB flash drive and it became her new machine, with the added advantage of being portable.  The big disadvantage was the need to plug in the Linux memory stick to run it.  The biggest problem was the short life span of flash drives when used as the primary drive on a Unix-like system.   And, the system was incredibly slow.

Well, in only a few months, the flash drive expired.  No problem, I thought, just build another and restore.  But, as many others have found, the ability to install in the first place was a fluke: these little Windows tablets masquerading as a laptop by having an attached keyboard are finicky: try as I might, I could not get it to boot from the install memory stick again.  So, the little blue Dell machine has become the new Turbotax home, effectively retiring the $80 refurbished Vista->Windows7->Windows10 desktop we acquired a couple years ago and built up (another $40 video card) to run Win10 (badly).

So, Judy was back to the old laptop.  This time, we pressed her old 2010 HP Netbook back into service, even slower and more clunky than the older Compaq (which she also used during this evolution), running a light-weight version of Mint Linux.  The office was beginning to look like the computer labs of the early 2000s, where we pressed piles of recycled obsolete machines into experimental compute clusters to get 21st-century performance out of 20th-century machines, a distributed computing architecture conceived in the mid-20th century but not economically feasible until there was a huge supply of retired machines awaiting landfill space.

A comprehensive check of Linux-capable systems was not promising.  The custom-built laptops (our new criteria is portability to fit our mobile retirement life-style) are expensive and beyond our budget with no current paying clients.  My own laptop, which was high-end (and expensive) in its day (2010) is showing its age, with a too-small disk, weak battery and some screen burn-in.  But, the equivalent new models are in excess of $2000, difficult to justify on a pension without contract income to support it.  Nevertheless, it is only a matter of time before it wheezes its last, so we need to first find an affordable and long-term solution for Judy’s immediate need while keeping in mind the future budget hit for a developer machine.  The other option for affordable Linux-ready or pre-installed laptops are refurbished machines proven to be compatible.  The disadvantage there is the same facing my situation: the batteries are in mid-life, so long-term reliability is an issue,as well as mobile portability.  Older machines are also heavier than desired.

So, we finally settled on a fourth option: installing Linux on a Chromebook.  Chrome is a laptop operating system from Google, a variant of the Android system used on smart phones and tablets.  Since Chrome and Android are basically embedded Linux, it is relatively easy to add Linux to them as a container, using Crouton, a management system similar to the Docker system used to install and manage containers on native Linux machines.  I say relatively, but that assumes some heavy-duty skills in system administration, since Chrome is a locked system.  First, the system must be unlocked into Developer mode, which basically voids the warranty and security protections.  At least, Google does not provide support for a machine in this state.  The hacker community, then, is at the mercy of well-intentioned, but not always well-informed peers for advice.

I followed the available advice in the hacker forums into the corners that most others had painted themselves, finally figuring out an important missing step.  To use the Crouton system, the wily hacker must first use the official Google developer shell to set the root and administrator passwords.  Then, Crouton can access the administrative account to install the Linux container.  The less-helpful solutions offered essentially required reinstalling Chrome and starting over, which was not necessary at all, but the default obvious solution to developers and admins raised in the Age of Windows, where frequent rebooting and periodic reinstalling is considered a normal operational necessity.  Unix and Linux admins only reboot after installing a new kernel, which may be months or years, depending on the stability of the system and whether the management decides to incorporate the latest patches when they are released (many don’t, for fear it will break fragile applications, despite the security risk of not upgrading).

So, now Judy has a brand-new system that, hopefully, will be stable for a reasonable time.  The advantage of this system is that it is still Chrome, so she can use the user-friendly desktop apps available to Chrome, similar to the ones on Apple iOS, and switch to the Linux desktop with a simple key-combination to run the Linux applications we depend on.  A similar key-press combination switches back to Chrome.  And, the Chromebook, though large enough to have a full HD display, is light enough to be highly portable, with a very long battery life.  The minor concession to our ill-advised attempt to co-exist with Windows is that we now have a portable tax preparation machine.  During the rest of the year, we may turn it on once in a while to let the updates install and top off the battery, so that it won’t take three days to update next tax season.

Chromebook, running Chrome with Crouton in a browser window.

 

Chromebook, running Ubuntu Linux with LXDE, with a simple browser, and essential apps, including Fiberworks Bronze weaving software running under WINE. (a Windows emulator running in a Linux container under Chrome–boggles the mind, no?)