Quarantine Diaries, Chapter 16: The Code Warrior

As the pandemic roars all around us and the U.S. is in turmoil surrounding the election, I decided it was time to return to an activity that makes the world go away for the duration, and something that focuses on what was supposed to be the main thrust of this blog:  Computer programming, the ultimate drugless out-of-body experience.

Warning: for the usual audience looking for tales of septuagenarian bicycling adventures or living with the quarantine, this is going to get a bit technical, since we’re reaching for a wider audience, fellow travelers on the road to code. But, if you are a Python programmer (or just curious about what goes on under the hood in your computer), read on.

The Code Warrior at his battle station. Browser with programming language references on the left, code editing and testing center, and requirements docs on the right, Photo captured by one of the cameras we’re seeking to automate.

As a Unix/Linux system administrator, I write a lot of scripts that are meant to be run from the command line, or embedded in other scripts to be run automatically, so providing a robust and feature-full
option list and argument list parser makes a given script more
versatile. I’ve done those kinds of scripts using the Perl language, but we’re shifting to Python now: I decided to explore a handy Python module called argparse that does just that, and includes an automatic help screen in the process, and find a use for it.

I’m relatively new to Python, having avoided it from its inception in 1991 until 2014, when I haltingly wrote a  script to operate a camera on a Raspberry Pi single-board computer, since the camera libraries were in the Python language.  It works, but I did become more interested in getting fluent in Python.  Python is more than Yet Another Object-Oriented Scripting Language: it has a distinctive style and philosophy to go with it, the Pythonic Way.  And, since languages evolve, a major dialectic shift from Python 2 to Python 3.  So, it’s time for some language immersion while we’re in the midst of the pandemic lockdown.

The core application for which this exercise is designed is a system to record from USB cameras. Since the arrival of Zoom World, where all of our organizational meetings are on Zoom, I have built up a collection of USB webcams from the local thrift store, and use them with Zoom, but would like be able to automate photo and video capture for other purposes, to augment the Raspberry Pi camera I use to monitor the driveway, recording for playback as a timelapse. Here, then, is the skeleton of my command line user interface for the USB camera project, which is inspired by, but not based on the option-rich raspistill and raspivideo programs that come with the Raspbian Linux distribution for Raspberry Pi.

This code snippet is a front-end to a larger script (not provided) that will capture output from USB webcams attached to the host system, built with the OpenCV system library, python-opencv package (for Debian-based Linux distributions), and a modified version of the acapture module.  It took a week or two of research into the OpenCV libraries and some experimentation to get enough insight into how that works in order to design a front-end command interface.  Those image-processing functions now need to be refactored into a back-end for this script, a project for next month.

The resulting system takes command-line options and arguments to control which camera to use, and whether to take or display or save camera output as a still, burst, video, or timelapse, or to display stored images as a slideshow, or single photo display. For simplicity, most of the arguments have default values, i.e., 10-photo burst, 10-second video clip, etc. so if we need a short clip or choice of several photos, we can just call the parent command without explicit arguments and get a standard output.  The script, as written, makes some assumptions about where to store the images and how to organize them, which may change as the total application is fleshed out.  As written, the idea is to store still images with file names serialized, from 0001 to 9999, as that’s a convenient way to assemble still images into a timelapse video.

The concept of having one program that serves multiple purposes makes maximum use of the feature in Unix-like file systems to be able to give a file multiple names, through hard links or symbolic links.  To use this, we make the final program executable and link to these names,  in Linux or macOS. Modifications may be necessary to run in Windows, like adding the “.py” file extension to the script names. File names aren’t hard-coded, but listed as keys in the Capabilities dictionary in the code, so this concept and construction method is adaptable to other application user interfaces, and the getargs() function and associated global constants can be plugged into any other set of programs.  As part of the design document, we list the program names and the arguments they take,

  • usbstill : take a photo, display on screen and save to disk:  ‘camera, imgdir, imgidx,
  • usbburst : take a series of photos, no delay.
    camera, imgdir, imgidx, frames (# of frames)
  • usbvideo : take a video, display on screen and optionally save camera, imgdir, imgidx, duration, scale, record?
  • usblapse : take a timelapse sequence camera, imgdir, imgidx, interval, duration, scale
  • usbslide : playback still photos as a slideshow. imgdir, imgidx, speed (0 = manual), directory
  • usbshow : display any photo resolution, path

The test harness–showHelp()–for this front-end program simply generates usage and help messages using the default values for each option: with the exception of usbshow, which requires a filepath for a  single image file, which prints the usage message and throws an exception.

Here’s the  output of the testing, using the showHelp() function to walk through the commands, with no arguments on the command line.  We’re simulating the command line in this test, so we substitute the command name for the default sys.argv[0] when calling getargs().

The main() function in the code listing below is a framework, the beginning of the main program, with stubs where the image processing and camera capture will go.  Most of that’s done, as a result of experimentation with OpenCV, but needs a lot of work and is beyond the scope of this post, which explores how to use the command line to not only tell the program what to do, but make sure the inputs are reasonable.

The main parts of the argparse module that we use are the ArgumentParser() class, the prog class variable, and the class methods add_argument(), parse_args(), and, in the test harness, print_help()

A lot of the power in Python and other programming languages is the literature–the code libraries (modules) that are either specialized extensions to the language or contributed software that are useful tools to quickly build new systems.  Python has a librarian function, pip, that accesses the master  library over the Internet to install modules not included with the basic Python installation.  Or, you can write your own or modify existing ones and rename them–Python is open source, after all.

It’s always easy enough to get a tutorial and list objects and methods for common modules in docs.python.org, but you can get a good idea of the features in a module by using the Python dir() function:  Here, we exclude the private variables and methods (which begin with an underscore character) to show the public methods.  By convention, objects are capitalized, constants  are in all uppercase, and methods and class variables begin with a lower case character:

>>> import argparse
>>> parselist = dir(argparse)
>>> for item in parselist:
... if item[0] != "_":
... print(item)
... 
Action
ArgumentDefaultsHelpFormatter
ArgumentError
ArgumentParser
ArgumentTypeError
FileType
HelpFormatter
MetavarTypeHelpFormatter
Namespace
ONE_OR_MORE
OPTIONAL
PARSER
REMAINDER
RawDescriptionHelpFormatter
RawTextHelpFormatter
SUPPRESS
ZERO_OR_MORE
ngettext

The primary object used in creating a command line parser is ArgumentParser

>>> parselist = dir(argparse.ArgumentParser)
>>> for item in parselist:
... if item[0] != "_":
... print(item)
... 
add_argument
add_argument_group
add_mutually_exclusive_group
add_subparsers
convert_arg_line_to_args
error
exit
format_help
format_usage
get_default
parse_args
parse_intermixed_args
parse_known_args
parse_known_intermixed_args
print_help
print_usage
register
set_defaults

Most of the rest of the class variables and methods are used internally by those major functions, to  perform data validation, print out usage and help, or handle errors.  add_argument() is an impressive method, providing a way to name options and arguments, constrain the data types and range of values, and make the arguments optional (with the nargs=”*” clause) or mandatory, with default values for optional arguments if not given .  Pre-validating the data greatly simplifies the logic in the rest of the program, since we can assume the input values are valid for the operations to be performed.

Here’s my code, modified slightly to fit in the page format.

#!/usr/bin/env python3
# Code

import argparse
import sys

# "Cameras" is system-specific: modify for number # of attached cameras, with /dev entries.

Cameras = ["/dev/video0","/dev/video2","/dev/video4"]
Scales = {"frames" : 0, "min" : 60, "sec" : 1}
Basecap = ["camera","imgdir","imgidx"] 
Capabilities = {"usbstill" : Basecap,
                "usbburst" : Basecap +   
                ["frames"],
                "usbvideo" : Basecap + 
                ["duration","scale","record"],
                "usblapse" : Basecap + 
                ["interval","duration","scale"],
                "usbslide" : 
                ["imgdir","imgidx","interval"],
                "usbshow"  : 
                ["resolution","path"]
                }

def getargs(myprog=sys.argv[0]):  #
    parser = argparse.ArgumentParser(prog=myprog)
    caps = Capabilities[parser.prog]
    if "camera" in caps:
        parser.add_argument('--camera', type=int, 
                            default=0, choices=range(len(Cameras)))
    if "imgdir" in caps:
        parser.add_argument('--imgdir', type=str, 
                            default='./camera')
    if "imgidx" in caps:
        parser.add_argument('--imgidx',
                            type=int,
                            default=1,
                            help="""Frame number to start: if missing or 1, files in imgdir will be deleted""")
    if "frames" in caps:
        parser.add_argument('frames', type=int, 
                            nargs='*', 
                            default=10,
                            help="numeric frame count")
    if "interval" in caps:
        parser.add_argument('interval', 
                            type=float, 
                            nargs="*", 
                            default=5.0,
                            help="time between frames in seconds")
    if "duration" in caps:
        parser.add_argument('duration', type=int, 
                            nargs="*", 
                            default=10,
                            help="Recording time, units of time")
    if "scale" in caps:
        parser.add_argument('scale',
                            type=str, nargs="*", 
                            default='sec',
                            choices=
                            ['sec','min'],
                            help="unit scale, seconds or minutes for video")
    if "record" in caps:                 
        parser.add_argument('--record', 
                            type=bool, 
                            default=False,
                            help="Save video in file: True or False (default)")
    if "resolution" in caps:
        parser.add_argument('--resolution',
                            type=str, 
                            default="640x480",
                            choices=
                            ["320x240", 
                            "640x480", "960x720", 
                            "1280x960"],
                            help="Display resolution")
    if "path" in caps:
        parser.add_argument("path",type=str)
    args = parser.parse_args()
    return(parser,vars(args))


# /Code

# Test

def showHelp(): # showHelp is a test to display the help screens
    for myprog in list(Capabilities.keys()):
        print("\n>>>> " + myprog + " <<<<\n")
        try:
            myparser,myargs = getargs(myprog)
            myparser.print_help()
        except:
            print("Error: missing required argument.")


# Skeleton for the image capture functions.
# imaging requires installation of OpenCV and 
# python-opencv
# import cv2 module
# this stub just sets up the variables needed by 
# the imaging functions.

def main():
    import os
    import os.path
    import glob

    if sys.argv[0] in list(Capabilities.keys()):
        call = sys.argv[0]
    else:
        call = "usbstill"  
        # default if running from template
    parser,argdict = getargs(call)
    keylist = list(argdict.keys())
    prog = parser.prog
    if 'camera' in keylist:
        camidx = argdict['camera']
    if 'imgdir' in keylist:
        imgdir = argdict['imgdir']
        if not os.path.exists(imgdir):
            os.makedirs(imgdir)
    if 'imgidx' in keylist:
        imgidx = argdict['imgidx']
        if imgidx == 1: 
            #delete existing files in directory 
            #if starting at 1
            files = glob.glob(imgdir + "/*.png")
            for fil in files:
                os.remove(fil)
    if 'frames' in keylist:
        frames = argdict['frames']
    if 'duration' in keylist:
        duration = argdict['duration']
    if 'scale' in keylist:
        scale = argdict['scale']
    if 'record' in keylist:
        record = argdict['record']
    if 'interval' in keylist:
        interval = argdict['interval']
    if 'path' in keylist:
        path = argdict['path']
    if 'resolution' in keylist:
        resolution = argdict['resolution']
    
    if prog == "usbstill":
        # take one photo, store
        return True
    if prog == "usbburst":
        # take frames number of photos
        return True
    if prog == "usbvideo":
        # record video for duration seconds or 
        # minutes, save if record is True
        return True
    if prog == "usblapse":
        # take pictures at interval seconds for 
        # duration seconds/minutes
        return True
    if prog == "usbslide":
        # display photos in imgdir sequentially, 
        # at interval
        return True
    if prog == "usbshow":
        # display a single image at the selected 
        # resolution
        return True
    return False  
# something went wrong--should never get here.

# /Test