A while back, we wrote about rebuilding a crashed Raspberry Pi system.  In the course of reinstalling the system (on a new chip–the old SD card that contains the operating system had “worn out”), we had made a fatal slip.  This system happens to be our gateway system, i.e., connected directly to the Internet to provide us access to our files and some web services while out of the office.  Unfortunately, this also provides the opportunity for the world-wide hacker community to try to break in.

Normally, we have safeguards in place, like restricting which network ports are open to the outside and which machines and accounts are allowed login access.  However, in our haste to get the new system up and running as quickly as possible, we connected the device to the Internet to download upgrades before the configuration was complete, meaning the system was exposed without full protection for several hours to several days.

Now, our security logs record, on a normal day, hundreds of break-in attempts (see screenshot above).  We aren’t the Democratic National Committee or Sony, just a small one-man semi-retired consulting business.  But, the hackers use automation: they don’t just seek out high-value targets, they scan the entire Internet, looking for any machine that isn’t fully protected.  If they can’t steal data or personal information, they will use your machine to hack other machines.  If you use Microsoft Windows, you are undoubtedly familiar with all sorts of malware, as there are many tens of thousands of viruses, trojan horses, adware, ransomware, and other malevolent software that invades, corrupts, and otherwise takes over or cripples your machine.  Unix systems are less susceptible to these common attacks, but, if an account can be compromised, or a bug in the login process exploited, eventually a persistent attacker can gain system privileges and install a ‘rootkit,’ a software package that replaces the common monitoring and logging software, redirecting calls through the rootkit, which hides its existence and activities from the reporting tools, even the directory listing utilities.

Once an attacker takes over a Unix or Linux machine, there is no limit to the damage they can do on the Internet, as Unix/Linux is the basis of most of the servers on the Internet, and can become as a SPAM server, web-spoofer, or hacker-bot itself.  (Microsoft Windows Server has the rest, nearly half, and they are even easier to break into.)  I began to suspect this might have happened when normal system functions failed to terminate or run correctly.  We have a lot of custom software built on this machine, which runs on a scheduler.  The machine got slower and slower, and it was apparent that the jobs run by the scheduler were never exiting, filling up the process table with jobs that weren’t doing anything, except taking up space, which is always at a premium.  Clearing out all the jobs, restarting the machine, and starting the processes manually worked–until about 2:00pm.  Very suspicious: a rootkit, once installed, can repair or re-install itself even if the administrator restores many of the co-opted command files by normal upgrades or by a conscious attempt to recover from the intrusion.

The main problem seemed to be with /bin/sh, the system shell, which is actually /bin/dash, a shared object.  Cron, the scheduler, uses dash to run the jobs, where the normal user login shell uses /bin/bash, a non-linkable executable shell with similar functionality.  A rootkit is generally constructed as a filter, wrapped around the co-opted commands, so it would be easier to link to the *real* /bin/dash in an undetectable manner from the filter program than it would to wrap /bin/bash.  In this case,  assuming an intrusion was the cause, something went wrong, rendering dash non-functional.  Perhaps the intrusion was not compiled for the ARM  processor used by Pi, though most of a rootkit would be scripted to be portable among different CPU architectures and Unix/Linux versions.

An analogy to the problem would be like finding out who let the horses out: it is easy to identify wolf or horse-thief tracks outside the barn when the door is barred, but, if you left it open and the horses have bolted, it is more difficult to find out what happened–the traces are covered. I did install some intrusion-detection software, but running it after the tracks are covered over is usual a futile effort.  However, there were enough questionable traces to warrant taking corrective action.  Besides, even if the problem had been caused by some inadvertent misconfiguration on my part (unlikely, considering the fact that the machine could be made to run for several hours before the problem reasserted itself), the solution was clear:  reinstall everything.

The first step is to backup the data, including configurations.  Now, this is not just an ordinary computer:  Raspbian, the Debian-based operating system distribution designed for the Raspberry Pi computer, comes with a simple desktop intended to introduce new users to Linux.  But, this machine doesn’t use the desktop and is not even connect to a monitor most of the time: it is an internet gateway, web server, and custom webcam driver, so has a lot of “extras,” both loaded from software repositories and written especially for this installation.  Backups are important, since much of the software only exists on this machine.  And, since we only have one camera, fail-over isn’t possible without physically moving the camera from one machine to another, not a trivial exercise, as the connection is on the motherboard rather than an external connector.

Now comes the glitch: Since the introduction of the Raspbian operating system, it has been based on Debian 7; but, since Debian 8 was recently released, a new version of Raspbian is also available.  So, the machine was rebuilt with Raspbian “Jessie”, replacing Raspbian “Wheezy” (the releases named after Toy Story characters rather than just the numbers–as with Apple OS/X, Debian releases tend to have names in addition to release numbers).  Installation on Raspberry Pi is not like other computers.  Since there is no external boot device, the operating system “live” image is loaded onto the SD card that serves as the boot device and operating system storage.  Initial configuration is best done without a network connection, since the startup password is preset and well-known.

So, avoiding that mistake (booting on a network with the default passwords, the single most preventable source of hacker intrusions), we booted with the network cable disconnected and a monitor and keyboard attached, changed the password and expanded the system to fill the SD card, set up the other user accounts, then shut down the system, removed the SD card, and mounted it on the backup server to finish transferring vital data, like the security keys and system security configurations.  In larger systems with a permanent internal boot drive, such “hacker-proof” installation is done on an isolated network, but, since the boot drive on a Pi is removable, it is easy enough to edit the configuration files on another system.

So, with the system fairly well hardened by securing the system accounts and user accounts, it was rebooted attached to the network and the system upgrades and extra software packages (like the web server) installed.  So far, so good.  But, since we upgraded the operating system, the server packages were also upgraded, most notably moving from the webserver, Apache, version 2.2 to version 2.4.  Apache has been the predominant web server software on the Internet for 20 years, so it is in a constant state of upgrade, for security and feature enhancements.  Between version 2.2 and 2.4, many changes to the structure of the configuration files  were made, so that not only did the site configuration need to be restored manually, but there was a fairly steep learning curve to identify the proper sequence and methodology by which to apply the changes.

Then, of course, were the additional Python modules needed to be installed to support the custom software, which involved downloading and compiling the latest versions of those, since Python 2 also upgraded from version 2.7.3 to 2.7.9 (we haven’t yet ported the applications to Python 3, which moved from version 3.2.3 to 3.4.2 between Debian 7 and Debian 8).  Finally, there were other tweaks, like comparing system configuration files to update group memberships for access to the camera hardware, loading the camera drivers, and setting file ownership and permissions for data and program files.

We could have saved most of this by sticking with Raspbian Wheezy, but eventually, support for older systems goes away, and the newer systems are usually more robust and faster: open source software evolves rapidly, with new minor releases every six months and new major releases every two years for most distributions, and an average life span of five years for maintenance of major releases and a year for minor releases.  As we said before, Linux is free, if your time is worth nothing.  The price of keeping current is constant maintenance.  Patch releases occur as they are available, with maintenance upgrades almost daily.

Finally, after a week of tweaking and fiddling, the webcam service is back up and running.  And, the security logs show break-in attempts every few minutes, from multiple sites all over the world (one from Portugal recently, others from unassigned addresses–ones with assigned addresses undoubtedly come from computers that have been compromised and used as hacking robots, as hackers don’t want to be traced back to their own computers, ever).

So, the moral of this post is:  don’t ever expose a stock, unmodified computer system directly on the Internet (which is difficult to do, when all upgrades, new software, etc is available only through download from the Internet–which should be accomplished only from behind a proven firewall).  But, you can set passwords and change system accounts before joining a network.  And, if your computer is hacked, take it to a professional, and don’t grumble about the cost or time it takes to restore it.  Pay for malware scanning software and keep your subscription up to date, as well as scheduling upgrades on a regular bases.  And, if you are a professional, don’t take shortcuts (i.e., install and configure off-net or behind a firewall), keep good backups, install intrusion-detection software early, and check for security upgrades daily.  Change the default passwords immediately, and create a new, weirdly named administrative user, and deny external logins for all administrative users.  Use two-factor authentication and public-key encryption on all authorized user accounts.  They are out there, and they are coming for your computer, even if you don’t have data worth stealing: they can use your computer to spread SPAM or steal data from someone else.