Upgrading from Ubuntu 12.04 LTS to 16.04 LTS

From a precise pangolin to a xenial xerus

I’ve been putting off upgrading my Linux box for months, but now that official support for Ubuntu 12.04 LTS (Precise Pangolin) has just ended, I figure it’s now or never. In the end I found that I should have done it months ago as it went very smoothly.

My first question was, should I just upgrade to 16.04 LTS, or something newer, or maybe even Linux Mint which has been capturing hearts and minds over the past couple of years? In the end I decided to stick with Ubuntu, and given that the Linux box is really just a backup file store, I thought it would be best to go with a distribution that’s not going to need updating again in the next 6 months. I suspect that this will be the last upgrade for this machine: the motherboard, RAM, CPU, and PSU (and fans) all date from 2008. (Which reminds me, the CPU fan is getting cranky too…)

So off I went and downloaded the relevant .iso from Ubuntu. Hmm – server or desktop version? I had vague notions of using VNC to run a desktop session so I went for desktop, knowing that it would be straightforward to add whatever server packages I’d need later. Download, burn to DVD (yes, it seems I needed a DVD – which had to be done on the Windows machine as the burn attempt failed on Linux), insert and reboot. Oh yes – did I mention that I made sure I had backups of any important data from that hard drive? No? Well, remember to back up your stuff before doing OS upgrades!

I don’t have screenshots so bear with my wordy descriptions.

Installation of 16.04

The machine booted up and presented me with a black screen containing two low-resolution icons at the bottom, one of which looked a bit like a keyboard (I think). I waited. Nothing happened, so I hit the space bar. Ta-da! We have liftoff. I have no idea what it was waiting for. Please use words if I need to do something!! The disc spun up to obnoxious levels and eventually I was presented with my next screen with the choice of trying out Ubuntu, or installing Ubuntu. I clicked “Install”, and on the next screen I checked the box to download updates during the installation, and to add some third-party software that (for licensing reasons) can’t be included by default.

And away it went. It recognized that I had an existing Ubuntu installation on my OS drive and gave me 5 options to choose from. I could let it upgrade 12.04 to 16.04, but I wasn’t sure about doing that: I’ve that it’s more trouble than it’s worth. Alternatively I could install 16.04 alongside 12.04 – not much point in doing that. Then there were two options that I felt were somewhat ambiguous. The first said: “Erase 12.04 and reinstall”. Reinstall what? Reinstall 12.04? Why would I do that? The other said: “Erase disk and install Ubuntu”. Presumably that means the version of Ubuntu on this disc (i.e. 16.04) but again, not entirely clear. Fortunately I had an escape route: the fifth option was off on its own and said: “Something else”. 🙂

So I clicked “Something else” and got on with the next step, which was to examine the hard disk partitions. It took me a minute (some of which was spent contemplating an error message) to work out exactly what was required here: I had to select the desired file system and mount point for the relevant partitions. Since I had a separate partition for /home, I made sure to select / for the main one, and /home for the other. In doing so I completely forgot about the other two hard drives in the machine (not an issue, but it would have saved me another post-installation step – see below).

With hard drives sorted, the next steps were setting the timezone (easy: it correctly identified me as being in Vancouver) and keyboard (standard US). Next was giving the computer a name, and I stuck with the previous one (my current home computer naming scheme makes use of lakes in the English Lake District). Then I had to create a username and set a password. Then it was time to restart and I was reminded to remove the DVD from the drive and asked to press enter to continue.

Post installation

A few moments later I logged in with my shiny new user account and began the task of getting things set up to taste. First up was a second user account, which left me a bit puzzled to being with as I had to click on a button that didn’t look clickable in order to bring up the option of actually enabling the account and setting a password.

The first job (actually, this should have been my first job, not the second user account) when installing a new operating system is to apply all available updates. The package manager said I only had 12, so I left it doing its thing and called it a day.

The following morning I picked up with the next round of tasks: installing tcsh (and using that as my preferred shell) as well as Geeqie and Dropbox. Then I wasted a couple of hours trying to get a headless VNC setup to work before I took a moment to think about how often I would actually use that. The answer came back pretty quickly: almost never, so changed my mind and decided to just leave it as a headless server, and start a VNC session if and when I actually needed one.

Headless operation

I’ve been running the Windows and Linux machines with a set of KVM cables for the past few years. Recently, one of the cables has been getting flaky, causing the display to blank from time to time, so I decided that I was going to cut those cords and have the Linux run without a keyboard, mouse, or monitor. After all, even with the KVM switch, I still logged in over SSH 99 % of the time. (Speaking of which, I was stumped by the fact that my first attempt to SSH in to the Linux machine failed. That is until I remembered to install the SSH server package! D’oh!)

Setting up for headless operation turned out to be trivially easy:
% sudo systemctl disable lightdm.service
In other words, disable the graphical login greeter. (In the old days I would have specified % sudo init 3.) Yay! And into the corner you go little computer!

Other hard drives

Next on my list was mounting my other hard drives in sensible places. By default, Ubuntu had mounted them under /media/andy, and they only mounted on logging in to a desktop session. Running headless, they were never mounted unless I did so explicitly. On the plus side, that made it very easy to get them set up properly. A quick search led me to this post at Ask Ubuntu with a nice clear layout of the steps required.

I created a couple of mount points for my hard drives under / (literally just % sudo mkdir /mountpoint1 /mountpoint2). After that I ascertained the UUIDs for each hard drive (using blkid), and edited /etc/fstab to add the relevant details in the right place. Then I ran
% sudo mount -a
to (re)mount the disks, and I was good to go.

Samba

Since this machine’s primary purpose is to be a file store/network drive, I needed to get Samba up and running. Installing Samba was easy, and setting up was just as easy thanks to this post at Linuxbabe. Installing Samba automatically enables it as a service and sets it running, which is very convenient. I tested it out by re-creating my network drive in Windows, and then mounted it from my Macbook Pro too. All seemed to work just fine.

Final thoughts

And that was pretty much it. Very easy, very smooth. Had I thought things through a bit more closely, I could have saved myself a bit of time and a couple of steps, but in the end it was no hassle thanks to the articles I mentioned above.

I’m sure I’ll find more things missing, and I haven’t done anything about setting up Python, Perl, Java, or any compilers. But much of my development work is done on my Macbook Pro these days, so I’ll just deal with it when the time comes.

Recovering MySQL after improper shutdown

As an amateur DBA I recently made a classic amateur DBA mistake: I restarted my computer without first shutting down MySQL. (The restart in this case was the result of updating macOS to 10.12.4.) After the reboot I couldn’t work out why XAMPP wasn’t launching MySQL again – apache started up fine – so it was time to dig into the log files.

A quick check of var/mysql/hostnameredacted.local.err in the XAMPP installation directory revealed a string of errors and a stacktrace. Reading it carefully I realized the key error message was this:

InnoDB: Operating system error number 2 in a file operation

The other clue was a few lines earlier in the logfile:

InnoDB: Database was not shutdown normally!

As ever, Google to the rescue which led me to a handy blog post which suggested adding the following line to the [mysqld] section of etc/my.cnf:

innodb_force_recovery = 1

I saved the file, and held my breath as I attempted to restart MySQL. Success!

I went back and removed that line again (actually just commented it out – there’s an outside chance I’ll repeat this error at some point in the future!) and re-restarted MySQL. Firing up SequelPro and I was relieved to be able to see all my tables as expected, and ran a few test queries to check it out. Everything looks OK. Phew!

Lesson learned: remember to shutdown MySQL properly before a reboot.

Resuming paused processes on macOS

My Macbook Pro has been annoying the hell out of me lately, running of out application memory on a regular basis. (You’d think that’d be hard to do with 16 GB of RAM, but apparently my Mac is quite good at it…) Sure, I can restart it to clear the memory and have it load things up again as needed, but there always comes a point when that little window pops up and complains about a lack of application memory.

A couple of times I’ve dismissed that window by mistake and wondered how to restart the suspended application(s). A quick search yielded an option to the kill command that I wasn’t aware of. So the next time I do that by mistake and leave an app or two suspended, I can start up the Activity Monitor, check the process ID of the suspended application(s), and then type:

% kill -CONT <PID>

Hey Presto, app gets unstuck. Yay!

That is, as long as macOS doesn’t decide to suspend the terminal…

On backups

In my previous post I mentioned that I’d been using CrashPlan for making backups on our Windows PC. Like many, I prefer to get a feel for the software before I pay for a copy, so I installed the free version. It all seemed to set up OK, I didn’t immediately come across any missing features and so I left it alone to do its thing.

And in the absence of any error messages I believed that it was working just fine. Alas, that was not the case.

In checking out my backup drive when dealing with our Trojan infection, I found out that CrashPlan had filled its allotted hard drive and in fact hadn’t backed up anything in weeks. And in the process had utterly failed to alert me to that fact. Hey CrashPlan – how about using the Windows notification area for alerts? I mean, a backup failing is a major error and the user should be told about it as soon as possible.

One of the reasons I installed CrashPlan in the first place was that I wanted something that was more intelligent backup than Windows’ own. However, the free version of CrashPlan has a set of default settings for the backup frequency and number/age of versions that runs the risk of filling your backup hard drive then nothing more will be backed up because nothing will be old enough to delete. CrashPlan’s official advice on solving this problem? Get a bigger hard drive. Okaaay….

OK I could solve this problem by paying for the basic version, which includes unlimited online backups too. After all, $60 a year isn’t that much to pay for peace of mind is it? Normally I’d argue that is cheap, but in this case all I wanted was a local fire-and-forget backup of our primary documents directories that didn’t rely on me remembering to run rsync. And $60 was too much for just that.

Enter Windows FileHistory. I hadn’t heard of it until I searched again for Windows backup solutions. After a bit of fiddling about (tip: don’t include any folders that have a frequently-updated cache, like that for a web browser!) I seem to have it working just fine. It’s been going for a few days now, and is updating only what it needs to update which is pretty much all I want in a backup program!

However, I’m not entirely convinced that it’s a long-term solution as it seems a bit fragile. When configuring FileHistory, it seemed to get itself into a state where I couldn’t add new folders to the backup, and remove one would remove them all. A round of disabling it, reboot, and re-enabling FileHistory fixed it, but I shouldn’t have to do that every time. (I shouldn’t have to do it at all, of course…)

So we’ll see how it goes. I didn’t like the fact that CrashPlan forced me to create yet another account even to use the free version – my Windows 10 box uses local logins, so no additional accounts needed to use FileHistory. Plus I’ll keep doing my rsync backups to the Linux box, so at least I’ll always have a second copy. Mind you, even that backup drive is filling up…

Postscript: I should admit some fallibility here. My initial backup plan included a directory tree that mistakenly contained our web browser caches, which of course tend to a) be large and b) change frequently. (Not to mention that they are pointless to back up!) That meant every time the backup ran, it saw that a few GB of files had changed and duly did exactly what I’d asked it to do. Alas, this resulted in my paltry 160 GB drive filling up way sooner than expected. So one of the things to make sure when you set up a backup is to make sure it really is only backing up the files that you wish to keep! I know this now… 🙂

Feeding your algorithm

The other day I included the phrase “stupid Instagram” in the caption for a photo on Instagram. I was quite proud of the photo, and figured it was a decent contribution to Mountain Monday. I was expecting a steady stream of likes, probably topping out around 30-40 or so (which is typical for me). So imagine my surprise when nothing happened. I got a couple of likes early on, then one or two more and finally another couple from followers whom I suspect have relatively short follower lists.

And that was it. A grand total of 9 likes in nearly 48 hours, none of which were from what I would describe as my “mutual admiration circle”; people with whom I tend to exchange likes. I was baffled, but then I began thinking about the fact that Instagram’s feed is now driven algorithmically and is not just a timestream (which I would prefer), and I wondered if the fact that I’d written “stupid Instagram” meant that Instagram’s algorithm significantly down-weighted my post to the point where almost none of my followers saw it? Could Instagram really be that sensitive to even the slightest perceived criticism?

Now, of course I realize that many of the people I follow have really long lists of others they follow, so it could simply be a numbers game. And I have seen that happen in the past, but I still usually rack up 20-30 likes. (By comparison, my latest photo is up to 35 likes in less than three hours.)

So I will try another experiment in the near future. I might even post the same photo, at the same time of day with all the same tags and caption, only with the perceived slight on Instagram’s part removed. And I’ll see what happens.

A passing encounter with ransomware

It all began with the most innocuous of actions: a click. One of those many articles on Facebook that pops up in the “Related stories” area when you “like” an article. Next thing we had a web page showing all manner of error codes along with advice on how to fix the “problem” that had been detected, pop-up dialogs galore and the computer began beeping, like the alarm system on a science-fiction spaceship. It was almost comical.

Until I realized what was happening.

Then I had the “Oh shit – ransomware” moment, and lunged for the power button. (In retrospect I should have just tried to shut down the web browser from the task manager but I didn’t want to delay shutting down the computer if it was about to start encrypting our hard drives.)

That done, I thought for a moment, trying to assess the damage in the worst case that the hard drives had been encrypted. We figured that we might lose the past two weeks of photos and any edits. That wasn’t too much of an issue, really as we hadn’t cleared the memory cards in our cameras for over a month. We would probably lose a few recent PDFs and text files – annoying but not disastrous.

Having weighed up the situation, I retrieved some cables for reading bare hard drives and then set about dismantling the computer. First I detached the external hard drive used for continuous backup of a few folders (more on that later). Then I opened the case and removed the 3-TB main hard drive that contains all our photos and documents.

I booted up the Linux box and plugged in the external backup drive. It mounted just fine and I could read the files on there – at least as much as you can read the files in a CrashPlan backup. There was no sign of any recent modifications to the file system, so I figured that if something had started the encryption process, it hadn’t got to that drive yet (after all, it was down the list at F:).

I breathed a sigh of relief: it looked like we’d actually have up-to-date copies of our documents. Yay!

Next step was to attach the 3-TB drive. That’s when my heart started to sink. It mounted as an empty volume. No partitions, nothing. Wait, that’s not quite true. In actual fact it didn’t even mount – it just showed up as empty in the disk utility. Oh crap. Was that it? Had we actually lost the contents of that drive? I didn’t dare hope it was just some quirk.

Unplugging that drive, I moved on to the main system drive. Plugging that in was immediately more hopeful: a partition called “System Restore” mounted, and then I got an error message saying that the main partition couldn’t be mounted because Windows had been put in hibernate mode. Another light bulb moment, this time a positive one – I think – as I remembered I’d configured the power switch to hibernate rather than power off. OK, so the drive had gone into hibernation correctly. I mounted it read-only and lo-and-behold I was in business.

Poking around on the C: drive it appeared that everything was in order. I couldn’t find any nefarious-looking files (not that I really knew what to look for) with the right timestamp. So I decided to make a local backup, just in case. I used the tried-and-tested rsync to make a copy on one of the drives in the Linux box, which gave me time to do a bit of research about ransomware…

What I found gave me hope. It seemed that a couple of the most common types were in fact browser-based and weren’t real ransomware at all. I looked at screenshots to compare with the picture in my memory from the brief glimpse I took before hitting the power button, and decided that – if I was lucky – we’d most likely been hit with the TechBrolo.C Trojan installer. Furthermore, Windows Defender seemed to know about them and could remove them.

Progress of sorts, then. Once my local backup was done, I decided it was time to test those ideas, and got back down on my hands and knees to reattach the hard drives. Leaving the network cable unplugged, I tentatively hit the power switch. I was soon looking at a normal login screen – very promising, which told me we hadn’t been hit by real hard-drive-encrypting ransomware. I logged in and waited. Everything looked fine. I checked the 3 TB drive and was relieved to see that all our files were still there. I started up ACDSee to browse some photos just to be sure. All looked OK. Phew!

I restarted Firefox – which of course tried to load all the previously-open tabs. At least this time I was able to kill the tab with the offending payload, and looked through the history to see what website had spawned it. I copied the URL and deleted the entry from the history. I won’t give the URL as I would hate for someone else to click on it, but the site is in the .ml top-level domain (which is Mali, where Timbuktu is located).

Everything still looked fine, so I started up Windows Defender and set it to run a full scan. However, not long after it got under way, a warning sign did show up saying that a potential malicious file had been discovered.

When the scan ended (two-and-a-half hours later), the report did indeed show a single potential threat: the TechBrolo.E trojan, hiding in our Firefox cache. I clicked “Remove” and within moments it was gone. I re-ran a quick scan just to double check and it came back clean. Yay! I plugged in the network cable, started the task manager and waited. Nothing was happening: just an idle computer. Next up was Firefox again, and it too wasn’t doing anything out of the ordinary.

Now, remember that backup I did on my Linux box? Well, that included the Firefox cache, so I went in and had a look. Sure enough the file was a gzipped Javascript file with some stern warnings and more newlines than I’ve ever seen in my life! It looks like it tries to load a PHP file with an insane name, which I’m guessing is where the main payload is.

I wonder if the Javascript downloaded is country-dependent? I mean, showing a ransomware web page with a 1-8XX number won’t get you very far outside the US and Canada.

And so there I was a few hours later, and all was as well as it could be.

At the end of the day, all I can say is I’m glad I had that Linux computer, both as a second (and more importantly non-Windows) computer and as a backup for all the files we care about, as well as having the right cables to connect hard drives to a USB port. I’m also glad that I could shrug and say that at worst, we’d have lost only a couple of weeks’ documents and photos.

My take away from this? Have a backup of all the files you care about. And make sure that backup runs as often as you need it to. Every day, if necessary, though weekly is probably good enough. And watch what you click on…!

Terminal trees

A long time ago, back when I got my first Mac, I installed MacPorts to get a few extra packages that I used under Linux. Fast forward a few years (and a new laptop) and I’ve forgotten about all of this 🙂 When it came to looking for a way to get the tree command (which displays a directory listing in a tree format), one of the answers was to use MacPorts which prompted me to check and lo-and-behold, I did indeed have MacPorts installed.

A quick check showed that my installed version was waaaaay out of date so I needed to update that before going ahead with installing the package I was after. Thankfully, MacPorts has a nice little migration guide that helped me through the process. (This kind of documentation is vital to cover those instances of redoing something that you might only touch one or twice in the lifetime of a computer.)

Of course, I had to reinstall the Xcode command-line tools along the way… (Why they aren’t included by default I do not know.) I also trimmed my list of port installations to drop a couple I knew I wouldn’t need (such as Perl, which I manage separately with Perlbrew).

Then all I had to do was:
% sudo port install tree
refresh my path and I was good to go. Yay! (Ignore the spacing in the example below – it seems the preformatted text for this WP theme adds some extra padding which breaks up the connecting lines. It should look fine in the terminal.)

/Users/macports/
├── Desktop
├── Documents
├── Downloads
├── Library
│   ├── Keychains
│   │   └── ....
│   └── Preferences
├── Movies
├── Music
├── Pictures
└── Sites

It seems that folks tend to prefer HomeBrew over MacPorts but that’s something for another day.