Butler DevOps tools for Qlik Sense running on Raspberry Pi based Kubernetes cluster

Lots of buzzwords in that title, but yes – things have now come together and the main members of the Butler family of tools for Qlik Sense now run as Docker containers on Arm64 and even Arm7.

In practice this means it is possible to run Butler SOS, Butler Cache Warming etc as Docker containers on for example a Raspberry Pi 4. If you are worried about that poor little Raspi failing you can add a bunch of them into a Kubernetes cluster, and then run the Butler tools there.

This actually works pretty well, here is a Raspberry Pi based Kubernetes (a.k.a “k8s”) cluster built using the excellent k3s distribution from Rancher, running the Butler tools:

Butler tools running Raspberry Pi based k3s cluster.
Butler tools running as Docker containers in Raspberry Pi based Kubernetes cluster.

And here is the Raspi cluster itself, with 4 of the 7 nodes installed:

Kubernetes cluster built on Raspberry Pis
Kubernetes cluster built on Raspberry Pis

Ok, ok – I hear you – is it really a good idea to run close to mission critical software on Raspberry Pi’s?
No of course not, but the cloud providers are starting to offer Arm based virtual machines (Amazon does this already with their EC2 A1 instances), with roughly twice the performance per dollar compared to Intel based VMs.

So running Butler on Arm VMs might be a way to save some money, if the cloud provider you use happens to offer Arm VMs.

Or just surrender to your inner geek and build your own Raspi k8s cluster… Lots of fun and a great way to learn about k8s!

Docker images available on Docker Hub, as always.

Fixing bash Shellshock vulnerability on Raspberry Pi

The recent bash vulnerability, a.k.a. “Shellshock”, is pretty bad, considering it might actually have been around for a very long time, maybe even dating back to the predecessor of bash. Not good.

So what about Raspberry Pi’s?
Are they vulnerable?

Turns out they are, but there is already a fix available for them and patching a Raspi is very simple. Whether your actual Raspi is vulnerable depends on what distribution you are using, and how recently you upgraded the software in it.
The Raspi used below is running IPE-R1, which is a blackout-proof version of Raspian.

First let’s find out what bash version we have:

root@raspi-2:~# dpkg -s bash | grep Version
Version: 4.2+dfsg-0.1
root@raspi-2:~#

You can also run this little script to determine whether your Raspi is vulnerable to Shellshock

root@raspi-2:~# env x='() { :;}; echo "WARNING: SHELLSHOCK DETECTED"' bash --norc -c ':' 2>/dev/null;
WARNING: SHELLSHOCK DETECTED
root@raspi-2:~#

Let’s fix this. Just refresh the repos and upgrade bash (the patched version is available in the main repos).

root@raspi-2:~# apt-get update && apt-get install --only-upgrade bash
Get:1 http://archive.raspberrypi.org wheezy Release.gpg [490 B]
Get:2 http://mirrordirector.raspbian.org wheezy Release.gpg [490 B]
Get:3 http://archive.raspberrypi.org wheezy Release [10.2 kB]
...
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
 bash-doc
The following packages will be upgraded:
 bash
1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.
Need to get 1,443 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://mirrordirector.raspbian.org/raspbian/ wheezy/main bash armhf 4.2+dfsg-0.1+deb7u3 [1,443 kB]
Fetched 1,443 kB in 1s (1,386 kB/s)
(Reading database ... 29754 files and directories currently installed.)
Preparing to replace bash 4.2+dfsg-0.1 (using .../bash_4.2+dfsg-0.1+deb7u3_armhf.deb) ...
Unpacking replacement bash ...
Processing triggers for man-db ...
Setting up bash (4.2+dfsg-0.1+deb7u3) ...
update-alternatives: using /usr/share/man/man7/bash-builtins.7.gz to provide /usr/share/man/man7/builtins.7.gz (builtins.7.gz) in auto mode
root@raspi1:~#

The installed version of bash is now +deb7u3

root@raspi-2:~# dpkg -s bash | grep Version
Version: 4.2+dfsg-0.1+deb7u3
root@raspi-2:~#

The short test script above also returns nothing:

root@raspi-2:~# env x='() { :;}; echo "WARNING: SHELLSHOCK DETECTED"' bash --norc -c ':' 2>/dev/null;
root@raspi-2:~#

You could of course also just do an “apt-get upgrade” to upgrade all packages on your Raspi, would take a bit longer but will work just as well.
Also, if you are not logged in as root you need to do a “sudo apt-get update”, of course.

Raspberry Pi won’t boot – only red power light comes on – SD card corrupt? Not so fast…

After having problems with SD cards failing after a few months in a Raspi that was supposed to be always-on, it was time to look for options.

The IPE distribution seemed like a perfect match – it effectively makes the Raspi only use a ramdisk, except possibly during boot. When needed the SD card file system can be made writeable, meaning that apt-get etc can be used as usual to install apps.

This worked great for ca 3 weeks, then this setup ALSO failed, with the same symptoms as before: at boot only the red power light would come on – nothing else. A fresh SD card with a fresh copy of IPE worked fine in the same Raspi hardware, thus not the Raspi itself that had fried (the cabinet where it lives is kind of warm, nothing too crazy though).

Now the weirdness starts for real. The failing SD card works perfectly fine when inserted in both an iMac and a Windows laptop. I spent hours and hours reformatting the card with various low-level tools, then writing the standard Raspian image to the card. Just for reference I did the same to a third SD card too, that card was identical to the failing one.

Still the same issue: that one card would still prevent the Raspi from booting.

The card in question is a Kingston 16 GB microSDHC card marked SDC4/16GB 081. It comes with an adapter that makes it usable also in large SD card connectors.

Photo on 11-09-14 at 06.52 #2

Finally, and I of course should have thought of this sooner, I tried swapping the two different microSD to SD adaptors. And of course… the solution was simple. It was not the SD card that had failed, it was the adapter. For some reason (heat? just poor quality?) the adapter had a small crack in its plastic housing, and as a result most likely some poor connection inside, between the microSD card itself and the surrounding adapter. When inserted into a proper computer, there was probably enough pressure on the adapter to squeeze it against the actual microSD card, while the SD card holder on the Raspi is kind of flimsy, and did not provide the needed pressure. When using a working adapter all three microSD cards work flawlessly in the Raspi.

Note to self: Use full sized SD cards in Rasperry Pis….