butler

Hey Butler, please meet Docker

Göran Sander

Sep 26, 2018 • 5 min read

Before the summer the Butler tool turned two years old – time flies!

Over those years I have installed, tweaked and upgraded a fair number of Butler instances… Not a problem per se, but maintaining a production grade Butler instance does assume a certain level of experience around Node.js, Linux, networking etc.

The most recent Butler version (v2.2) attempts to make it easier to deploy and operate Butler.

This is achieved by deploying Butler as a Docker container instead of a regular Node.js app.

The Docker image (from which a container is created) contains exactly the same Node.js app that you can run right on your server or laptop – i.e. there is no functional difference what so ever between running the Node app natively, and running it as a Docker container.

There are some significant benefits of running Butler under Docker:

No need to install Node.js on your server(s)
Make use of your existing Docker runtime environments, or use those offered by Amazon, Google, Microsoft etc
Benefit from the extremely comprehensive tools ecosystem (monitoring, deployment etc) that is available for Docker
Updating Butler to the latest version is as easy as stopping the Butler container, doing a “docker pull ptarmiganlabs/butler:latest” and then starting the container again.
Fewer dependencies on whether you run Butler on Windows, Linux, cloud servers etc. With Docker, things usually just work.

Continuous Integration – CI

As mentioned, the ecosystem around Docker is amazingly rich. This means it is possible (even trivial) to automate creation of new Docker images when the GitHub project updates.

After reviewing several different free online CI (continuous integration) tools I settled with Codefresh. Some other services had more generous free tiers, but Codefresh really shines when it comes to ease of use, easy to get started and integration with things like Github and Docker Hub.

Creating a Codefresh pipeline just takes a few clicks, but results in

Automatic building of a new Docker image every time a new release of Butler is done on Github.
The new image is automatically tagged and pushed to Docker Hub. From there the latest version can be retrieved by anyone, by simple running “docker pull ptarmiganlabs/butler:latest”

Right – that link above is a referral link. Feel free to use it if you want to try out Codefresh, as it gives me some extra builds with them each month.
Those builds will come in handy when I keep working on the Butler tools. Thanks!

Hands on with Docker

Let’s take a look at what it looks like setting up Butler as a Docker container. The commands below are done on a Mac, but will be very similar on Linux and Windows. The instructions are also available on the Butler documentation site.

First let’s create some directories where the Butler config file and Sense certificates will be stored:

proton:~ goran$ mkdir /Users/goran/butler
proton:~ goran$ cd /Users/goran/butler
proton:butler goran$ mkdir -p config/certificate

Next, let’s pull down the docker-compose.yml file from the Butler Github repository. This file tells Docker how to create a container from a Docker image, and how to configure the container.

Let’s also take a look at what the docker-compose.yml looks like.

proton:butler goran$ wget https://raw.githubusercontent.com/ptarmiganlabs/butler/master/src/docker-compose.yml
--2018-09-26 16:27:14-- https://raw.githubusercontent.com/ptarmiganlabs/butler/master/src/docker-compose.yml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.84.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.84.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 565 [text/plain]
Saving to: 'docker-compose.yml.1'

docker-compose.yml.1 100%[========================================================================================================================================>] 565 --.-KB/s in 0s

2018-09-26 16:27:15 (6.34 MB/s) - 'docker-compose.yml.1' saved [565/565]

proton:butler goran$ cat docker-compose.yml
# docker-compose.yml
version: '2.2'
services:
verisure-mqtt:
image: ptarmiganlabs/butler:latest
container_name: butler
restart: always
ports:
- "8180:8080" # REST API available on port 8180 to services outside the container
- "9997:9997" # UDP port for session connection events
- "9998:9998" # UDP port for task failure events
volumes:
# Make config file accessible outside of container
- "./config:/nodeapp/config"
environment:
- "NODE_ENV=production"
logging:
driver: json-file
proton:butler goran$

Now it’s time to copy the certificates you have exported from the Qlik Sense QMC into the config/certificate folder.

You should also make a copy of the template configuration file available in the Github repository. Place the file in the config directory, rename it to “production.yaml” and edit it as needed for your local environment/systems.

You now have these files:

proton:butler goran$ pwd
/Users/goran/butler
proton:butler goran$ ls -la
total 8
drwxr-xr-x   4 goran  staff   128 Sep 26 16:36 .
drwxr-xr-x+ 59 goran  staff  1888 Sep 26 16:24 ..
drwxr-xr-x   4 goran  staff   128 Sep 26 16:36 config
-rw-r--r--   1 goran  staff   565 Sep 26 16:25 docker-compose.yml
proton:butler goran$
proton:butler goran$ ls -la config/
total 8
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 .
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 ..
drwxr-xr-x  6 goran  staff   192 Sep 26 16:36 certificate
-rw-r--r--  1 goran  staff  1861 Sep 26 16:36 production.yaml
proton:butler goran$
proton:butler goran$ ls -la config/certificate/
total 32
drwxr-xr-x  6 goran  staff   192 Sep 26 16:36 .
drwxr-xr-x  4 goran  staff   128 Sep 26 16:36 ..
-rw-r--r--@ 1 goran  staff  1166 Sep 26 16:36 client.pem
-rw-r--r--@ 1 goran  staff  1702 Sep 26 16:36 client_key.pem
-rw-r--r--@ 1 goran  staff  1192 Sep 26 16:36 root.pem
proton:butler goran$

Almost there!

Run “docker-compose up” to create and start the container:

proton:butler goran$ docker-compose up
Pulling butler (ptarmiganlabs/butler:latest)...
latest: Pulling from ptarmiganlabs/butler
f189db1b88b3: Pull complete
3d06cf2f1b5e: Pull complete
687ebdda822c: Pull complete
99119ca3f34e: Pull complete
e771d6006054: Pull complete
b0cc28d0be2c: Pull complete
7225c154ac40: Pull complete
7659da3c5093: Pull complete
32189a059676: Pull complete
e5e9e893e38f: Pull complete
0e4e951cc7f3: Pull complete
22a23c1ada03: Pull complete
4ad2d8e68c8e: Pull complete
4f5cfbe37ef7: Pull complete
79c448ba92be: Pull complete
Digest: sha256:7b69a49e46677a3d8528c4a9bb29f4b82ebadcc9b708341e7d16f5ab31051ed7
Status: Downloaded newer image for ptarmiganlabs/butler:latest
Creating butler ... done
Attaching to butler
butler    | 2018-09-26T19:02:01.847Z - debug: Server for UDP server: localhost
butler    | 2018-09-26T19:02:01.851Z - info: REST server listening on http://[::]:8080
butler    | 2018-09-26T19:02:01.871Z - info: UDP server listening on 127.0.0.1:9997
butler    | 2018-09-26T19:02:01.871Z - info: UDP server listening on 127.0.0.1:9998
butler    | 2018-09-26T19:02:01.875Z - info: Connected to MQTT server 192.168.1.51:1884, with client ID mqttjs_d3b2b024
butler    | 2018-09-26T19:02:01.918Z - info: MQTT message received
butler    | 2018-09-26T19:02:01.918Z - info: qliksense/notification/handleExecutionResult/bffb268f-f87f-41c8-a5e7-f80ba6106ad4
butler    | 2018-09-26T19:02:01.919Z - info: bffb268f-f87f-41c8-a5e7-f80ba6106ad4

That’s it – you now have a fully functioning Butler instance running. The REST API is available on http://localhost:8180 (assuming you ran the above commands on your local computer). We can test this by hitting the ping endpoint using cURL (open a new terminal and run the command there):

proton:~ goran$ curl "http://localhost:8180/v2/butlerPing"
{"response":"Butler reporting for duty"}
proton:~ goran$

Mission accomplished.

Next in turn is to do the same exercise for the rest of the members of the Butler family… Stay tuned.