Teaser video: Realtime reload dashboard

Just to get your interest – this video shows what Butler-MQTT can be used for. All the pieces are included in the Github repo. The concept of a reload dashboard can be very useful if you have long running (hours!) reloads. Instead of relying on Sense’s own reload log window, you can get an easy to understand visual feedback from a reload dashboard.

The video shows

1. A reloading Sense app (upper left). The app has two nested loops (date and country), with half a second delay in each.
2. Within each loop a status message is sent to  Butler-MQTT (bottom left), which forwards the message to a MQTT server running on localhost.
3. A Node-RED dashboard picks up the MQTT messages and render a dashboard showing the progress of the running reload.

The “Sales” line chart deserves some comment too.

In the outer loop in the app’s load script, the script calculates sum(sales) for that particular loop iteration. That value is then sent to a MQTT topic, and then plotted in the dashboard.

This way you can get an immediate, visual feedback on the actual data produced by the reload. Not by any means always needed – but it can also be very, very useful.

Back to basics – breaking up Butler

Feedback on the Butler project has been generally positive. Some people however found Butler hard to configure.

I can understand this. Even though Butler comes with a default config file with most settings already filled in, you need to configure certificates, UDP servers etc. And maybe you don’t even plan to use all the features offered by Butler.

Combine this with a general desire to have smaller micro services that provide one (or few) services, I am starting an effort to break Butler up into multiple smaller projects.

I do not yet know where this will end. Maybe the big monolithic Butler will come back, or maybe the future is entire micro service based.
A major advantage of micro services is that we can use a process monitor like PM2 to  ensure that all services are always running, and start/stop individual services as needed. This has been  tested and works really well.

Butler-MQTT

The first step towards breaking up Butler into smaller pieces is Butler-MQTT, available on GitHub.

It is a small Node.js server that provides a single REST API for publishing messages to MQTT.
The GitHub repository also includes some Qlik Sense code for calling the API, pushing data from Sense load scripts to MQTT is now a matter of two (!) function calls.

As of this writing the repository contains the following:

• A Node.js server functioning as a bridge between Qlik Sense and MQTT.
• A .qvs file containing two Sense subroutines. One for initialising Butler-MQTT and one for publishing messages to MQTT topics.
• A very simple config file. Only two configuration settings need to be set for Butler-MQTT to work.
• A 1000 line test data set created using Mockaroo.
• A Sense app that drives the reload dashboard demo.
• A Node-RED flow that creates the actual dashboard, based on data coming in as MQTT messages. The flow is configured to use an MQTT server running on localhost.

The Node-RED part consists of a “flow”, where you visually connect modules together into a Node-RED app. In this case the app outputs data to a dashboard, which is also shown below.

Dependencies

The repository contains everything needed to test the realtime reload dashboard on a Windows computer (or as I do – on Windows 10 running on VirtualBox on an iMac). Some external software is however needed:

• Qlik Sense Desktop. Butler-MQTT works equally well with Sense Desktop and Enterprise though.
• MQTT server. Mosquitto works really well. Installing on Linux is trivial, as is OSX (using Homebrew). Windows users can also use Mosquitto, good guides are found here and here.
• Node-RED requires Node.js. Both are very easy to install on any (reasonable) platform.
• The Sense app included in the repository rely on a couple of data connections to be present. Depending on where you extract the downloaded Butler-MQTT repository, you might need to edit the data connections in the Sense demo app.

Configuration

There is a template config file available in the repo at config/default_template.json. Make a copy of it, rename to default.json and edit as needed. The format is very simple:

{ 
  Butler-MQTT: { 
    mqttConfig: { 
      brokerIP: "<IP of MQTT server>" 
    }, 
    restServerConfig: { 
      serverPort: 8081 
    } 
  } 
}

In the video above I run everything in VirtualBox on a Mac, brokerIP would then be localhost.

In order to set up the Butler-MQTT Node.js service, cd into the directory where the package.json file is located, then run “npm i” (or possibly “sudo npm i” on Linux/OSX).

Start the Butler-MQTT service by running “node butler-mqtt.js”. Or use the PM2 process monitor to handle the life cycle of Butler-MQTT.

Go crazy!

Node-RED is very, very cool…

While it might not qualify as verified for mission critical purposes, I have used it for many months without any major issues. It is stable, feature rich and has a rich library of third party modules. Generic HTML modules allow you to embed almost any HTML/CSS code you find online.

So, with an easy to use link between Sense and Node-RED using MQTT, it suddenly becomes quite easy to send SMS from the load script, have your Amazon Echo device speak alerts or status messages during script reload, send messages to Slack etc etc etc.

Go crazy, experiment, hack and have fun!
Feel free to report back on what solutions you put together, of course – sharing is caring.

Happy hacking!