This one is long overdue, but finally here: Butler SOS v2.0
The new version is an almost complete re-write of v1.0. Changes are plentiful and include
All warnings and errors stored by Sense in its log database are now pulled into Butler SOS, from where it can be graphed and acted upon. This is a big deal, as it was previously not possible to get notifications or alerts when errors or warnings started to pile up in the logs.
Operational health metrics are still pulled from Qlik Sense, but this is now done directly from the QIX engine rather than via a hard-to-secure virtual proxy.
Using certificates for authentication with Sense removes potential security issues with v1.0.
Config file is now YAML instead of JSON. More human readable and with inline comments.
Config file now allows for more fine-grained control of Butler SOS.
Several bugs fixed, especially around sending metrics to MQTT.
The readme file on GitHub has all the details, here are some screen shots to get you started though:
When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.
A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.
This leads us to the conclusion that the services must automatically be
a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.
Enter process monitors.
At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).
Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.
Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.
Switched to using Enigma.js for talking to the Sense engine, instead of the previously used Qsocks library. Enigma is open source just like Qsocks, but the former is supported and actively developed by Qlik themselves – that is really important.
Much improved logging.
A nasty bug has been fixed.
The app duplicator would fail semi-silently when multiple app duplication requests were sent to it simultaneously. No more of that – the duplicator now handles also high-load scenarios gracefully, and have been battle-tested in hands-on sessions when many developers created their own apps from templates at the same time. Nice.The 2.0 version is available on GitHub, as always.
I was recently in Helsinki, giving a talk at a QlikDevGroup event. Great event, great crowd. The topic was SenseOps and Butler SOS, and I showcased the lamp above as an example of a funky, but still relevant way to monitor user activity in a Qlik Sense Enterprise environment.
A person in the audience asked how the map works. I claimed it was super simple, costing less than USD 10 to build (assuming you already have a suitable enclosure) and uses just four wires hooked up between some pre-made modules. Time to prove it.
The four wires part might have been a slight exaggeration… but it depends on which wires you count – right?
I got somewhat distracted from the idea of breaking up the existing Butler software into smaller, stand-alone micro services.
Or rather, an idea came to mind. An idea too good not to explore…
The healthcheck API of Qlik Sense provides basic metrics for both the Qlik Sense engine itself, and the server it is running on. Things like CPU load, available RAM, number of connected users and what apps are loaded into the Sense engine.
The idea behind Butler SOS ( SOS = SenseOps Stats) is very simple:
Get the healthcheck metrics for all servers in a Sense cluster. Then send the information to MQTT for immediate, real-time use cases.
It is directly aimed at bringing better features to the monitoring step of SenseOps – please visit SenseOps.rocks for more info on SenseOps.
Butler SOS is nice and sending data via MQTT make the health metrics available in for example Node-RED. Node-RED has some basic graph options, but not anywhere near those offered by Grafana. Grafana is very, very cool… A live demo is available here – do check it out – it is very nice indeed.
Creating real-time dashboards in Grafana is greatly simplified if the data is stored in some kind of time series database. Influxdb is an obvious choice. It is open source, installation is very easy, and there are good Node.js libraries that make it trivial to insert data into a Influxdb database.
Thus – Butler SOS also sends the Sense health metrics to an Influx db of choice.
Only need Influxdb and not MQTT? Or the other way around?
No problem, the Butler SOS config file include options for independently turning on/off sending of data to MQTT and Influxdb.
Just to get your interest – this video shows what Butler-MQTT can be used for. All the pieces are included in the Github repo. The concept of a reload dashboard can be very useful if you have long running (hours!) reloads. Instead of relying on Sense’s own reload log window, you can get an easy to understand visual feedback from a reload dashboard.
The video shows
A reloading Sense app (upper left). The app has two nested loops (date and country), with half a second delay in each.
Within each loop a status message is sent to Butler-MQTT (bottom left), which forwards the message to a MQTT server running on localhost.
A Node-RED dashboard picks up the MQTT messages and render a dashboard showing the progress of the running reload.
The “Sales” line chart deserves some comment too.
In the outer loop in the app’s load script, the script calculates sum(sales) for that particular loop iteration. That value is then sent to a MQTT topic, and then plotted in the dashboard.
This way you can get an immediate, visual feedback on the actual data produced by the reload. Not by any means always needed – but it can also be very, very useful.
For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?
I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….
The more I looked at it, the more I felt “wow – SenseOps really rocks!”
Real-time monitoring of failed reload tasks, with notifications sent as emails, to Slack channels and as MQTT messages
Starting Sense reload tasks from the load script or from external systems, using a REST API
Posting messages to Slack from the reload script. Great for sysadmin notifications as well as notifying end users that new data is available in an app
Sending MQTT pub-sub messages from the load script
Get full metadata, including load script, for any app
Getting free disk info for the Sense server
All in all, Butler 2.0 includes a set of features that offers greatly enhanced features for anyone involved in the operation of a Sense Enterprise environment.
Butler is available via Qlik Branch, with all source code hosted on GitHub. Documentation is available on github.io. That documentation is the best starting point for learning more about what Butler does and how to use it.
Some sample screen shots shows what is possible to achieve:
The goal of the previously described Slack proxy project was to allow posting of messages from Qlik Sense load scripts to Slack (great instant messaging platform!). The project has however developed into something larger and slightly more ambitious, integrating other connections than Slack. The original project name “Slack proxy” has become less and less relevant, and the project has thus been renamed to “Butler”.
As of right now (May 2016), Butler’s feature set include
Send messages from Sense load script to Slack
Send/publish MQTT messages from Sense load script (i.e. outbound MQTT).
Sense reload failures as emails, and to Slack.
Sense audit events (session start/stop and connection open/close) to Slack and to MQTT messages.
Create new directories on the Sense server’s (where Butler is running) disks.
Get disk space info, for disks on the Sense server where Butler is running.