Following up on previous posts (here and here) about the Butler family of tools being Dockerized, here is another one on the same topic:
Butler SOS can now be run in a Docker container.
This is good news as it makes it a lot easier to set up real-time monitoring of a Qlik Sense enterprise environment, compared to the previous (still working, btw) method of installing Node.js and then running Butler SOS on top of Node.
Before the summer the Butler tool turned two years old – time flies!
Over those years I have installed, tweaked and upgraded a fair number of Butler instances… Not a problem per se, but maintaining a production grade Butler instance does assume a certain level of experience around Node.js, Linux, networking etc.
The most recent Butler version (v2.2) attempts to make it easier to deploy and operate Butler. This is achieved by deploying Butler as a Docker container instead of a regular Node.js app.
The Docker image (from which a container is created) contains exactly the same Node.js app that you can run right on your server or laptop – i.e. there is no functional difference what so ever between running the Node app natively, and running it as a Docker container.
There are some significant benefits of running Butler under Docker:
When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.
A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.
This leads us to the conclusion that the services must automatically be
a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.
Enter process monitors.
At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).
Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.
Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.
Switched to using Enigma.js for talking to the Sense engine, instead of the previously used Qsocks library. Enigma is open source just like Qsocks, but the former is supported and actively developed by Qlik themselves – that is really important.
Much improved logging.
A nasty bug has been fixed.
The app duplicator would fail semi-silently when multiple app duplication requests were sent to it simultaneously. No more of that – the duplicator now handles also high-load scenarios gracefully, and have been battle-tested in hands-on sessions when many developers created their own apps from templates at the same time. Nice.The 2.0 version is available on GitHub, as always.
I was recently in Helsinki, giving a talk at a QlikDevGroup event. Great event, great crowd. The topic was SenseOps and Butler SOS, and I showcased the lamp above as an example of a funky, but still relevant way to monitor user activity in a Qlik Sense Enterprise environment.
A person in the audience asked how the map works. I claimed it was super simple, costing less than USD 10 to build (assuming you already have a suitable enclosure) and uses just four wires hooked up between some pre-made modules. Time to prove it.
The four wires part might have been a slight exaggeration… but it depends on which wires you count – right?
For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?
I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….
The more I looked at it, the more I felt “wow – SenseOps really rocks!”
For a long time I wanted to learn more about client side development.
Javascipt, CSS, reactive frameworks etc – all those concepts that make today’s web sites tick. I really think backend developers can benefit from having at least a basic understanding of how the front-end works – this will make the finished app better as a whole.
I worked on a small node.js server during the past couple of weeks. The idea came from the fact that
I over and over again find myself creating Sense apps that are almost identical, and
people starting out as Sense developers spend too much time learning the basics. I wanted to bootstrap their learning process by providing well written skeleton apps for them.
This could of course be achieved by just duplicating apps using the QMC, but it would be way better if there was a nice little web app that listed the available Sense app templates, and allowed anyone (with permissions to create apps) to create new apps based on them. Or maybe this feature could even be integrated into one of the different Sense hubs that are now available…