Wow – after 10 years and one day at Spotify the time has come to take the next step.
Going forward I will work in my own company, Ptarmigan Labs AB.
The company’s focus will be more or less the same as what I have spent the past 6-7 years doing at Spotify, i.e. helping people, teams and organisations understanding, enriching and making use of their data.
Qlik Sense will remain my main focus, the Qlikosphere is really heating up with lots of interesting new features both launched and around the corner.
During the past 12 months Qlik Sense has really taken great steps towards becoming a proper enterprise grade BI platform.
Given this development I am really looking forward to Qonnections 2018 and the announcements that are likely to happen there. Interesting times!
You can reach me via info <at> ptarmiganlabs.com, or on LinkedIn.
My open source projects are found on GitHub, as always.
Last weekend I finally got around to doing something I have postponed for several years now.
Being somewhat of a gadget hoarder, I have ended up with quite a large stack of rechargeable batteries. Everything from 12 Volt lead-acid ones from old UPS:s to Lithium-ion variants from mobile phones.
The last kind are actually pretty interesting. They pack a significant punch when it comes to energy storage, the problem is that they are designed to go into cell phones (at least pre iPhone ones where you could remove the battery!), and there are thus no separate charging docks or battery holders.
Still, it would be sweet if they could be used to power electronics gadgets that I build myself… It is of course possible, and even quite easy to do – here is the very first prototype in action:
Google has licensed Simba’s driver, which is great! The Simba driver has proven very stable and performant, and being able to use it freely to access data in BigQuery is likely to benefit BigQuery adoption.
The Simba drivers support both Standard and Legacy SQL modes in BigQuery, as well as large result sets, and are confirmed to be working very nicely with Qlik Sense September 2017 version.
There are versions available for 32/64 bit Windows, OS X, and 32/64 bit Linux.
I recently had a need for an isolated Qlik Sense environment, in order to test some of the new features of Qlik Sense September 2017.
While it works perfectly fine to run Sense with self-signed certificates, you then get browser warnings that the certificates are not valid etc. That might be fine, but as the test at hand involved testing my app duplicator service for Qlik Sense (which require a proper SSL cert) together with the September 2017 version of Sense, I needed a proper SSL certificate.
As I have good experiences using the free certificates of Let’s Encrypt (they secure this blog, for example) I thought it would be a good exercise figuring out how to use them together with Qlik Sense Enterprise.
The notes below are largely reminders-to-self, in case I need to do this again some day. Maybe they can also be useful for others out there.
When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.
A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.
This leads us to the conclusion that the services must automatically be
a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.
Enter process monitors.
At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).
Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.
Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.
Switched to using Enigma.js for talking to the Sense engine, instead of the previously used Qsocks library. Enigma is open source just like Qsocks, but the former is supported and actively developed by Qlik themselves – that is really important.
Much improved logging.
A nasty bug has been fixed.
The app duplicator would fail semi-silently when multiple app duplication requests were sent to it simultaneously. No more of that – the duplicator now handles also high-load scenarios gracefully, and have been battle-tested in hands-on sessions when many developers created their own apps from templates at the same time. Nice.The 2.0 version is available on GitHub, as always.
I was recently in Helsinki, giving a talk at a QlikDevGroup event. Great event, great crowd. The topic was SenseOps and Butler SOS, and I showcased the lamp above as an example of a funky, but still relevant way to monitor user activity in a Qlik Sense Enterprise environment.
A person in the audience asked how the map works. I claimed it was super simple, costing less than USD 10 to build (assuming you already have a suitable enclosure) and uses just four wires hooked up between some pre-made modules. Time to prove it.
The four wires part might have been a slight exaggeration… but it depends on which wires you count – right?
I got somewhat distracted from the idea of breaking up the existing Butler software into smaller, stand-alone micro services.
Or rather, an idea came to mind. An idea too good not to explore…
The healthcheck API of Qlik Sense provides basic metrics for both the Qlik Sense engine itself, and the server it is running on. Things like CPU load, available RAM, number of connected users and what apps are loaded into the Sense engine.
The idea behind Butler SOS ( SOS = SenseOps Stats) is very simple:
Get the healthcheck metrics for all servers in a Sense cluster. Then send the information to MQTT for immediate, real-time use cases.
It is directly aimed at bringing better features to the monitoring step of SenseOps – please visit SenseOps.rocks for more info on SenseOps.
Butler SOS is nice and sending data via MQTT make the health metrics available in for example Node-RED. Node-RED has some basic graph options, but not anywhere near those offered by Grafana. Grafana is very, very cool… A live demo is available here – do check it out – it is very nice indeed.
Creating real-time dashboards in Grafana is greatly simplified if the data is stored in some kind of time series database. Influxdb is an obvious choice. It is open source, installation is very easy, and there are good Node.js libraries that make it trivial to insert data into a Influxdb database.
Thus – Butler SOS also sends the Sense health metrics to an Influx db of choice.
Only need Influxdb and not MQTT? Or the other way around?
No problem, the Butler SOS config file include options for independently turning on/off sending of data to MQTT and Influxdb.