Posted on

Butler SOS v2.0 – Real-time error and warning dashboards for Qlik Sense

This one is long overdue, but finally here: Butler SOS v2.0

The new version is an almost complete re-write of v1.0. Changes are plentiful and include

  • All warnings and errors stored by Sense in its log database are now pulled into Butler SOS, from where it can be graphed and acted upon.
    This is a big deal, as it was previously not possible to get notifications or alerts when errors or warnings started to pile up in the logs.
  • Operational health metrics are still pulled from Qlik Sense, but this is now done directly from the QIX engine rather than via a hard-to-secure virtual proxy.
  • Using certificates for authentication with Sense removes potential security issues with v1.0.
  • Config file is now YAML instead of JSON. More human readable and with inline comments.
  • Config file now allows for more fine-grained control of Butler SOS.
  • Several bugs fixed, especially around sending metrics to MQTT.

The readme file on GitHub has all the details, here are some screen shots to get you started though:

 

The new version is available on GitHub.

 

PS. This is blog post # 100. Yaaay!!!

Posted on

BigQuery ODBC driver now free from Google

 

A freebie for the weekend:

Looks like Google is now offering a free ODBC driver for BigQuery.

 

Google has licensed Simba’s driver, which is great! The Simba driver has proven very stable and performant, and being able to use it freely to access data in BigQuery is likely to benefit BigQuery adoption.

The Simba drivers support both Standard and Legacy SQL modes in BigQuery, as well as large result sets, and are confirmed to be working very nicely with Qlik Sense September 2017 version.

There are versions available for 32/64 bit Windows, OS X, and 32/64 bit Linux.

 

Posted on

How to set up free SSL certificates for Qlik Sense

I recently had a need for an isolated Qlik Sense environment, in order to test some of the new features of Qlik Sense September 2017.

While it works perfectly fine to run Sense with self-signed certificates, you then get browser warnings that the certificates are not valid etc. That might be fine, but as the test at hand involved testing my app duplicator service for Qlik Sense (which require a proper SSL cert) together with the September 2017 version of Sense, I needed a proper SSL certificate.

As I have good experiences using the free certificates of Let’s Encrypt (they secure this blog, for example) I thought it would be a good exercise figuring out how to use them together with Qlik Sense Enterprise.

The notes below are largely reminders-to-self, in case I need to do this again some day. Maybe they can also be useful for others out there.

Continue reading “How to set up free SSL certificates for Qlik Sense”

Posted on

Monitoring and auto-starting Node.js services on Windows Server

When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.

A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.

This leads us to the conclusion that the services must automatically be

a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.

Enter process monitors.

At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).

Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.

Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.

Continue reading “Monitoring and auto-starting Node.js services on Windows Server”

Posted on

Qlik Sense app duplicator 2.0 released

On the surface everything looks the same.

Under the hood everything has changed.

The 2.0 release of the Qlik Sense app duplicator is a major update. Most code has been refactored.

The most notable changes are:

  • Switched to using Enigma.js for talking to the Sense engine, instead of the previously used Qsocks library. Enigma is open source just like Qsocks, but the former is supported and actively developed by Qlik themselves – that is really important.
  • Much improved logging.
  • A nasty bug has been fixed.
    The app duplicator would fail semi-silently when multiple app duplication requests were sent to it simultaneously. No more of that – the duplicator now handles also high-load scenarios gracefully, and have been battle-tested in hands-on sessions when many developers created their own apps from templates at the same time. Nice.The 2.0 version is available on GitHub, as always.
Posted on

Butler SOS goes main stage at Qonnections 2017

Edit:

Qlik has made available a video from Anthony Deighton’s (Qlik’s CTO) keynote session.

Very much worth looking through the entire video – it was very interesting indeed. 
His highlight of customer and partner project’s is found 12 min 36 sec into the video.

 

 

A few weeks back something cool happened.

During the big, main stage keynote session at Qlik‘s annual Qonnections conference, my Butler SOS project was highlighted as an example of a great application built on top of the Sense APIs.

I have to admit that it is pretty neat to get that kind of recognition from industry peers… a bit of ego boost :).

And it makes it easier and even more fun to work out coming, similar concepts and services. Lots more to be done around DevOps and Qlik Sense.

Speaking of which:
I started playing around with Qlik’s new Enigma.js library a week or so ago, to see if it could be used in coming projects.

Very nice indeed. Nothing wrong with good old Qsocks, but Enigma is a step in the right direction. Just like qsocks, Enigma is open source – but also fully supported by Qlik. That is a huge advantage!

Posted on

Let there be (blinky) light!

I was recently in Helsinki, giving a talk at a QlikDevGroup event. Great event, great crowd. The topic was SenseOps and Butler SOS, and I showcased the lamp above as an example of a funky, but still relevant way to monitor user activity in a Qlik Sense Enterprise environment.

A person in the audience asked how the map works. I claimed it was super simple, costing less than USD 10 to build (assuming you already have a suitable enclosure) and uses just four wires hooked up between some pre-made modules. Time to prove it.

The four wires part might have been a slight exaggeration… but it depends on which wires you count – right?

Continue reading “Let there be (blinky) light!”

Posted on

Butler SOS: real-time server stats for Qlik Sense

I got somewhat distracted from the idea of breaking up the existing Butler software into smaller, stand-alone micro services.
Or rather, an idea came to mind. An idea too good not to explore…

The healthcheck API of Qlik Sense provides basic metrics for both the Qlik Sense engine itself, and the server it is running on. Things like CPU load, available RAM, number of connected users and what apps are loaded into the Sense engine.

The idea behind Butler SOS ( SOS = SenseOps Stats) is very simple:
Get the healthcheck metrics for all servers in a Sense cluster. Then send the information to MQTT for immediate, real-time use cases.
It is directly aimed at bringing better features to the monitoring step of SenseOps – please visit SenseOps.rocks for more info on SenseOps.

Butler SOS is nice and sending data via MQTT make the health metrics available in for example Node-RED. Node-RED has some basic graph options, but not anywhere near those offered by Grafana. Grafana is very, very cool… A live demo is available here – do check it out – it is very nice indeed.

Creating real-time dashboards in Grafana is greatly simplified if the data is stored in some kind of time series database. Influxdb is an obvious choice. It is open source, installation is very easy, and there are good Node.js libraries that make it trivial to insert data into a Influxdb database.
Thus – Butler SOS also sends the Sense health metrics to an Influx db of choice.

Only need Influxdb and not MQTT? Or the other way around?
No problem, the Butler SOS config file include options for independently turning on/off sending of data to MQTT and Influxdb.

Butler SOS, including a sample Grafana dashboard, is available on GitHub.

Using Grafana we get dashboards and charts like these:

Showing CPU load, available RAM, active users, cache hit ratio and more.
A 100% cache hit ratio (left y axis) means that no additional calculations were needed to serve the requested data to the client. The yellow dots are associated with the right y axis, which tells us how big the request in question was.

 

Continue reading “Butler SOS: real-time server stats for Qlik Sense”

Posted on

Butler-MQTT and a realtime reload dashboard for Sense

Teaser video: Realtime reload dashboard

Just to get your interest – this video shows what Butler-MQTT can be used for. All the pieces are included in the Github repo. The concept of a reload dashboard can be very useful if you have long running (hours!) reloads. Instead of relying on Sense’s own reload log window, you can get an easy to understand visual feedback from a reload dashboard.

The video shows

  1. A reloading Sense app (upper left). The app has two nested loops (date and country), with half a second delay in each.
  2. Within each loop a status message is sent to  Butler-MQTT (bottom left), which forwards the message to a MQTT server running on localhost.
  3. A Node-RED dashboard picks up the MQTT messages and render a dashboard showing the progress of the running reload.

The “Sales” line chart deserves some comment too.
In the outer loop in the app’s load script, the script calculates sum(sales) for that particular loop iteration. That value is then sent to a MQTT topic, and then plotted in the dashboard.
This way you can get an immediate, visual feedback on the actual data produced by the reload. Not by any means always needed – but it can also be very, very useful.

Continue reading “Butler-MQTT and a realtime reload dashboard for Sense”

Posted on

Icons for Qlik Sense apps

Developing apps on Qlik Sense is lots of fun, but also poses some challenges.
The look and feel of your Sense apps is really important when it comes to end users’ first impressions of Sense. Digging deeper into the possibilities of Sense, extensions are an interesting way to enhance the capabilities of your apps.

Once you start using extensions though, apps start to diverge in terms of visual look look and feel.
That might be fine, but then you have icons, which are used for buttons, to visually categorise data in tables, they are used in mashups etc.
Lacking a common set of icons and app/sheet thumbnails, developers will add their own icons to apps, eventually resulting in a non-consistent end user experience.
A lot could be gained by all Sense app developers in your organisation using the same set of icons.

Enter Font Awesome (=”FA”). It really is open source at its best:

“Font Awesome is fully open source and is GPL friendly. You can use it for commercial projects, open source projects, or really just about whatever you want.”
Font Awesome by Dave Gandy – http://fontawesome.io

My idea was to make it really simple to bring FA into the Sense domain. There are two parts to this: Making the icons available as general bitmaps, and as sheet/app thumbnails.
Turns out it is pretty easy. The basic steps are:

  1. Convert the FA icons to bitmaps (.png files).
  2. Create variants of the bitmap icons that have the aspect ratio needed for sheet/app thumbnails to look good.

Once again turning to the open source community, a couple of projects give us all we need: icon-font-to-png and imagemagick.

Assuming you are on OS X or Linux (sorry Windows users…), this GitHub script will download all 700+ icons from Font Awesome, convert them into 256×256 icons using any named color you pass on the command line, and also create a separate folder with 8:5 aspect ratio png images that can be used as app or sheet thumbnails.

Import into a couple of Sense resource libraries (one for the square icons, and one for the 8:5 ones) and you have a very comprehensive set of icons to use in your Sense apps.

An obvious improvement would be to automatically upload the icons to Sense using the QRS API – there is always room for improvement. Feel free to fork the repository and add the needed code.

Onward!