Author Archives: mountaindude

About mountaindude

Tinkerer from age 7, travelled through business, management and back to engineering, programming and systems development for a living, and electronics as a hobby. Outdoor geek. Father and husband.

Butler SOS: real-time server stats for Qlik Sense

I got somewhat distracted from the idea of breaking up the existing Butler software into smaller, stand-alone micro services.
Or rather, an idea came to mind. An idea too good not to explore…

The healthcheck API of Qlik Sense provides basic metrics for both the Qlik Sense engine itself, and the server it is running on. Things like CPU load, available RAM, number of connected users and what apps are loaded into the Sense engine.

The idea behind Butler SOS ( SOS = SenseOps Stats) is very simple:
Get the healthcheck metrics for all servers in a Sense cluster. Then send the information to MQTT for immediate, real-time use cases.
It is directly aimed at bringing better features to the monitoring step of SenseOps – please visit SenseOps.rocks for more info on SenseOps.

Butler SOS is nice and sending data via MQTT make the health metrics available in for example Node-RED. Node-RED has some basic graph options, but not anywhere near those offered by Grafana. Grafana is very, very cool… A live demo is available here – do check it out – it is very nice indeed.

Creating real-time dashboards in Grafana is greatly simplified if the data is stored in some kind of time series database. Influxdb is an obvious choice. It is open source, installation is very easy, and there are good Node.js libraries that make it trivial to insert data into a Influxdb database.
Thus – Butler SOS also sends the Sense health metrics to an Influx db of choice.

Only need Influxdb and not MQTT? Or the other way around?
No problem, the Butler SOS config file include options for independently turning on/off sending of data to MQTT and Influxdb.

Butler SOS, including a sample Grafana dashboard, is available on GitHub.

Using Grafana we get dashboards and charts like these:

Showing CPU load, available RAM, active users, cache hit ratio and more.

A 100% cache hit ratio (left y axis) means that no additional calculations were needed to serve the requested data to the client. The yellow dots are associated with the right y axis, which tells us how big the request in question was.

 

Continue reading

Butler-MQTT and a realtime reload dashboard for Sense

Teaser video: Realtime reload dashboard

Just to get your interest – this video shows what Butler-MQTT can be used for. All the pieces are included in the Github repo. The concept of a reload dashboard can be very useful if you have long running (hours!) reloads. Instead of relying on Sense’s own reload log window, you can get an easy to understand visual feedback from a reload dashboard.

The video shows

  1. A reloading Sense app (upper left). The app has two nested loops (date and country), with half a second delay in each.
  2. Within each loop a status message is sent to  Butler-MQTT (bottom left), which forwards the message to a MQTT server running on localhost.
  3. A Node-RED dashboard picks up the MQTT messages and render a dashboard showing the progress of the running reload.

The “Sales” line chart deserves some comment too.
In the outer loop in the app’s load script, the script calculates sum(sales) for that particular loop iteration. That value is then sent to a MQTT topic, and then plotted in the dashboard.
This way you can get an immediate, visual feedback on the actual data produced by the reload. Not by any means always needed – but it can also be very, very useful.

Continue reading

Icons for Qlik Sense apps

Developing apps on Qlik Sense is lots of fun, but also poses some challenges.
The look and feel of your Sense apps is really important when it comes to end users’ first impressions of Sense. Digging deeper into the possibilities of Sense, extensions are an interesting way to enhance the capabilities of your apps.

Once you start using extensions though, apps start to diverge in terms of visual look look and feel.
That might be fine, but then you have icons, which are used for buttons, to visually categorise data in tables, they are used in mashups etc.
Lacking a common set of icons and app/sheet thumbnails, developers will add their own icons to apps, eventually resulting in a non-consistent end user experience.
A lot could be gained by all Sense app developers in your organisation using the same set of icons.

Enter Font Awesome (=”FA”). It really is open source at its best:

“Font Awesome is fully open source and is GPL friendly. You can use it for commercial projects, open source projects, or really just about whatever you want.”
Font Awesome by Dave Gandy – http://fontawesome.io

My idea was to make it really simple to bring FA into the Sense domain. There are two parts to this: Making the icons available as general bitmaps, and as sheet/app thumbnails.
Turns out it is pretty easy. The basic steps are:

  1. Convert the FA icons to bitmaps (.png files).
  2. Create variants of the bitmap icons that have the aspect ratio needed for sheet/app thumbnails to look good.

Once again turning to the open source community, a couple of projects give us all we need: icon-font-to-png and imagemagick.

Assuming you are on OS X or Linux (sorry Windows users…), this GitHub script will download all 700+ icons from Font Awesome, convert them into 256×256 icons using any named color you pass on the command line, and also create a separate folder with 8:5 aspect ratio png images that can be used as app or sheet thumbnails.

Import into a couple of Sense resource libraries (one for the square icons, and one for the 8:5 ones) and you have a very comprehensive set of icons to use in your Sense apps.

An obvious improvement would be to automatically upload the icons to Sense using the QRS API – there is always room for improvement. Feel free to fork the repository and add the needed code.

Onward!

Sense + DevOps = SenseOps

For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?

Some initiatives are starting to appear. Qlik themselves are open sourcing microservices such as the Governed Metrics Service , as well as open sourcing key libraries that make it easier to use Sense as a general purpose analytics platform.  I have a couple of contributions in this space too, with the Butler and App duplicator microservices.

I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….

 

SenseOps

 

The more I looked at it, the more I felt “wow – SenseOps really rocks!”

Thus, please meet a new site: senseops.rocks

Head over for a look, and feel free to feedback in the discussion forums there.
Enjoy!

 

Setting up priority reloads in Qlik Sense

qlik sense priority reload image

Image: Wikimedia Commons

In a large Qlik Sense installations you are likely faced with a high number of reloads every hour.

But some apps are more important than others – those apps should reload immediately when triggered.
The alternative is that those important apps end up being queued after ongoing long running tasks – not good.

In the current Sense version (3.1.5) it is not possible to set priorities for app reload tasks. But we can achieve the same thing using security rules – let’s take a look at how.

Continue reading

Qlik Luminaries 2017 announced – yay!

And guess what? I am one of them! 🙂

I value this quite high. Being part of the 2016 Luminary program too, I know how much value it brought both in terms of great networking opportunities with fellow Luminaries (and others), as well as more and closer contacts with Qlik themselves.

The one thing that makes Sense stand out (IMHO) is the fact that it is a very solid platform, on top of which you can build all kinds of interesting apps.
Qlik’s own user-facing standard Sense client is great, but there are for sure times when you want something else. The APIs very nicely enable that kind of development.

On the other hand, with so much great work happening in the open source field (around back-end technologies, visualisations etc), it is also extremely promising to see Qlik open sourcing some of their tech.

2017 has indeed started in an interesting way for Sense, can’t wait what’s in store for the rest of the year.

Qlik Sense app duplicator 1.2: User interface included


For a long time I wanted to learn more about client side development.

Javascipt, CSS, reactive frameworks etc – all those concepts that make today’s web sites tick. I really think backend developers can benefit from having at least a basic understanding of how the front-end works – this will make the finished app better as a whole.

Couple this with a desire to learn Vue.js specifically, and we get a user interface for the Qlik Sense app duplicator service.

Continue reading