Superpowers to Qlik Sense Enterprise, part 2: Free https certificates from Let’s Encrypt

Photo by Silas Köhler on Unsplash

Security is important – period.

Usability of web based systems is off course important, but so is security. In the previous (first) article in this series we looked at how the Traefik reverse proxy can be used set up end user friendly URLs for Qlik Sense Enterprise (QSE) environments, while at the same time adding a layer of security by hiding the internal structure of the QSE cluster.

Today we’ll look at how you can get free SSL/TLS certificates from Let’s Encrypt and how these are used to provide https for QSE.

Article 1: Superpowers to Qlik Sense Enterprise: The Traefik reverse proxy, part 1

All code, scripts etc used in this article is available in the Part 2 folder in this GitHub repository.

Let’s go!

Continue reading “Superpowers to Qlik Sense Enterprise, part 2: Free https certificates from Let’s Encrypt”

Superpowers to Qlik Sense Enterprise: The Traefik reverse proxy, part 1

Edit: A GitHub repository has been created with the files mentioned in this article:
https://github.com/mountaindude/qlik-sense-and-traefik

What’s Traefik and how does it relate to Qlik Sense?

Traefik is one of today’s most hyped reverse proxies. It’s available in both a commercial and a (very capable!) open source version. According to the company behind Traefik it’s been downloaded more than 2 billion times – that’s a lot…

In this blog post we’ll take a look at how Traefik can run natively on Windows server, providing services to a Qlik Sense Enterprise on Windows cluster.
We will see how this can make Qlik Sense easier to use for end users as well as easier to run and operate for sysadmins.

Specifically, we’ll use Traefik to set up TLS secured access (a.k.a. https) to Qlik Sense, while at the same time establishing a solid platform to which we can add more features in coming blog posts.

Starting with the basics makes sense though – let’s get to it!

Continue reading “Superpowers to Qlik Sense Enterprise: The Traefik reverse proxy, part 1”

Black box monitoring of Butler tools – monitoring the monitor

This is part 3 in a series about how healthchecks.io can be used to solve various kinds of monitoring scenarios for Qlik Sense Enterprise on Windows (QSEoW) environments.

Previous articles:

Part 1: Oh data where art thou?
Part 2: Black box monitoring of Windows servers

Tl;dr

Most of the Butler family of DevOps tools for Qlik Sense can now be monitored using healthchecks.io or similar tool.

If one of the supported Butler tools go down, an alarm is raised and one or more alerts are sent via email, Slack, Teams, PagerDuty or other supported channels.

This means that both infrastructure (servers etc), applications/services and data sources can be monitored using the same tool and on the same dashboards.

Health dashboard at Ptarmigan Labs, covering both servers, applications and data.
Continue reading “Black box monitoring of Butler tools – monitoring the monitor”

Butler DevOps tools for Qlik Sense running on Raspberry Pi based Kubernetes cluster

Lots of buzzwords in that title, but yes – things have now come together and the main members of the Butler family of tools for Qlik Sense now run as Docker containers on Arm64 and even Arm7.

In practice this means it is possible to run Butler SOS, Butler Cache Warming etc as Docker containers on for example a Raspberry Pi 4. If you are worried about that poor little Raspi failing you can add a bunch of them into a Kubernetes cluster, and then run the Butler tools there.

This actually works pretty well, here is a Raspberry Pi based Kubernetes (a.k.a “k8s”) cluster built using the excellent k3s distribution from Rancher, running the Butler tools:

Butler tools running Raspberry Pi based k3s cluster.
Butler tools running as Docker containers in Raspberry Pi based Kubernetes cluster.

And here is the Raspi cluster itself, with 4 of the 7 nodes installed:

Kubernetes cluster built on Raspberry Pis
Kubernetes cluster built on Raspberry Pis

Ok, ok – I hear you – is it really a good idea to run close to mission critical software on Raspberry Pi’s?
No of course not, but the cloud providers are starting to offer Arm based virtual machines (Amazon does this already with their EC2 A1 instances), with roughly twice the performance per dollar compared to Intel based VMs.

So running Butler on Arm VMs might be a way to save some money, if the cloud provider you use happens to offer Arm VMs.

Or just surrender to your inner geek and build your own Raspi k8s cluster… Lots of fun and a great way to learn about k8s!

Docker images available on Docker Hub, as always.

Butler SOS 5.4 is out: good looks and app details

Butler SOS provides real-time insights into Qlik Sense Enterprise on Windows environments.

Time for another update of Butler SOS, this time to version 5.4.

Github release : https://github.com/ptarmiganlabs/butler-sos/releases
Docker image   : https://hub.docker.com/r/ptarmiganlabs/butler-sos
Documentation  : butler-sos.ptarmiganlabs.com

This release both adds some nice new features as well as enhancing existing ones and fixing some bugs. Let’s take a look at the highlights.

  • Track in detail what apps are loaded into each Sense server.
  • Regular apps and session apps are handled separately, making app metrics easier to understand and more relevant
  • Sample dashboards are now built using the brand new, shiny and all together awesome Grafana 7. Did I mention that Grafana 7 is awesome? Awesome.
  • Ever wondered how long Butler SOS has been running or how much memory it uses? The new uptime messages have you covered.
  • You are properly impressed with the uptime messages – good. Why not store them to Influxdb, so you can also visualize Butler SOS’ own memory use? It’s just a couple of changes in the config file away.
  • Better control over what features are enabled. Don’t need Docker health checks? Now you can turn that (and other) feature off.
  • Ah, you are a serious Sense user and have separate DEV and PROD environments? Good – now Butler SOS supports multiple instances running on a single server.
  • Who will monitor the monitor? Butler SOS can now send heartbeats to customisable URLs at desired intervals. Perfect if you want to monitor Butler SOS using for example healthchecks.io. Very, very cool actually.
  • Bugs, bugs and bugs. The known ones have been fixed. Keep reporting new ones!
  • Update all dependencies to latest versions, to ensure security concerns are adressed.

Curious what it looks like in practice?
Seeing is believing:

Black box monitoring of Windows servers

This is part 2 in a series about how healthchecks.io can be used together with Qlik Sense Enterprise on Windows (QSEoW).

Previous articles:
Oh data where art thou?

Windows Server just works… right?

Usually, yes.
Still, things do happen and it certainly would be nice to get an push alert when a server hasn’t checked in according to schedule.

The most common monitoring – for Windows/Linux/…. servers, databases, Qlik Sense etc is based on the tool keeping an eye on some measurement and then alert when the measurement goes beyond some threshold.
This is fine, and this is a very important monitoring use case. But in cases where a server just hangs the last measurement received might be fine, and no alerts are sent.

Black box monitoring kind of reverses the roles:

The monitored system has to prove that it’s doing fine. Failing to do so within some predefined schedule will trigger an alarm, with an optional alert being sent.

The previous article showed how this concept can be used to ensure that some Qlik Sense app has reloaded as intended, before a specific time each day. A concrete, common use case would be that yesterday’s data should be processed and loaded into Sense before 7 am next day. Alert if not)

Now, let’s use the same tool and concept to monitor also the Windows servers that Qlik Sense Enterprise on Windows (QSEoW) runs on.

Continue reading “Black box monitoring of Windows servers”

Oh data where art thou?

When implementing Qlik Sense solutions for enterprise clients they usually have various requirements regarding alerts for failed reloads, support SLAs etc.
Those are all interesting challenges to implement, but the most common request is probably:

“The system should alert me if data doesn’t arrive on time”

It’s a reasonable request. If some source system is delayed and doesn’t delivery data to Sense on time, they as a system or application owner should be notified.

Don’t bother me unless it’s broken

The second thing almost all clients say is:

“Oh, I only want alerts when data is delayed. No messages when data is on time.”

Again, this is very reasonable.

Let’s say the extract app in Sense sends a notification email once data has been loaded from the source system. Great – we now know that data has arrived, and when it did.
Except that we will drown in such notifications from all our dozens (or hundreds..) of extract apps.

It’s kind of hard to implement this in a good way though, at least I have never found a really good, generic solution for this request.
Sure, you can have a Sense app who’s only job is to execute every 15 minutes and check whether data has arrived, and alert if data is delayed. That app will however put load on your Sense environment and use one of the reload slots. Even if the app reloads quickly it’s still pretty bad system design, IMHO.

I have a suggestion for a better option:

Set up a monitoring tool that views the Sense app (or rather its load script) as a black box with unknown function. The only thing the monitoring tool cares about is whether that black box has checked in within some configured interval.

I stumbled upon this while looking at ways to monitor servers in general, but quickly realised it could be used also with Sense.
By the way: while I have only used the described concept on Qlik Sense, it should work equally well with QlikView.

Don’t care for reading? This video outlines the concept, otherwise keep reading below.

Continue reading “Oh data where art thou?”

60 second trick saves hours debugging Qlik Sense apps

During the past weeks I have developed what has turned into a fairly complex Qlik Sense solution, where reloading the main application takes on the order of 10 minutes.

I know – Qlik apps should be kept small, with intermediate data sets stored as QVDs. But sometimes that’s just not practical.

Qlik apps works the same as most apps – if you made a typo somewhere the reload (or compilation for regular computer applications) will fail. Sometimes after a few seconds, more often right towards the end, just before those 10 minutes have passed… Fix the typo and start another 10-minute reload. Sigh.


The other day I realised that I had been using an old trick without even thinking about it, to cut down the reload times from 10 minutes to a few seconds. Talking to some colleagues, it turned out they didn’t use this trick – so here it is: The best time saver ever for Qlik developers.

The idea is very simple: You want to run the entire load script to make sure it works, but without processing 100s of megabytes of data or more.

In Qlik scripting, this is achieved by using the First prefix, added before a Load statement. Two things needed:

  1. Add a variable at the beginning of the script, and make it easy to change it to a few different values (for loading 1000 rows of source datasets, or something larger).
  2. Add the variable before all load statements where large datasets are loaded into the app.

It can look something like this:

Sense’s load script editor does not quite like using variables in this way, and complains about the syntax being incorrect. Don’t worry about that, the script loads fine.

To do a quick load (to ensure the script is solid from a syntax perspective), just enable one of the commented out lines at the top. This will limit the numer of lines loaded in the following load(s). The load time will be nice and short.

Once the script works, comment out that first variable again, setting it to an empty string. All data will be loaded from sources.

Easy trick – I know. But it has literally saved me days during past months!

Animated GIF horror for Qlik Sense

Parrot, anyone?

Qlik Sense is very much my go-to tool for many data related questions.
It’s very powerful to be able to throw together a proof of concept or test a hypothesis in half an hour.
Seeing is however believing, so those prototypes does need some polishing before they can be seen as production grade.

I have always pushed my peers in the Qlikosphere to become better at using icons for their Qlik Sense apps in general, and also for the sheets within those apps.
Having an icon for a sheet within an app makes it way easier for the user to find a particular sheet – humans are visual by nature.

Continue reading “Animated GIF horror for Qlik Sense”

Butler SOS 5.0 is out – new features, new doc site

Butler SOS has matured quite a bit during past couple of years, with latest additions being fine-grained monitoring of what users are connected to what servers.
Opens up for various interesting use cases, including notification to users before server restarts etc.

A brand new doc site (butler-sos.ptarmiganlabs.com) also goes live today (built with Hugo and Docsy, vastly better than the previous one-pager at GitHub.

Starting today that site is the goto-place for everything Butler SOS related.
The site contains both installation and configuration instructions, as well as a (hopefully growing) set of examples on how Butler SOS can be used in various way.

Make sure to check out the blog section of that site too!