What’s Traefik and how does it relate to Qlik Sense?
Traefik is one of today’s most hyped reverse proxies. It’s available in both a commercial and a (very capable!) open source version. According to the company behind Traefik it’s been downloaded more than 2 billion times – that’s a lot…
In this blog post we’ll take a look at how Traefik can run natively on Windows server, providing services to a Qlik Sense Enterprise on Windows cluster. We will see how this can make Qlik Sense easier to use for end users as well as easier to run and operate for sysadmins.
Specifically, we’ll use Traefik to set up TLS secured access (a.k.a. https) to Qlik Sense, while at the same time establishing a solid platform to which we can add more features in coming blog posts.
Starting with the basics makes sense though – let’s get to it!
This release both adds some nice new features as well as enhancing existing ones and fixing some bugs. Let’s take a look at the highlights.
Track in detail whatapps are loaded into each Sense server.
Regular apps and session apps are handled separately, making app metrics easier to understand and more relevant
Sample dashboards are now built using the brand new, shiny and all together awesome Grafana 7. Did I mention that Grafana 7 is awesome? Awesome.
Ever wondered how long Butler SOS has been running or how much memory it uses? The new uptime messages have you covered.
You are properly impressed with the uptime messages – good. Why not store them to Influxdb, so you can also visualize Butler SOS’ own memory use? It’s just a couple of changes in the config file away.
Better control over what features are enabled. Don’t need Docker health checks? Now you can turn that (and other) feature off.
Ah, you are a serious Sense user and have separate DEV and PROD environments? Good – now Butler SOS supports multiple instances running on a single server.
Who will monitor the monitor? Butler SOS can now send heartbeats to customisable URLs at desired intervals. Perfect if you want to monitor Butler SOS using for example healthchecks.io. Very, very cool actually.
Bugs, bugs and bugs. The known ones have been fixed. Keep reporting new ones!
Update all dependencies to latest versions, to ensure security concerns are adressed.
Curious what it looks like in practice? Seeing is believing:
When implementing Qlik Sense solutions for enterprise clients they usually have various requirements regarding alerts for failed reloads, support SLAs etc. Those are all interesting challenges to implement, but the most common request is probably:
“The system should alert me if data doesn’t arrive on time”
It’s a reasonable request. If some source system is delayed and doesn’t delivery data to Sense on time, they as a system or application owner should be notified.
Don’t bother me unless it’s broken
The second thing almost all clients say is:
“Oh, I only want alerts when data is delayed. No messages when data is on time.”
Again, this is very reasonable.
Let’s say the extract app in Sense sends a notification email once data has been loaded from the source system. Great – we now know that data has arrived, and when it did. Except that we will drown in such notifications from all our dozens (or hundreds..) of extract apps.
It’s kind of hard to implement this in a good way though, at least I have never found a really good, generic solution for this request. Sure, you can have a Sense app who’s only job is to execute every 15 minutes and check whether data has arrived, and alert if data is delayed. That app will however put load on your Sense environment and use one of the reload slots. Even if the app reloads quickly it’s still pretty bad system design, IMHO.
I have a suggestion for a better option:
Set up a monitoring tool that views the Sense app (or rather its load script) as a black box with unknown function. The only thing the monitoring tool cares about is whether that black box has checked in within some configured interval.
I stumbled upon this while looking at ways to monitor servers in general, but quickly realised it could be used also with Sense. By the way: while I have only used the described concept on Qlik Sense, it should work equally well with QlikView.
Don’t care for reading? This video outlines the concept, otherwise keep reading below.
Butler SOS has matured quite a bit during past couple of years, with latest additions being fine-grained monitoring of what users are connected to what servers. Opens up for various interesting use cases, including notification to users before server restarts etc.
A brand new doc site (butler-sos.ptarmiganlabs.com) also goes live today (built with Hugo and Docsy, vastly better than the previous one-pager at GitHub.
Butler SOS 4.0 is out, adding features that make it easier than ever to monitor large Qlik Sense environments. We’ll return to this topic of course, but let’s first take a few steps back.
There are many variants of that quote: “If you can’t measure it, you can’t improve it”, “Measure what matters”, “Measure what is measurable, and make measurable what is not so.” and others. The last one supposedly originates with Galileo Galilei. Smart guy.
The development of Butler SOS continues in that spirit. Qlik Sense provides an awesome platform on top of which all kinds of data analysis, visualisation and presentation solutions can be built. A key word there is platform. Sense does not offer solutions to all software development challenges, nor should it. Instead, use the tools and best practices that millions of developers around the world have refined over the years.
Qlik Sense does on the other hand offer a very comprehensive set of APIs that give developers access to its internals – and this is part of why it’s such a powerful platform. Butler SOS taps into some of these APIs, exposing their data in the form of real-time dashboards, charts and alerts. Suddenly sysadmins know can get both an overview of how all servers are doing, as well as look at detailed server metrics when so needed. All made possible using the Sense APIs, but in general powered by various open source tools.
We’re basically back to Galileo – let’s make sure the important parts of Qlik Sense are measurable. It is then possible to improve the parts that don’t work well.
The basics are the same, i.e. one-click creation of Qlik Sense apps, using regular Sense apps as templates. Several new features however take the tool to a new level, making it easier to set up, manage and more enterprise grade. Good news thus!
For years I have thought about ways to get data lineage info for all apps in a Qlik Sense Enterprise environment.
It would be super useful to know exactly what apps use a particular data source, as well as vice versa (what data sources are used by a specific app). I know there are commercial tools doing this and much more, but I wanted something easy to use, yet still effective and free.
Same thing for app load scripts: Extracting and storing them to disk in human readable format has more than once save days of work, when something has gone badly wrong in an app. Dumping load scripts to disk was possible in my original Butler tool, but then only one app at a time. So not quite what was needed in an enterprise context.
The latest version fo Butler SOS is out, taking the version number to 3.0. A lot of the code has been fine tuned to better meet the needs of enterprise grade Qlik Sense deployments.
Docker (or some compatible container platform) is now the preferred and recommended way of running Butler SOS. Butler SOS has been developed and tested on Linux and Mac OS, but should in theory run also other Docker enabled platforms.
Version 3 adds a few – but useful – features:
Per-server config option “serverGroup”. Use this to group or categorize servers, for example as being part of a production vs development Qlik Sense cluster. This enables the creation of Grafana dashboards that use Grafana variables to automatically show metrics for all PRODUCTION servers. This greatly simplifies using Butler SOS in large Qlik Sense environments, where servers are frequently added/removed. No need to manually update the Grafana dashboards any more.
Config option “queryPeriod” for controlling how far back querying for Sense log entries should be done. Used together with the logdb.pollingInterval setting, it is now possible to fine tune how often the Qlik Sense log database is queried for errors and warnings.