Black box monitoring of Butler tools - monitoring the monitor

Black box monitoring of Butler tools - monitoring the monitor
Photo by Sigmund / Unsplash

This is part 3 in a series about how can be used to solve various kinds of monitoring scenarios for Qlik Sense Enterprise on Windows (QSEoW) environments.

Previous articles:

Part 1: Oh data where art thou?
Part 2: Black box monitoring of Windows servers


Most of the Butler family of DevOps tools for Qlik Sense can now be monitored using or similar tool.

If one of the supported Butler tools go down, an alarm is raised and one or more alerts are sent via email, Slack, Teams, PagerDuty or other supported channels.

This means that both infrastructure (servers etc), applications/services and data sources can be monitored using the same tool and on the same dashboards.
Health dashboard at Ptarmigan Labs, covering both servers, applications and data.

Black box monitoring

The previous posts in this series has already explored black box testing, but it’s worth repeating what it means. The idea comes from the software engineering world:

BLACK BOX TESTING, also known as Behavioural Testing, is a software testing method in which the internal structure/design/implementation of the item being tested is not known to the

In our case we modify the definition slightly and apply it to a monitoring scenario (rather than software testing).

The idea is the same though: the monitoring tool does not know – or care – what’s inside of the system being monitored. The monitored system is a “black box” which we cannot see inside.

The Butler family

The Butler family has grown to a handful of tools by now. They all focus on different aspects of Qlik Sense Enterprise on Windows, adding features that make life easier for Sense admins and developers.

WhichWhatAvailableCan be monitored
ButlerExport apps to JSON, start tasks etc, misc utilitiesNowYes
Butler SOSReal time monitoring of operational metricsNowYes
Butler CWCache warming (preload apps into memory)NowYes
Butler AuthSign into Sense using for example Google accountSoonYes
Butler App DuplicatorUse templates when creating Sense appsNowNo
Butler Icon UploaderUpload batches of images to SenseNowNot applicable
Butler SpyglassData lineage toolNowNo

The Butler family of tools for Qlik Sense Enterprise on Windows, as of July 2020.

While some of the tools (such as Butler App Duplicator) are nice to have, other tools (Butler SOS, Butler Auth) can be close to mission critical.

All Butler tools are built on the same technology (Node.js) and are generally very stable. Still, it would be nice to monitor them for failures, and get notified when that happens.

Failures could actually happen for many reasons, not only the Butler tool itself failing:

  • Network issues
  • Server crashes
  • Changed firewall rules etc
Thank’s to the passive nature of black box monitoring, all such issues will be caught if they result in the monitored system (i.e. one of the Butler tools) is no longer able to contact the monitoring tool (

Setting up monitoring of the Butler tools

It’s really quite easy to get started, especially if the cloud based/SaaS version of is used. In that case the whole setup and configuration probably takes just a few minutes and their free tier will get you started at zero cost.

The general process is as follows (looking at Butler SOS in this example):

It’s a good idea to set the heart beat frequency in the Butler config file to a (slightly) lower value, compared to the value set in This way you can avoid false alerts being triggered.

The Butler SOS config file could look something like below.

Alerts when something breaks

The whole point of this exercise is to get alerts when something breaks, so let’s take a look at some examples. As integrates with many different tools, you have a lot of options when it comes to the alerting part of the monitoring setup.

Alert email sent from alert in Slack
Overview in
Monitored Butler tools
Status and history for a specific tool
Monitored physical and virtual servers