SenseOps 2023-Q4 summary

Wow, that went fast.
Only a week until the holidays kick in and 2023 comes to an end.

The last few months have been good though, when looking at things from a SenseOps tools perspective - quite a lot of new features and fixed bugs are now on the "done" list.

Let's take a look at what's new in the various tools in the extended Butler family.

By the way - all those tools for Qlik Sense are still open source of course. ✌️

Windows service monitoring

Are all Sense services running, on all nodes?
Are you sure?

Qlik Sense is usually very stable, but at times services will fail for one reason or another. Or you want to ensure that some other important service is up and running - and get immediate notifications when it goes down.

Butler now includes a module that can monitor multiple services on multiple Windows servers. Alerts are done wither within Butler itself (email, Slack, Teams, webhooks, InfluxDB, New Relic, MQTT) or via Grafana's excellent alerting features.

Grafana also gives good visual feedback on service status:

Improved InfluxDB support

The time-series database InfluxDB was used as a data destination already in Butler's early days. Little has however been done over the years with respect to InfluxDB - until now.

There are several new features:

Reload task results

A reload failed - why?

Get all relevant info presented in Grafana, included the reload log up until the failure.

Butler has long supported sending emails, Teams and Slack messages when reload tasks fail. Earlier in 2023 New Relic was added as an alert destination too.

In version 9.3.0 InfluxDB was added as a destination for reload failure alerts - and also successful reloads. That enables great looking dashboards in Grafana:

Windows service states in InfluxDb

I want a Teams notification AND a PagerDuty ticket created when a critical Windows service stops.

Butler provides a built-in set of alert destinations, but if service alerts are sent to InfluxDB, Grafana can read them from there.

Grafana in turn integrates with tools like PagerDuty, OpsGenie, VictorOps and others.

Azure Event Grid MQTT support

MQTT is a lightweight publish-subscribe protocol commonly used in IoT applications.

Butler can use MQTT as a destination for Windows service and failed reload alerts, as well as source for starting reload tasks (i.e. start a task when there's an incoming MQTT message).

In Butler 9.3.0 support was added for using Azure Event Grid as an MQTT broker.

This can be a great alternative when you cannot (or don't want to) host your own broker, with excellent security via client certificates and pre-approved IP addresses.

Azure Event Grid has a generous free tier of 100 000 messages per month.

If used to start Qlik Sense reload tasks, that's a lot of started tasks during a month...

One use case can be to have external systems start reload tasks in Sense, without having network access to the Sense environment itself.
The MQTT messages act as an isolation layer in this case, preventing access to Sense while still allowing reload tasks to be started.

User events with browser & OS info

What browsers and OSs are users connecting to Sense with?
What apps have been used during past 60 minutes?

User events occur when users log in/out to Sense or open/close apps.

When apps are opened/closed we now also get information about what browser and OS the user uses. This information is stored in both InfluxDB and New Relic.

This enables Grafana charts like these:

We also get app ID and app name as part of the app-related user events, which in turn let us create real-time views of usage per app during the selected time interval:

New documentation site

With Ctrl-Q becoming more and more capable, it eventually deserved its own documentation site. Which is now live at https://ctrl-q.ptarmiganlabs.com.

There you find complete docs for all Ctrl-Q commands, as well as examples.

Better bulk import of tasks

Manually creating tasks in the QMC is no problem for a few tasks, but with dozens or hundreds of tasks it takes time and becomes prone to errors.

Ctrl-Q version 3.14.0 brings greatly improved bulk import for Qlik Sense tasks:

• More robust import logic
• Detecting and dealing with circular task chains
• Full support for external program tasks

There is a tutorial explaining the task definitions file here, check it out to get an understanding of how tasks can be defined in a simple Excel file, then imported into Sense.