Black box monitoring of Butler tools – monitoring the monitor

This is part 3 in a series about how healthchecks.io can be used to solve various kinds of monitoring scenarios for Qlik Sense Enterprise on Windows (QSEoW) environments.

Previous articles:

Part 1: Oh data where art thou?
Part 2: Black box monitoring of Windows servers

Tl;dr

Most of the Butler family of DevOps tools for Qlik Sense can now be monitored using healthchecks.io or similar tool.

If one of the supported Butler tools go down, an alarm is raised and one or more alerts are sent via email, Slack, Teams, PagerDuty or other supported channels.

This means that both infrastructure (servers etc), applications/services and data sources can be monitored using the same tool and on the same dashboards.

Health dashboard at Ptarmigan Labs, covering both servers, applications and data.
Continue reading “Black box monitoring of Butler tools – monitoring the monitor”

Black box monitoring of Windows servers

This is part 2 in a series about how healthchecks.io can be used together with Qlik Sense Enterprise on Windows (QSEoW).

Previous articles:
Oh data where art thou?

Windows Server just works… right?

Usually, yes.
Still, things do happen and it certainly would be nice to get an push alert when a server hasn’t checked in according to schedule.

The most common monitoring – for Windows/Linux/…. servers, databases, Qlik Sense etc is based on the tool keeping an eye on some measurement and then alert when the measurement goes beyond some threshold.
This is fine, and this is a very important monitoring use case. But in cases where a server just hangs the last measurement received might be fine, and no alerts are sent.

Black box monitoring kind of reverses the roles:

The monitored system has to prove that it’s doing fine. Failing to do so within some predefined schedule will trigger an alarm, with an optional alert being sent.

The previous article showed how this concept can be used to ensure that some Qlik Sense app has reloaded as intended, before a specific time each day. A concrete, common use case would be that yesterday’s data should be processed and loaded into Sense before 7 am next day. Alert if not)

Now, let’s use the same tool and concept to monitor also the Windows servers that Qlik Sense Enterprise on Windows (QSEoW) runs on.

Continue reading “Black box monitoring of Windows servers”

Oh data where art thou?

When implementing Qlik Sense solutions for enterprise clients they usually have various requirements regarding alerts for failed reloads, support SLAs etc.
Those are all interesting challenges to implement, but the most common request is probably:

“The system should alert me if data doesn’t arrive on time”

It’s a reasonable request. If some source system is delayed and doesn’t delivery data to Sense on time, they as a system or application owner should be notified.

Don’t bother me unless it’s broken

The second thing almost all clients say is:

“Oh, I only want alerts when data is delayed. No messages when data is on time.”

Again, this is very reasonable.

Let’s say the extract app in Sense sends a notification email once data has been loaded from the source system. Great – we now know that data has arrived, and when it did.
Except that we will drown in such notifications from all our dozens (or hundreds..) of extract apps.

It’s kind of hard to implement this in a good way though, at least I have never found a really good, generic solution for this request.
Sure, you can have a Sense app who’s only job is to execute every 15 minutes and check whether data has arrived, and alert if data is delayed. That app will however put load on your Sense environment and use one of the reload slots. Even if the app reloads quickly it’s still pretty bad system design, IMHO.

I have a suggestion for a better option:

Set up a monitoring tool that views the Sense app (or rather its load script) as a black box with unknown function. The only thing the monitoring tool cares about is whether that black box has checked in within some configured interval.

I stumbled upon this while looking at ways to monitor servers in general, but quickly realised it could be used also with Sense.
By the way: while I have only used the described concept on Qlik Sense, it should work equally well with QlikView.

Don’t care for reading? This video outlines the concept, otherwise keep reading below.

Continue reading “Oh data where art thou?”