Qlik Sense Archives - Page 2 of 6

2020-06-24

Black box monitoring of Windows servers

This is part 2 in a series about how healthchecks.io can be used together with Qlik Sense Enterprise on Windows (QSEoW).

Previous articles:
Oh data where art thou?

Windows Server just works… right?

Usually, yes.
Still, things do happen and it certainly would be nice to get an push alert when a server hasn’t checked in according to schedule.

The most common monitoring – for Windows/Linux/…. servers, databases, Qlik Sense etc is based on the tool keeping an eye on some measurement and then alert when the measurement goes beyond some threshold.
This is fine, and this is a very important monitoring use case. But in cases where a server just hangs the last measurement received might be fine, and no alerts are sent.

Black box monitoring kind of reverses the roles:

The monitored system has to prove that it’s doing fine. Failing to do so within some predefined schedule will trigger an alarm, with an optional alert being sent.

The previous article showed how this concept can be used to ensure that some Qlik Sense app has reloaded as intended, before a specific time each day. A concrete, common use case would be that yesterday’s data should be processed and loaded into Sense before 7 am next day. Alert if not)

Now, let’s use the same tool and concept to monitor also the Windows servers that Qlik Sense Enterprise on Windows (QSEoW) runs on.

2020-06-23

Oh data where art thou?

When implementing Qlik Sense solutions for enterprise clients they usually have various requirements regarding alerts for failed reloads, support SLAs etc.
Those are all interesting challenges to implement, but the most common request is probably:

“The system should alert me if data doesn’t arrive on time”

It’s a reasonable request. If some source system is delayed and doesn’t delivery data to Sense on time, they as a system or application owner should be notified.

Don’t bother me unless it’s broken

The second thing almost all clients say is:

“Oh, I only want alerts when data is delayed. No messages when data is on time.”

Again, this is very reasonable.

Let’s say the extract app in Sense sends a notification email once data has been loaded from the source system. Great – we now know that data has arrived, and when it did.
Except that we will drown in such notifications from all our dozens (or hundreds..) of extract apps.

It’s kind of hard to implement this in a good way though, at least I have never found a really good, generic solution for this request.
Sure, you can have a Sense app who’s only job is to execute every 15 minutes and check whether data has arrived, and alert if data is delayed. That app will however put load on your Sense environment and use one of the reload slots. Even if the app reloads quickly it’s still pretty bad system design, IMHO.

I have a suggestion for a better option:

Set up a monitoring tool that views the Sense app (or rather its load script) as a black box with unknown function. The only thing the monitoring tool cares about is whether that black box has checked in within some configured interval.

I stumbled upon this while looking at ways to monitor servers in general, but quickly realised it could be used also with Sense.
By the way: while I have only used the described concept on Qlik Sense, it should work equally well with QlikView.

Don’t care for reading? This video outlines the concept, otherwise keep reading below.

2020-04-16

60 second trick saves hours debugging Qlik Sense apps

During the past weeks I have developed what has turned into a fairly complex Qlik Sense solution, where reloading the main application takes on the order of 10 minutes.

I know – Qlik apps should be kept small, with intermediate data sets stored as QVDs. But sometimes that’s just not practical.

Qlik apps works the same as most apps – if you made a typo somewhere the reload (or compilation for regular computer applications) will fail. Sometimes after a few seconds, more often right towards the end, just before those 10 minutes have passed… Fix the typo and start another 10-minute reload. Sigh.

The other day I realised that I had been using an old trick without even thinking about it, to cut down the reload times from 10 minutes to a few seconds. Talking to some colleagues, it turned out they didn’t use this trick – so here it is: The best time saver ever for Qlik developers.

The idea is very simple: You want to run the entire load script to make sure it works, but without processing 100s of megabytes of data or more.

In Qlik scripting, this is achieved by using the First prefix, added before a Load statement. Two things needed:

Add a variable at the beginning of the script, and make it easy to change it to a few different values (for loading 1000 rows of source datasets, or something larger).
Add the variable before all load statements where large datasets are loaded into the app.

It can look something like this:

Sense’s load script editor does not quite like using variables in this way, and complains about the syntax being incorrect. Don’t worry about that, the script loads fine.

To do a quick load (to ensure the script is solid from a syntax perspective), just enable one of the commented out lines at the top. This will limit the numer of lines loaded in the following load(s). The load time will be nice and short.

Once the script works, comment out that first variable again, setting it to an empty string. All data will be loaded from sources.

Easy trick – I know. But it has literally saved me days during past months!

2020-04-06

Hack the Crisis hackathon

Wow, that was a busy weekend.

Hack the Crisis was a government sponsored hackathon with 5000+ people taking part – pretty impressive.

Friday night our hackathon team kicked off our project. The goal was to develop the core concepts of a service that makes it easier to help those who need help in these challenging times.

The verdict is still to come regarding official hackathon results, but the general feeling is pretty good. There is a distinct need out there, we demoed the core concepts of a solution, and there are clear paths onward from here.

And yes – a 48 hour online hackathon is a pretty effective way of achieving social distancing from your family living in the same house…

Take care and stay safe out there.

2020-01-10

Animated GIF horror for Qlik Sense

Qlik Sense is very much my go-to tool for many data related questions.
It’s very powerful to be able to throw together a proof of concept or test a hypothesis in half an hour.
Seeing is however believing, so those prototypes does need some polishing before they can be seen as production grade.

I have always pushed my peers in the Qlikosphere to become better at using icons for their Qlik Sense apps in general, and also for the sheets within those apps.
Having an icon for a sheet within an app makes it way easier for the user to find a particular sheet – humans are visual by nature.

2019-04-16

If you don’t measure it, you can’t improve it: DevOps concepts meets Qlik Sense

Real-time monitoring of user sessions using Butler SOS

Butler SOS 4.0 is out, adding features that make it easier than ever to monitor large Qlik Sense environments. We’ll return to this topic of course, but let’s first take a few steps back.

There are many variants of that quote: “If you can’t measure it, you can’t improve it”, “Measure what matters”, “Measure what is measurable, and make measurable what is not so.” and others. The last one supposedly originates with Galileo Galilei. Smart guy.

The development of Butler SOS continues in that spirit. Qlik Sense provides an awesome platform on top of which all kinds of data analysis, visualisation and presentation solutions can be built. A key word there is platform. Sense does not offer solutions to all software development challenges, nor should it. Instead, use the tools and best practices that millions of developers around the world have refined over the years.

Qlik Sense does on the other hand offer a very comprehensive set of APIs that give developers access to its internals – and this is part of why it’s such a powerful platform. Butler SOS taps into some of these APIs, exposing their data in the form of real-time dashboards, charts and alerts. Suddenly sysadmins know can get both an overview of how all servers are doing, as well as look at detailed server metrics when so needed.
All made possible using the Sense APIs, but in general powered by various open source tools.

We’re basically back to Galileo – let’s make sure the important parts of Qlik Sense are measurable. It is then possible to improve the parts that don’t work well.

2019-04-14

One click creation of Qlik Sense apps: Butler App Duplicator 3.0 is out

Heading for the mountains here, but a quick update first.

Everyone’s favourite app wizard for Qlik Sense (ok… my favourite wiz at least..) has had a major face lift – yay!

The basics are the same, i.e. one-click creation of Qlik Sense apps, using regular Sense apps as templates. Several new features however take the tool to a new level, making it easier to set up, manage and more enterprise grade. Good news thus!

2019-03-10

Butler Spyglass: Data lineage and metadata tool for Qlik Sense

For years I have thought about ways to get data lineage info for all apps in a Qlik Sense Enterprise environment.

It would be super useful to know exactly what apps use a particular data source, as well as vice versa (what data sources are used by a specific app). I know there are commercial tools doing this and much more, but I wanted something easy to use, yet still effective and free.

Same thing for app load scripts: Extracting and storing them to disk in human readable format has more than once save days of work, when something has gone badly wrong in an app.
Dumping load scripts to disk was possible in my original Butler tool, but then only one app at a time. So not quite what was needed in an enterprise context.

Enter Butler Spyglass.

2019-01-21

Monitoring for Qlik Sense: Butler SOS v3 is out

The latest version fo Butler SOS is out, taking the version number to 3.0.
A lot of the code has been fine tuned to better meet the needs of enterprise grade Qlik Sense deployments.

Docker (or some compatible container platform) is now the preferred and recommended way of running Butler SOS. Butler SOS has been developed and tested on Linux and Mac OS, but should in theory run also other Docker enabled platforms.

New features

Version 3 adds a few – but useful – features:

Per-server config option “serverGroup”. Use this to group or categorize servers, for example as being part of a production vs development Qlik Sense cluster. This enables the creation of Grafana dashboards that use Grafana variables to automatically show metrics for all PRODUCTION servers. This greatly simplifies using Butler SOS in large Qlik Sense environments, where servers are frequently added/removed. No need to manually update the Grafana dashboards any more.
Config option “queryPeriod” for controlling how far back querying for Sense log entries should be done. Used together with the logdb.pollingInterval setting, it is now possible to fine tune how often the Qlik Sense log database is queried for errors and warnings.

2018-10-16

The beauty of Docker – how to run all Butler tools with a single command

Docker is great.

Docker is one of those tools that have the potential to fundamentally transform how you develop and run software – once you have tried Docker it is hard to imagine going back to something else.

In previous posts we have seen how Butler, Butler SOS and Butler CW can be run as Docker containers.
But we can do even better – why not control all the Butler tools from a single docker-compose file? Maybe even specifying the dependencies on influxdb and mqtt in there too?

Setting this up is incredibly easy – a single docker-compose file tells Docker what containers to use, and some config files tells the Butler tools where to find things.

Let’s get started!

Continue reading “The beauty of Docker – how to run all Butler tools with a single command”