The step from 4.x to 5.0 indicates this is a major release.
Very much so in fact. The main focus of the new version is on reload alerts. Rich email alerts were introduced in Butler 4.2, but that was really just the testing ground for what’s now available in 5.0.
Here’s the short version of what’s new:
Alerts for failed or aborted reloads can be sent to Slack, MS Teams, outgoing webhooks, MQTT or as email.
Alerts sent as email, Slack and MS Teams can all make use of all the rich formatting and layout options available on those platforms.
Alert emails can be sent to the owner of the Sense app that failed reloading.
A new API for doing partial (and full) app reloads was added. This means it’s now possible and even easy to do partial app reloads in Qlik Sense. One of the epic QlikView features finally become usable in Qlik Sense!
Curious what the alerts look like in reality? Check out this video for a hands-on demo:
Usually, yes. Still, things do happen and it certainly would be nice to get an push alert when a server hasn’t checked in according to schedule.
The most common monitoring – for Windows/Linux/…. servers, databases, Qlik Sense etc is based on the tool keeping an eye on some measurement and then alert when the measurement goes beyond some threshold. This is fine, and this is a very important monitoring use case. But in cases where a server just hangs the last measurement received might be fine, and no alerts are sent.
Black box monitoring kind of reverses the roles:
The monitored system has to prove that it’s doing fine. Failing to do so within some predefined schedule will trigger an alarm, with an optional alert being sent.
The previous article showed how this concept can be used to ensure that some Qlik Sense app has reloaded as intended, before a specific time each day. A concrete, common use case would be that yesterday’s data should be processed and loaded into Sense before 7 am next day. Alert if not)
Now, let’s use the same tool and concept to monitor also the Windows servers that Qlik Sense Enterprise on Windows (QSEoW) runs on.
When implementing Qlik Sense solutions for enterprise clients they usually have various requirements regarding alerts for failed reloads, support SLAs etc. Those are all interesting challenges to implement, but the most common request is probably:
“The system should alert me if data doesn’t arrive on time”
It’s a reasonable request. If some source system is delayed and doesn’t delivery data to Sense on time, they as a system or application owner should be notified.
Don’t bother me unless it’s broken
The second thing almost all clients say is:
“Oh, I only want alerts when data is delayed. No messages when data is on time.”
Again, this is very reasonable.
Let’s say the extract app in Sense sends a notification email once data has been loaded from the source system. Great – we now know that data has arrived, and when it did. Except that we will drown in such notifications from all our dozens (or hundreds..) of extract apps.
It’s kind of hard to implement this in a good way though, at least I have never found a really good, generic solution for this request. Sure, you can have a Sense app who’s only job is to execute every 15 minutes and check whether data has arrived, and alert if data is delayed. That app will however put load on your Sense environment and use one of the reload slots. Even if the app reloads quickly it’s still pretty bad system design, IMHO.
I have a suggestion for a better option:
Set up a monitoring tool that views the Sense app (or rather its load script) as a black box with unknown function. The only thing the monitoring tool cares about is whether that black box has checked in within some configured interval.
I stumbled upon this while looking at ways to monitor servers in general, but quickly realised it could be used also with Sense. By the way: while I have only used the described concept on Qlik Sense, it should work equally well with QlikView.
Don’t care for reading? This video outlines the concept, otherwise keep reading below.
During the past weeks I have developed what has turned into a fairly complex Qlik Sense solution, where reloading the main application takes on the order of 10 minutes.
I know – Qlik apps should be kept small, with intermediate data sets stored as QVDs. But sometimes that’s just not practical.
Qlik apps works the same as most apps – if you made a typo somewhere the reload (or compilation for regular computer applications) will fail. Sometimes after a few seconds, more often right towards the end, just before those 10 minutes have passed… Fix the typo and start another 10-minute reload. Sigh.
The other day I realised that I had been using an old trick without even thinking about it, to cut down the reload times from 10 minutes to a few seconds. Talking to some colleagues, it turned out they didn’t use this trick – so here it is: The best time saver ever for Qlik developers.
The idea is very simple: You want to run the entire load script to make sure it works, but without processing 100s of megabytes of data or more.
In Qlik scripting, this is achieved by using the First prefix, added before a Load statement. Two things needed:
Add a variable at the beginning of the script, and make it easy to change it to a few different values (for loading 1000 rows of source datasets, or something larger).
Add the variable before all load statements where large datasets are loaded into the app.
It can look something like this:
Sense’s load script editor does not quite like using variables in this way, and complains about the syntax being incorrect. Don’t worry about that, the script loads fine.
To do a quick load (to ensure the script is solid from a syntax perspective), just enable one of the commented out lines at the top. This will limit the numer of lines loaded in the following load(s). The load time will be nice and short.
Once the script works, comment out that first variable again, setting it to an empty string. All data will be loaded from sources.
Easy trick – I know. But it has literally saved me days during past months!
When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.
A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.
This leads us to the conclusion that the services must automatically be
a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.
Enter process monitors.
At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).
Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.
Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.
For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?
I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….
The more I looked at it, the more I felt “wow – SenseOps really rocks!”
With so much cool data available online, tools like QlikView and Sense becomes real Swiss Army knives of data integration.
Pull in some data from company internal databases, some data from previously created QVDs, and more and more commonly also from various online sources, both public and private. Depending on how you call those online APIs you might get away with just sending in query parameters as they are, but in other cases – and this is especially true if you need to send more complex text strings to the API – you need to URL encode the query parameters.
Over the years I have run into this numerous times, but the other weekend I realised it’s actually very easy. Just create a mapping table using an online source for the utf8-to-URL encoded mapping, then use MapSubstring to convert each character in the URL parameter to its hex counterpart.
This obviously means that the apps name – “slack_proxy” – is not really that relevant or correct any longer… And with more end points considered, the name will become even less correct…. Oh well – I’ll keep the name for now – maybe it will change sometime up ahead.
One of the great things about both QlikView (QV) and Qlik Sense is their integrations with other systems. Given the native connectivity to any ODBC source, web pages in general, Salesforce, REST APIs, BigQuery etc – combined with the dozens of connectors provided by tools like QVSource – I have yet to find a single system or data source we could not pull data from.
The great thing about this is that it makes it easy to quickly get some real data and then start building your application.
That said, sometimes you need a limited data set to start with, or just some conceptual data to try out an idea on. Enter online test data generators, which can be used to generate test data for
I finally got around to start using GitHub Gists a bit more systematically for storing useful bits of code. Going forward I will post useful, reusable pieces of Qlik Sense and QlikView code there. My own experience tells me there is tons of time to be saved by reusing existing code rather than writing it from scratch each time..
IMHO GitHub could improve the gists concept by adding tags to them, that way it would be way easier to find relevant gists. Or I’ll move to some other tool if/once I find it… Any suggestions on good ones? Leave a note in the comments!