Monitoring and auto-starting Node.js services on Windows Server

When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.

A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.

This leads us to the conclusion that the services must automatically be

a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.

Enter process monitors.

At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).

Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.

Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.

Continue reading “Monitoring and auto-starting Node.js services on Windows Server”

Sense + DevOps = SenseOps

For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?

Some initiatives are starting to appear. Qlik themselves are open sourcing microservices such as the Governed Metrics Service , as well as open sourcing key libraries that make it easier to use Sense as a general purpose analytics platform.  I have a couple of contributions in this space too, with the Butler and App duplicator microservices.

I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….




The more I looked at it, the more I felt “wow – SenseOps really rocks!”

Thus, please meet a new site:

Head over for a look, and feel free to feedback in the discussion forums there.


URL encoding in Qlik Sense and QlikView

URLWith so much cool data available online, tools like QlikView and Sense becomes real Swiss Army knives of data integration.

Pull in some data from company internal databases, some data from previously created QVDs, and more and more commonly also from various online sources, both public and private. Depending on how you call those online APIs you might get away with just sending in query parameters as they are, but in other cases – and this is especially true if you need to send more complex text strings to the API – you need to URL encode the query parameters.

Over the years I have run into this numerous times, but the other weekend I realised it’s actually very easy. Just create a mapping table using an online source for the utf8-to-URL encoded mapping, then use MapSubstring to convert each character in the URL parameter to its hex counterpart.

Code for this found over at GitHub, it’s just 10 or so lines of code.

Occam’s razor holds true again – the easy solutions prevail and are usually preferred!


Posting to Slack – and more – from Qlik Sense load scripts (part 3)

Quick update to the previous post on this topic.

There is a new version of slack_proxy available on GitHub. It adds a new endpoint for creating directories on the server where slack_proxy is running.

This obviously means that the apps name – “slack_proxy” – is not really that relevant or correct any longer… And with more end points considered, the name will become even less correct…. Oh well – I’ll keep the name for now – maybe it will change sometime up ahead.

The endpoints currently supported are:

  • /slack for posting to Slack
  • /createDir for creating directories on local disk

Other endpoints on the radar are for sending tweets, sending messages to Pushover, and controlling Blink(1) USB lights.

There is also now reasonably complete documentation on GitHub.

Generating test data for QlikView and Qlik Sense

Real data and test data – both are needed

One of the great things about both QlikView (QV) and Qlik Sense is their integrations with other systems. Given the native connectivity to any ODBC source, web pages in general, Salesforce, REST APIs, BigQuery etc – combined with the dozens of connectors provided by tools like QVSource – I have yet to find a single system or data source we could not pull data from.

The great thing about this is that it makes it easy to quickly get some real data and then start building your application.

That said, sometimes you need a limited data set to start with, or just some conceptual data to try out an idea on. Enter online test data generators, which can be used to generate test data for

Continue reading “Generating test data for QlikView and Qlik Sense”

Qlik Sense and QlikView code snippets available as GitHub Gists

I finally got around to start using GitHub Gists a bit more systematically for storing useful bits of code. Going forward I will post useful, reusable pieces of Qlik Sense and QlikView code there. My own experience tells me there is tons of time to be saved by reusing existing code rather than writing it from scratch each time..

All gists available here.

IMHO GitHub could improve the gists concept by adding tags to them, that way it would be way easier to find relevant gists. Or I’ll move to some other tool if/once I find it… Any suggestions on good ones? Leave a note in the comments!

Using Slack to monitor Qlikview and Qlik Sense

Slack has only been around since August 2013, but I would definitively say it’s one of the better Slack integrationsteam communication services out there. The web client is great, and the OSX and IOS clients are truly awesome. It integrates with tons of other services, including Dropbox, GitHub, IFTTT, Jira, Google Drive, RSS, Nagios, Yo, Twitter and Pingdom are just some of the services it supports (as of today they seem to support ca 75 integrations). There are also generic connectors for incoming and outgoing webhooks, especially the incoming webhook feature will be interesting from a systems monitoring perspective – it will allows us to post messages to Slack by just calling a certain URL.

In this post we will look how we can use this to both monitor the various QlikView services, as well as monitor the transfer of files (e.g. data files used by QV) to a QV server, and a as a generic way of sending notifications from QlikView Management Console (QMC).

Most of the concepts below also apply to Qlik Sense, of course.

Continue reading “Using Slack to monitor Qlikview and Qlik Sense”

Qlik Sense keyboard shortcuts

Qlik Sense is very nice in most ways, the development environment however still falls short when compared to the one found in QlikView. Having a proper Windows client (which QlikView has) just gives more flexibility than using a web based editor and development environment.

Don’t get me wrong – Sense gets the job done – just that for people whose muscle memory is tuned to the QlikView development environment, Sense feels somewhat limiting and slow. That said, after spending a fair bit of time both porting existing QlikView apps to Sense, as well as developing new Sense apps from scratch, the development environment becomes less and less of an issue. I’d say that Qlik still has way to go though. But hey – we are looking at version 1 – I am sure there are plenty of good things to come in future versions.

Anyway – if you spend significant parts of your days in an application, you want to be as effective as possible in your work. QlikView has quite a few powerful keyboard shortcuts, Qlik Sense has somewhat fewer, but still useful. These are listed in this online Qlik Sense help page.

Seems that the help page is not complete though – some shortcuts are not listed there.
Let’s compile a list of them here, please feel free to add additional ones in the comments!

Ctrl-KComment (using //) the lines currently selected in the code editor.


Windows update breaks QlikView server 11.20 SR10

Update: Turns out it is the “Update for Windows Server 2012 (KB3013767)” that when installed on the server in question prevents the QlikView Directory Service to start. Installing the other two updates mentioned below (KB3004365 and KB3029449) does not affect QV operation.


While doing regular maintenance of a QlikView server (running 11.20 SR8) recently (as part of my daytime job), the latest Windows service packs were installed, as usual. This is normally just routine procedure, but not this time.

QlikView Directory Service fails to start
QlikView Directory Service fails to start

After installation was done and the server restarted, it was impossible to log into the QlikView Access Point (i.e. the system’s web UI). The reason was that QlikView’s Directory Service, QDS, failed to start. Even manual attempts at starting it gave the error shown to the right. The error dialog mentions dependencies that fail to start, but this particular service does not have any dependencies listed in its properties dialog. Strange…

Continue reading “Windows update breaks QlikView server 11.20 SR10”

Using CoffeeScript to create QlikView extensions

I am not a Javascript fan. To be honest, I find it rather abusive. There are probably lots of theoretically sound reasons for designing the Javascript (=JS) syntax the way it is – but that does not make it more readable or easy to learn.

I am however a QlikView (=QV) fan. Or rather: QV is a very powerful data visualisation, exploration and discovery tool, but the main drawbacks is its somewhat dated visualisation options. In a world used to fancy angular.js based dynamic web sites and great looking HighCharts graphs, QV’s visualisation options aren’t quite up there.

QV however has a rather interesting “extension” mechanism. You can create object extensions that add new visualisations to QV, using JS to develop the extensions. Document extensions are used to modify the deeper levels of QV applications – very useful and cool (someone created a document extensions using the accelerometers in iPhones to enable new ways to interacts with QV applications – pretty cool!) but not the focus of this post.

So, we want to create QlikView object extensions. JS is the mandated language. Ouch. We can however use CoffeeScript to remove the bad parts of JS, making the code base smaller and more easy to read and maintain. CoffeeScript is very cool, lots of testimonies to it’s greatness out there (DropBox is using CoffeeScript these days, and have shared some experiences).

Note: You need to install node.js before CoffeeScript. I’ve tried this on both Windows 8.1 and OS X – compiling CoffeeScript works without problems on both platforms.

Tuns out this works quite well. Brian Munz of QlikTech has created a couple of very nice templates to make it easier to create extensions, I’ve taken the liberty of converting one of them to CoffeeScript, to show how easy it is to convert JS to CoffeeScript (and make my own future extension development easier).

The CoffeeScript code can also be found in my repo at GitHub.

The Javascript version first:

[code language=”text”]

var template_path = Qva.Remote + "?public=only&name=Extensions/template_simple/";
function extension_Init()
// Use QlikView’s method of loading other files needed by an extension. These files should be added to your extension .zip file (.qar)
if (typeof jQuery == ‘undefined’) {
Qva.LoadScript(template_path + ‘jquery.js’, extension_Done);
else {

function extension_Done(){
//Add extension
Qva.AddExtension(‘template_simple’, function(){
//Load a CSS style sheet
Qva.LoadCSS(template_path + "style.css");
var _this = this;
//add a unique name to the extension in order to prevent conflicts with other extensions.
//basically, take the object ID and add it to a DIV
var divName = _this.Layout.ObjectId.replace("\", "_");
if(_this.Element.children.length == 0) {//if this div doesn’t already exist, create a unique div with the divName
var ui = document.createElement("div");
ui.setAttribute("id", divName);
} else {
//if it does exist, empty the div so we can fill it again
$("#" + divName).empty();

//create a variable to put the html into
var html = "";
//set a variable to the dataset to make things easier
var td = _this.Data;
//loop through the data set and add the values to the html variable
for(var rowIx = 0; rowIx < td.Rows.length; rowIx++) {
//set the current row to a variable
var row = td.Rows[rowIx];
//get the value of the first item in the dataset row
var val1 = row[0].text;
//get the value of the second item in the dataset row
var m = row[1].text;
//add those values to the html variable
html += "value 1: " + val1 + " expression value: " + m + "<br />";
//insert the html from the html variable into the extension.
$("#" + divName).html(html);

//Initiate extension

Now the CoffeeScript version. A lot more readable, at least to me:

[code language=”text”]
template_path = Qva.Remote + "?public=only&name=Extensions/template_simple_coffeescript/"

extension_Init = ->
# Use QlikView’s method of loading other files needed by an extension. These files should be added to your extension .zip file (.qar)
if typeof jQuery == ‘undefined’
Qva.LoadScript(template_path + ‘jquery.js’, extension_Done)

extension_Done = ->
# Add extension
Qva.AddExtension(‘template_simple_coffeescript’, ->
_this = this

# add a unique name to the extension in order to prevent conflicts with other extensions.
# basically, take the object ID and add it to a DIV
divName = _this.Layout.ObjectId.replace("\", "_")
if _this.Element.children.length == 0
# if this div doesn’t already exist, create a unique div with the divName
ui = document.createElement("div")
ui.setAttribute("id", divName)
# if it does exist, empty the div so we can fill it again
$("#" + divName).empty()

# create a variable to put the html into
html = ""

# set a variable to the dataset to make things easier
td = _this.Data

# loop through the data set and add the values to the html variable
for rowIx in [0..(td.Rows.length-1)]

# set the current row to a variable
row = td.Rows[rowIx]

# get the value of the first item in the dataset row
val1 = row[0].text

# get the value of the second item in the dataset row
m = row[1].text

# add those values to the html variable
html += "value 1: " + val1 + " expression value: " + m + "<br />"

# insert the html from the html variable into the extension.
$("#" + divName).html(html)

# Initiate extension

Note that you need to include the –bare option when compiling the CoffeeScript code:

[code language=”bash”]

coffee –bare –compile


This will give us the following Javascript file, which is functionally equivalent to the first JS file above:

[code language=”text”]
// Generated by CoffeeScript 1.6.3
var extension_Done, extension_Init, template_path;

template_path = Qva.Remote + "?public=only&name=Extensions/template_simple_coffeescript/";

extension_Init = function() {
if (typeof jQuery === ‘undefined’) {
return Qva.LoadScript(template_path + ‘jquery.js’, extension_Done);
} else {
return extension_Done();

extension_Done = function() {
return Qva.AddExtension(‘template_simple_coffeescript’, function() {
var divName, html, m, row, rowIx, td, ui, val1, _i, _ref, _this;
_this = this;
divName = _this.Layout.ObjectId.replace("\", "_");
if (_this.Element.children.length === 0) {
ui = document.createElement("div");
ui.setAttribute("id", divName);
} else {
$("#" + divName).empty();

html = "";
td = _this.Data;
for (rowIx = _i = 0, _ref = td.Rows.length – 1; 0 <= _ref ? _i <= _ref : _i >= _ref; rowIx = 0 <= _ref ? ++_i : –_i) {
row = td.Rows[rowIx];
val1 = row[0].text;
m = row[1].text;
html += "value 1: " + val1 + " expression value: " + m + "<br />";
return $("#" + divName).html(html);