Tag Archives: senseops

Monitoring and auto-starting Node.js services on Windows Server

When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.

A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.

This leads us to the conclusion that the services must automatically be

a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.

Enter process monitors.

At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).

Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.

Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.

Continue reading

Butler SOS goes main stage at Qonnections 2017

Edit:

Qlik has made available a video from Anthony Deighton’s (Qlik’s CTO) keynote session.

Very much worth looking through the entire video – it was very interesting indeed. 
His highlight of customer and partner project’s is found 12 min 36 sec into the video.

 

 

A few weeks back something cool happened.

During the big, main stage keynote session at Qlik‘s annual Qonnections conference, my Butler SOS project was highlighted as an example of a great application built on top of the Sense APIs.

I have to admit that it is pretty neat to get that kind of recognition from industry peers… a bit of ego boost :).

And it makes it easier and even more fun to work out coming, similar concepts and services. Lots more to be done around DevOps and Qlik Sense.

Speaking of which:
I started playing around with Qlik’s new Enigma.js library a week or so ago, to see if it could be used in coming projects.

Very nice indeed. Nothing wrong with good old Qsocks, but Enigma is a step in the right direction. Just like qsocks, Enigma is open source – but also fully supported by Qlik. That is a huge advantage!

Let there be (blinky) light!

I was recently in Helsinki, giving a talk at a QlikDevGroup event. Great event, great crowd. The topic was SenseOps and Butler SOS, and I showcased the lamp above as an example of a funky, but still relevant way to monitor user activity in a Qlik Sense Enterprise environment.

A person in the audience asked how the map works. I claimed it was super simple, costing less than USD 10 to build (assuming you already have a suitable enclosure) and uses just four wires hooked up between some pre-made modules. Time to prove it.

The four wires part might have been a slight exaggeration… but it depends on which wires you count – right?

Continue reading

Butler SOS: real-time server stats for Qlik Sense

I got somewhat distracted from the idea of breaking up the existing Butler software into smaller, stand-alone micro services.
Or rather, an idea came to mind. An idea too good not to explore…

The healthcheck API of Qlik Sense provides basic metrics for both the Qlik Sense engine itself, and the server it is running on. Things like CPU load, available RAM, number of connected users and what apps are loaded into the Sense engine.

The idea behind Butler SOS ( SOS = SenseOps Stats) is very simple:
Get the healthcheck metrics for all servers in a Sense cluster. Then send the information to MQTT for immediate, real-time use cases.
It is directly aimed at bringing better features to the monitoring step of SenseOps – please visit SenseOps.rocks for more info on SenseOps.

Butler SOS is nice and sending data via MQTT make the health metrics available in for example Node-RED. Node-RED has some basic graph options, but not anywhere near those offered by Grafana. Grafana is very, very cool… A live demo is available here – do check it out – it is very nice indeed.

Creating real-time dashboards in Grafana is greatly simplified if the data is stored in some kind of time series database. Influxdb is an obvious choice. It is open source, installation is very easy, and there are good Node.js libraries that make it trivial to insert data into a Influxdb database.
Thus – Butler SOS also sends the Sense health metrics to an Influx db of choice.

Only need Influxdb and not MQTT? Or the other way around?
No problem, the Butler SOS config file include options for independently turning on/off sending of data to MQTT and Influxdb.

Butler SOS, including a sample Grafana dashboard, is available on GitHub.

Using Grafana we get dashboards and charts like these:

Showing CPU load, available RAM, active users, cache hit ratio and more.

A 100% cache hit ratio (left y axis) means that no additional calculations were needed to serve the requested data to the client. The yellow dots are associated with the right y axis, which tells us how big the request in question was.

 

Continue reading