When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.
A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.
This leads us to the conclusion that the services must automatically be
a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.
Enter process monitors.
At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).
Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.
Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.
Continue reading “Monitoring and auto-starting Node.js services on Windows Server”
On the surface everything looks the same.
Under the hood everything has changed.
The 2.0 release of the Qlik Sense app duplicator is a major update. Most code has been refactored.
The most notable changes are:
- Switched to using Enigma.js for talking to the Sense engine, instead of the previously used Qsocks library. Enigma is open source just like Qsocks, but the former is supported and actively developed by Qlik themselves – that is really important.
- Much improved logging.
- A nasty bug has been fixed.
The app duplicator would fail semi-silently when multiple app duplication requests were sent to it simultaneously. No more of that – the duplicator now handles also high-load scenarios gracefully, and have been battle-tested in hands-on sessions when many developers created their own apps from templates at the same time. Nice.The 2.0 version is available on GitHub, as always.
Qlik has made available a video from Anthony Deighton’s (Qlik’s CTO) keynote session.
Very much worth looking through the entire video – it was very interesting indeed.
His highlight of customer and partner project’s is found 12 min 36 sec into the video.
A few weeks back something cool happened.
During the big, main stage keynote session at Qlik‘s annual Qonnections conference, my Butler SOS project was highlighted as an example of a great application built on top of the Sense APIs.
I have to admit that it is pretty neat to get that kind of recognition from industry peers… a bit of ego boost :).
And it makes it easier and even more fun to work out coming, similar concepts and services. Lots more to be done around DevOps and Qlik Sense.
Speaking of which:
I started playing around with Qlik’s new Enigma.js library a week or so ago, to see if it could be used in coming projects.
Very nice indeed. Nothing wrong with good old Qsocks, but Enigma is a step in the right direction. Just like qsocks, Enigma is open source – but also fully supported by Qlik. That is a huge advantage!
I was recently in Helsinki, giving a talk at a QlikDevGroup event. Great event, great crowd. The topic was SenseOps and Butler SOS, and I showcased the lamp above as an example of a funky, but still relevant way to monitor user activity in a Qlik Sense Enterprise environment.
A person in the audience asked how the map works. I claimed it was super simple, costing less than USD 10 to build (assuming you already have a suitable enclosure) and uses just four wires hooked up between some pre-made modules. Time to prove it.
The four wires part might have been a slight exaggeration… but it depends on which wires you count – right?
Continue reading “Let there be (blinky) light!”
For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?
Some initiatives are starting to appear. Qlik themselves are open sourcing microservices such as the Governed Metrics Service , as well as open sourcing key libraries that make it easier to use Sense as a general purpose analytics platform. I have a couple of contributions in this space too, with the Butler and App duplicator microservices.
I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….
The more I looked at it, the more I felt “wow – SenseOps really rocks!”
Thus, please meet a new site: senseops.rocks
Head over for a look, and feel free to feedback in the discussion forums there.
For a long time I wanted to learn more about client side development.
Javascipt, CSS, reactive frameworks etc – all those concepts that make today’s web sites tick. I really think backend developers can benefit from having at least a basic understanding of how the front-end works – this will make the finished app better as a whole.
Couple this with a desire to learn Vue.js specifically, and we get a user interface for the Qlik Sense app duplicator service.
Continue reading “Qlik Sense app duplicator 1.2: User interface included”
I worked on a small node.js server during the past couple of weeks. The idea came from the fact that
- I over and over again find myself creating Sense apps that are almost identical, and
- people starting out as Sense developers spend too much time learning the basics. I wanted to bootstrap their learning process by providing well written skeleton apps for them.
This could of course be achieved by just duplicating apps using the QMC, but it would be way better if there was a nice little web app that listed the available Sense app templates, and allowed anyone (with permissions to create apps) to create new apps based on them. Or maybe this feature could even be integrated into one of the different Sense hubs that are now available…
Continue reading “App duplicator service for Qlik Sense”
Butler 2.0 is out!
A summer project is over, and Butler has been released in a version that is a total rewrite of Butler version 1.x. Butler 2.0 includes features such as
- Real-time monitoring of what users are active on a Sense Enterprise system
- Real-time monitoring of failed reload tasks, with notifications sent as emails, to Slack channels and as MQTT messages
- Starting Sense reload tasks from the load script or from external systems, using a REST API
- Posting messages to Slack from the reload script. Great for sysadmin notifications as well as notifying end users that new data is available in an app
- Sending MQTT pub-sub messages from the load script
- Get full metadata, including load script, for any app
- Getting free disk info for the Sense server
- …and more
All in all, Butler 2.0 includes a set of features that offers greatly enhanced features for anyone involved in the operation of a Sense Enterprise environment.
Butler is available via Qlik Branch, with all source code hosted on GitHub.
Documentation is available on github.io. That documentation is the best starting point for learning more about what Butler does and how to use it.
Some sample screen shots shows what is possible to achieve:
Continue reading “Butler for Qlik Sense version 2.0”
Continuing from the previous post about publishing data from Telldus Live to MQTT, here is small piece of related Node-RED code. It outputs a list of all (wirelessly controlled) 220V/mains switches I have linked to Telldus Live, using a Tellstick Net.
The background of the code was simply that I wanted a convenient way of getting a complete list of all devices (switches, dimmers, door bells, …) known to my Telldus Live account. When testing different devices, moving them around, renaming them etc, it is very easy to loose track of which device does what, and what their respective IDs are. This little function solves that quite nicely.
Continue reading “Listing all Telldus Live devices in Node-RED”
Telldus has a set of nice little gadgets (“Tellstick”, for short) that both allow you to control remote switches over radio (433.92 MHz), and to read sensors transmitting on that same frequency. Telldus also has a backend service, Telldus Live, which offer Tellstick users scheduling features (turning lamps on/off at certain times, or when certain conditions occur), as well as showing the latest sensor readings.
The above is at least true if you have a Tellstick Net, which connects to your home network and sends device and sensor data to the Telldus Live service. You can also achieve the same thing with the non-connected Tellstick models, and an always-on computer running Telldus’ software.
Anyway – let’s assume that Telldus Live can see your switches, sensors and other connected devices. Would it not be cool if you could bring all that data into Node-RED, and from there create whatever feature you dreamt of.
How about sending an SMS when the garage door is still open, but your presence data indicate that you have left for work? Easy.
Or the opposite: Send a tweet to your Node-RED server, which will then fire off an event to Telldus Live, turning a switch on, and by doing so closing the garage door? No problem.
Continue reading “Using Node-RED for publishing Telldus Live data to MQTT”