Category Archives: MQTT

Butler SOS goes main stage at Qonnections 2017

Edit:

Qlik has made available a video from Anthony Deighton’s (Qlik’s CTO) keynote session.

Very much worth looking through the entire video – it was very interesting indeed. 
His highlight of customer and partner project’s is found 12 min 36 sec into the video.

 

 

A few weeks back something cool happened.

During the big, main stage keynote session at Qlik‘s annual Qonnections conference, my Butler SOS project was highlighted as an example of a great application built on top of the Sense APIs.

I have to admit that it is pretty neat to get that kind of recognition from industry peers… a bit of ego boost :).

And it makes it easier and even more fun to work out coming, similar concepts and services. Lots more to be done around DevOps and Qlik Sense.

Speaking of which:
I started playing around with Qlik’s new Enigma.js library a week or so ago, to see if it could be used in coming projects.

Very nice indeed. Nothing wrong with good old Qsocks, but Enigma is a step in the right direction. Just like qsocks, Enigma is open source – but also fully supported by Qlik. That is a huge advantage!

Let there be (blinky) light!

I was recently in Helsinki, giving a talk at a QlikDevGroup event. Great event, great crowd. The topic was SenseOps and Butler SOS, and I showcased the lamp above as an example of a funky, but still relevant way to monitor user activity in a Qlik Sense Enterprise environment.

A person in the audience asked how the map works. I claimed it was super simple, costing less than USD 10 to build (assuming you already have a suitable enclosure) and uses just four wires hooked up between some pre-made modules. Time to prove it.

The four wires part might have been a slight exaggeration… but it depends on which wires you count – right?

Continue reading

Butler SOS: real-time server stats for Qlik Sense

I got somewhat distracted from the idea of breaking up the existing Butler software into smaller, stand-alone micro services.
Or rather, an idea came to mind. An idea too good not to explore…

The healthcheck API of Qlik Sense provides basic metrics for both the Qlik Sense engine itself, and the server it is running on. Things like CPU load, available RAM, number of connected users and what apps are loaded into the Sense engine.

The idea behind Butler SOS ( SOS = SenseOps Stats) is very simple:
Get the healthcheck metrics for all servers in a Sense cluster. Then send the information to MQTT for immediate, real-time use cases.
It is directly aimed at bringing better features to the monitoring step of SenseOps – please visit SenseOps.rocks for more info on SenseOps.

Butler SOS is nice and sending data via MQTT make the health metrics available in for example Node-RED. Node-RED has some basic graph options, but not anywhere near those offered by Grafana. Grafana is very, very cool… A live demo is available here – do check it out – it is very nice indeed.

Creating real-time dashboards in Grafana is greatly simplified if the data is stored in some kind of time series database. Influxdb is an obvious choice. It is open source, installation is very easy, and there are good Node.js libraries that make it trivial to insert data into a Influxdb database.
Thus – Butler SOS also sends the Sense health metrics to an Influx db of choice.

Only need Influxdb and not MQTT? Or the other way around?
No problem, the Butler SOS config file include options for independently turning on/off sending of data to MQTT and Influxdb.

Butler SOS, including a sample Grafana dashboard, is available on GitHub.

Using Grafana we get dashboards and charts like these:

Showing CPU load, available RAM, active users, cache hit ratio and more.

A 100% cache hit ratio (left y axis) means that no additional calculations were needed to serve the requested data to the client. The yellow dots are associated with the right y axis, which tells us how big the request in question was.

 

Continue reading

Butler-MQTT and a realtime reload dashboard for Sense

Teaser video: Realtime reload dashboard

Just to get your interest – this video shows what Butler-MQTT can be used for. All the pieces are included in the Github repo. The concept of a reload dashboard can be very useful if you have long running (hours!) reloads. Instead of relying on Sense’s own reload log window, you can get an easy to understand visual feedback from a reload dashboard.

The video shows

  1. A reloading Sense app (upper left). The app has two nested loops (date and country), with half a second delay in each.
  2. Within each loop a status message is sent to  Butler-MQTT (bottom left), which forwards the message to a MQTT server running on localhost.
  3. A Node-RED dashboard picks up the MQTT messages and render a dashboard showing the progress of the running reload.

The “Sales” line chart deserves some comment too.
In the outer loop in the app’s load script, the script calculates sum(sales) for that particular loop iteration. That value is then sent to a MQTT topic, and then plotted in the dashboard.
This way you can get an immediate, visual feedback on the actual data produced by the reload. Not by any means always needed – but it can also be very, very useful.

Continue reading

Sense + DevOps = SenseOps

For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?

Some initiatives are starting to appear. Qlik themselves are open sourcing microservices such as the Governed Metrics Service , as well as open sourcing key libraries that make it easier to use Sense as a general purpose analytics platform.  I have a couple of contributions in this space too, with the Butler and App duplicator microservices.

I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….

 

SenseOps

 

The more I looked at it, the more I felt “wow – SenseOps really rocks!”

Thus, please meet a new site: senseops.rocks

Head over for a look, and feel free to feedback in the discussion forums there.
Enjoy!

 

Butler for Qlik Sense – connecting Sense to the world

QlikLogoThe goal of the previously described Slack proxy project was to allow posting of messages from Qlik Sense load scripts to Slack (great instant messaging platform!). The project has however developed into something larger and slightly more ambitious, integrating other connections than Slack. The original project name “Slack proxy” has become less and less relevant, and the project has thus been renamed to “Butler”.

As of right now (May 2016), Butler’s feature set include

  • Send messages from Sense load script to Slack
  • Send/publish MQTT messages from Sense load script (i.e. outbound MQTT).
  • Sense reload failures as emails, and to Slack.
  • Sense audit events (session start/stop and connection open/close) to Slack and to MQTT messages.
  • Create new directories on the Sense server’s (where Butler is running) disks.
  • Get disk space info, for disks on the Sense server where Butler is running.

Butler version 1.1 was just released, please refer to the Butler GitHub repository for further information. Feel free to fork it and contribute if you feel some feature is missing –  the node.js app should be pretty easy to understand and extend upon.

Using Node-RED for publishing Telldus Live data to MQTT

Tellstick Net and DuoTelldus has a set of nice little gadgets (“Tellstick”, for short) that both allow you to control remote switches over radio (433.92 MHz), and to read sensors transmitting on that same frequency. Telldus also has a backend service, Telldus Live, which offer Tellstick users scheduling features (turning lamps on/off at certain times, or when certain conditions occur), as well as showing the latest sensor readings.

The above is at least true if you have a Tellstick Net, which connects to your home network and sends device and sensor data to the Telldus Live service. You can also achieve the same thing with the non-connected Tellstick models, and an always-on computer running Telldus’ software.

Anyway – let’s assume that Telldus Live can see your switches, sensors and other connected devices. Would it not be cool if you could bring all that data into Node-RED, and from there create whatever feature you dreamt of.

How about sending an SMS when the  garage door is still open, but your presence data indicate that you have left for work? Easy.

Or the opposite: Send a tweet to your Node-RED server, which will then fire off an event to Telldus Live, turning a switch on, and by doing so closing the garage door? No problem.

Continue reading