Teaser video: Realtime reload dashboard
Just to get your interest – this video shows what Butler-MQTT can be used for. All the pieces are included in the Github repo. The concept of a reload dashboard can be very useful if you have long running (hours!) reloads. Instead of relying on Sense’s own reload log window, you can get an easy to understand visual feedback from a reload dashboard.
The video shows
- A reloading Sense app (upper left). The app has two nested loops (date and country), with half a second delay in each.
- Within each loop a status message is sent to Butler-MQTT (bottom left), which forwards the message to a MQTT server running on localhost.
- A Node-RED dashboard picks up the MQTT messages and render a dashboard showing the progress of the running reload.
The “Sales” line chart deserves some comment too.
In the outer loop in the app’s load script, the script calculates sum(sales) for that particular loop iteration. That value is then sent to a MQTT topic, and then plotted in the dashboard.
This way you can get an immediate, visual feedback on the actual data produced by the reload. Not by any means always needed – but it can also be very, very useful.
Continue reading “Butler-MQTT and a realtime reload dashboard for Sense”
For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?
Some initiatives are starting to appear. Qlik themselves are open sourcing microservices such as the Governed Metrics Service , as well as open sourcing key libraries that make it easier to use Sense as a general purpose analytics platform. I have a couple of contributions in this space too, with the Butler and App duplicator microservices.
I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….
The more I looked at it, the more I felt “wow – SenseOps really rocks!”
Thus, please meet a new site: senseops.rocks
Head over for a look, and feel free to feedback in the discussion forums there.
Butler 2.0 is out!
A summer project is over, and Butler has been released in a version that is a total rewrite of Butler version 1.x. Butler 2.0 includes features such as
- Real-time monitoring of what users are active on a Sense Enterprise system
- Real-time monitoring of failed reload tasks, with notifications sent as emails, to Slack channels and as MQTT messages
- Starting Sense reload tasks from the load script or from external systems, using a REST API
- Posting messages to Slack from the reload script. Great for sysadmin notifications as well as notifying end users that new data is available in an app
- Sending MQTT pub-sub messages from the load script
- Get full metadata, including load script, for any app
- Getting free disk info for the Sense server
- …and more
All in all, Butler 2.0 includes a set of features that offers greatly enhanced features for anyone involved in the operation of a Sense Enterprise environment.
Butler is available via Qlik Branch, with all source code hosted on GitHub.
Documentation is available on github.io. That documentation is the best starting point for learning more about what Butler does and how to use it.
Some sample screen shots shows what is possible to achieve:
Continue reading “Butler for Qlik Sense version 2.0”
The goal of the previously described Slack proxy project was to allow posting of messages from Qlik Sense load scripts to Slack (great instant messaging platform!). The project has however developed into something larger and slightly more ambitious, integrating other connections than Slack. The original project name “Slack proxy” has become less and less relevant, and the project has thus been renamed to “Butler”.
As of right now (May 2016), Butler’s feature set include
- Send messages from Sense load script to Slack
- Send/publish MQTT messages from Sense load script (i.e. outbound MQTT).
- Sense reload failures as emails, and to Slack.
- Sense audit events (session start/stop and connection open/close) to Slack and to MQTT messages.
- Create new directories on the Sense server’s (where Butler is running) disks.
- Get disk space info, for disks on the Sense server where Butler is running.
Butler version 1.1 was just released, please refer to the Butler GitHub repository for further information. Feel free to fork it and contribute if you feel some feature is missing – the node.js app should be pretty easy to understand and extend upon.