Posted on

Butler Spyglass: Data lineage and metadata tool for Qlik Sense

Latest member of the Butler family

For years I have thought about ways to get data lineage info for all apps in a Qlik Sense Enterprise environment.

It would be super useful to know exactly what apps use a particular data source, as well as vice versa (what data sources are used by a specific app). I know there are commercial tools doing this and much more, but I wanted something easy to use, yet still effective and free.

Same thing for app load scripts: Extracting and storing them to disk in human readable format has more than once save days of work, when something has gone badly wrong in an app.
Dumping load scripts to disk was possible in my original Butler tool, but then only one app at a time. So not quite what was needed in an enterprise context.

Enter Butler Spyglass. 

Continue reading “Butler Spyglass: Data lineage and metadata tool for Qlik Sense”
Posted on

Monitoring for Qlik Sense: Butler SOS v3 is out

The latest version fo Butler SOS is out, taking the version number to 3.0.
A lot of the code has been fine tuned to better meet the needs of enterprise grade Qlik Sense deployments.

Docker (or some compatible container platform) is now the preferred and recommended way of running Butler SOS. Butler SOS has been developed and tested on Linux and Mac OS, but should in theory run also other Docker enabled platforms.

New features

Version 3 adds a few – but useful – features:

  • Per-server config option “serverGroup”. Use this to group or categorize servers, for example as being part of a production vs development Qlik Sense cluster. This enables the creation of Grafana dashboards that use Grafana variables to automatically show metrics for all PRODUCTION servers. This greatly simplifies using Butler SOS in large Qlik Sense environments, where servers are frequently added/removed. No need to manually update the Grafana dashboards any more.
  • Config option “queryPeriod” for controlling how far back querying for Sense log entries should be done. Used together with the logdb.pollingInterval setting, it is now possible to fine tune how often the Qlik Sense log database is queried for errors and warnings.
Continue reading “Monitoring for Qlik Sense: Butler SOS v3 is out”
Posted on

The beauty of Docker – how to run all Butler tools with a single command

Docker is great.

Docker is one of those tools that have the potential to fundamentally transform how you develop and run software – once you have tried Docker it is hard to imagine going back to something else.

In previous posts we have seen how Butler, Butler SOS and Butler CW can be run as Docker containers.
But we can do even better – why not control all the Butler tools from a single docker-compose file? Maybe even specifying the dependencies on influxdb and mqtt in there too?

Setting this up is incredibly easy – a single docker-compose file tells Docker what containers to use, and some config files tells the Butler tools where to find things.

Let’s get started!

Continue reading “The beauty of Docker – how to run all Butler tools with a single command”

Posted on

Docker everywhere – Qlik Sense operations monitor using Butler SOS

Following up on previous posts (here and here) about the Butler family of tools being Dockerized, here is another one on the same topic:

Butler SOS can now be run in a Docker container. 

This is good news as it makes it a lot easier to set up real-time monitoring of a Qlik Sense enterprise environment, compared to the previous (still working, btw) method of installing Node.js and then running Butler SOS on top of Node.

The Github repository has all the details – head over there to get the latest release of the code as well as instructions how to run Butler SOS using Docker.

The Docker image is available on Docker Hub, if you want to get started right away.

 

Posted on

Turning off the alarm system – and the lights come on!

Following up on the previous post about pulling sensor data from sensors in a Verisure alarm system, here is a slightly more hardware focused post.

The Verisure – USB – ESP8266 sandwich

One use case that the new Verisure alarm system does not handle nicely is machine-to-machine notification when the alarm is disarmed (turned off). Such notifications are sent to the mobile app – so the data exists somewhere. The problem is that it’s all undocumented and thus hard to build something on top of.

Let’s hack a solution instead – be aware, some soldering required….

Continue reading “Turning off the alarm system – and the lights come on!”

Posted on

Reading sensor data from a Verisure home alarm system

I recently upgraded our house alarm system to the latest model from Verisure.

Pretty nice actually, the new system comes with a good mobile app, remote querying of both alarm status and status of the various sensors (smoke, movement etc) that are included in the system. Oh, the new system actually cost less per month too, compared to the old one. Go figure.

Being the geek that I am, I immediately wondered if it was possible to get hold of the sensor readings (temperature and humidity) that were shown in the mobile app.

Turns out it is pretty simple. There is a REST API which can be queried, and from there you get back all kinds of status for the system. With that data nicely structured in a JSON it’s then a breeze to send the data to a MQTT pub-sub broker for later use by whatever system that might need it. Easy in theory.

Continue reading “Reading sensor data from a Verisure home alarm system”

Posted on

Butler Cache Warmer + Docker = True

More Docker in the Butler family!

Today version 2.0 of Butler CW was released, including a Docker image and configuration files needed to run Butler CW as a Docker container.

Feature wise the stand-alone Node.js app and the new container-based solution are identical, it’s just way more convenient to run it as a Docker container.

Additional info is available in the GitHub repository.

Yesterday the original Butler tool was Dockerized, today Butler CW… Do you see the trend?

Continue reading “Butler Cache Warmer + Docker = True”

Posted on

Hey Butler, please meet Docker

Before the summer the Butler tool turned two years old – time flies!

Over those years I have installed, tweaked and upgraded a fair number of Butler instances… Not a problem per se, but maintaining a production grade Butler instance does assume a certain level of experience around Node.js, Linux, networking etc.

The most recent Butler version (v2.2) attempts to make it easier to deploy and operate Butler.  This is achieved by deploying Butler as a Docker container instead of a regular Node.js app.

The Docker image (from which a container is created) contains exactly the same Node.js app that you can run right on your server or laptop – i.e. there is no functional difference what so ever between running the Node app natively, and running it as a Docker container.

There are some significant benefits of running Butler under Docker:

Continue reading “Hey Butler, please meet Docker”

Posted on

SOLVED: Not possible to install new nodes in Qlik Sense June 2018 environment

I worked out the issue preventing me to add additional nodes to a Qlik Sense Enterprise suystem whose central node had been updated to June 2018.

The June 2018 version of Qlik Sense Enterprise relies on a database called “SenseServices” to be available in the Postgres db (as in earlier versions, Postgres can run on the central node or on some other server).

When running the June 2018 installer on a new server (which is to be added as a new node in a Sense Enterprise cluster), the installer verifies that the SenseServices database exists. If it does not, you get an error half-way through the installer:

SenseServices db is missing

 

Maybe there is info somewhere that you need to create a new “SenseServices” db in Postgress before installing additional Sense nodes… but I looked (a lot) without finding anything.

Continue reading “SOLVED: Not possible to install new nodes in Qlik Sense June 2018 environment”