Sense + DevOps = SenseOps

For some time I have been thinking about how to improve the Sense development process as a whole. There is a lot of gathered experience and best practices from the wider software development community, but how can we apply this to Qlik Sense development?

Some initiatives are starting to appear. Qlik themselves are open sourcing microservices such as the Governed Metrics Service , as well as open sourcing key libraries that make it easier to use Sense as a general purpose analytics platform.  I have a couple of contributions in this space too, with the Butler and App duplicator microservices.

I think more can be done though.
Looking at the concepts promoted in DevOps, it struck me that Sense development follows about the same phases as those in DevOps. Combining Sense and DevOps of course gives us….

 

SenseOps

 

The more I looked at it, the more I felt “wow – SenseOps really rocks!”

Thus, please meet a new site: senseops.rocks

Head over for a look, and feel free to feedback in the discussion forums there.
Enjoy!

 

Butler for Qlik Sense version 2.0

Butler 2.0 is out!

Champagne

A summer project is over, and Butler  has been released in a version that is a total rewrite of Butler version 1.x. Butler 2.0 includes features such as

  • Real-time monitoring of what users are active on a Sense Enterprise system
  • Real-time monitoring of failed reload tasks, with notifications sent as emails, to Slack channels and as MQTT messages
  • Starting Sense reload tasks from the load script or from external systems, using a REST API
  • Posting messages to Slack from the reload script. Great for sysadmin notifications as well as notifying end users that new data is available in an app
  • Sending MQTT pub-sub messages from the load script
  • Get full metadata, including load script, for any app
  • Getting free disk info for the Sense server
  • …and more

All in all, Butler 2.0 includes a set of features that offers greatly enhanced features for anyone involved in the operation of a Sense Enterprise environment.

Butler is available via Qlik Branch, with all source code hosted on GitHub.
Documentation is available on github.io. That documentation is the best starting point for learning more about what Butler does and how to use it.

Some sample screen shots shows what is possible to achieve:

Continue reading “Butler for Qlik Sense version 2.0”

Butler for Qlik Sense – connecting Sense to the world

QlikLogoThe goal of the previously described Slack proxy project was to allow posting of messages from Qlik Sense load scripts to Slack (great instant messaging platform!). The project has however developed into something larger and slightly more ambitious, integrating other connections than Slack. The original project name “Slack proxy” has become less and less relevant, and the project has thus been renamed to “Butler”.

As of right now (May 2016), Butler’s feature set include

  • Send messages from Sense load script to Slack
  • Send/publish MQTT messages from Sense load script (i.e. outbound MQTT).
  • Sense reload failures as emails, and to Slack.
  • Sense audit events (session start/stop and connection open/close) to Slack and to MQTT messages.
  • Create new directories on the Sense server’s (where Butler is running) disks.
  • Get disk space info, for disks on the Sense server where Butler is running.

Butler version 1.1 was just released, please refer to the Butler GitHub repository for further information. Feel free to fork it and contribute if you feel some feature is missing –  the node.js app should be pretty easy to understand and extend upon.

Posting to Slack – and more – from Qlik Sense load scripts (part 3)

Quick update to the previous post on this topic.

There is a new version of slack_proxy available on GitHub. It adds a new endpoint for creating directories on the server where slack_proxy is running.

This obviously means that the apps name – “slack_proxy” – is not really that relevant or correct any longer… And with more end points considered, the name will become even less correct…. Oh well – I’ll keep the name for now – maybe it will change sometime up ahead.

The endpoints currently supported are:

  • /slack for posting to Slack
  • /createDir for creating directories on local disk

Other endpoints on the radar are for sending tweets, sending messages to Pushover, and controlling Blink(1) USB lights.

There is also now reasonably complete documentation on GitHub.

Posting to Slack from Qlik Sense load scripts – web service style (part 2)

As the first version of the Sense-to-Slack integration required you to lower the Sense security level, it is probably not a good option for enterprise Sense users, for example.

Instead, let’s set up a small node.js web service, which exposes an API that we can call just like we can retrieve data from any web page in the load script.

There are a couple of pre-requisites:

  1. Node.js must be installed on the machine where Sense is running (or some other machine in your network – for sake of simplicity I assume it’s on the same machine). You also need to have the Node libraries Restify and Node-Slack installed.
  2. You also need Qlik’s REST connector, at least if you are to avoid some messy URL character encodings. More on that later.

Setting up the Node.js server

Create a directory where you will store your Node apps. C:\node or something similar – you decide what’s suitable on your system. Then, get the code for the node server from GitHub. The code is pretty basic (make sure to use your own Slack web hook URL):

var restify = require('restify');
var Slack = require('node-slack');

var slackWebhookURL = '<fill in your web hook URL from Slack>';
var slack = new Slack(slackWebhookURL);

function respondSlack(req, res, next) {
// console.log(req.params);

slack.send({
 text: req.params.msg,
 channel: req.params.channel,
 username: req.params.from_user,
 icon_emoji: req.params.emoji
 });

res.send(req.params);
 next();
}

var server = restify.createServer({
 name: 'SlackProxy'
});

server.use(restify.queryParser()); // Enable parsing of http parameters
server.get('/slack', respondSlack);

server.listen(8080, function() {
 console.log('%s listening at %s', server.name, server.url);
});

Run the server:

C:\node\slack_proxy>node slack_proxy.js
SlackProxy listening at http://[::]:8080

Log Slack entries from Sense load script, using the REST connector

Connection to Slack proxy

Now, create a new Sense app to try out the notification. In that app, set up a new data connection using the REST connector:

With the new connection in place, use the rest connector to connect to the proxy, and let the REST connector generate the code for you.

Finally, add the “WITH CONNECTION”… statement to send in the actual data that should be passed on to Slack. A good thing here is that the REST connector handles encoding of special characters like space, # etc – no need to do that manually. Very nice!

The resulting load script looks something like this (your data connection names will differ, of course):

LIB CONNECT TO 'Slack';

RestConnectorMasterTable:
SQL SELECT 
 "channel",
 "from_user",
 "msg",
 "emoji"
FROM JSON (wrap on) "root"
WITH CONNECTION (
 QUERY "channel" "#general",
 QUERY "from_user" "SenseBot",
 QUERY "msg" "Posted from Sense",
 QUERY "emoji" ":smile:"
 );

Slack:
LOAD 
 [channel] AS [channel],
 [from_user] AS [from_user],
 [msg] AS [msg],
 [emoji] AS [emoji]
RESIDENT RestConnectorMasterTable;

DROP TABLE RestConnectorMasterTable;

Reload the app, and voila – in Slack we get this:

Slack message created by Sense load script

Mission accomplished.

Log Slack entries from Sense load script, using the FROM command

If you don’t want to use the REST connector, or don’t have it installed, there is another way of achieving the same result as above.

Your Sense load script will instead look something like this:

Slack:
LOAD
 *
FROM [http://localhost:8080/slack?channel=%23general&from_user=SenseBot2&msg=Another%20message%20from%20Sense&emoji=:smile:]
(txt, codepage is 1252, embedded labels, delimiter is '\t', msq);

Notice how we have to convert space, # and other “special” characters to their hex equivalents? You could write a small function that does this for you (don’t think Sense has any built in function for URL escaping – or?), but with the REST connector being so easy to use – that should be the preferred option for most people.

In this case we get a Slack message like this:

Slack message using FROM command

 

 

Posting to Slack from Qlik Sense load scripts, part 1

When thinking about it, it’s kind of strange that Slack has been so enormously successful.

Sure, it’s a great instant messaging tool, but it’s not a new concept – IM tools have been around for decades, and IRC, Google chat and others rose to be very dominant and present in everyone’s minds.

Still, Slack does bring something new with the ease of setup, the slick user interface, really good clients on both web, Windows, OSX, IOS and Android. And more than anything else, they provide a very extensive set of integrations (also mentioned in earlier post on great OSX tools), which makes it a breeze to have other systems send messages, status notifications etc to your Slack channel. Or vice versa – messages in Slack can be sent to those other systems. As of this writing the list of integrations is more than 85 entries long, including services like Dropbox, IFTTT, Twitter, various email integrations, Jira etc.

Continue reading “Posting to Slack from Qlik Sense load scripts, part 1”

Using Slack to monitor Qlikview and Qlik Sense

Slack has only been around since August 2013, but I would definitively say it’s one of the better Slack integrationsteam communication services out there. The web client is great, and the OSX and IOS clients are truly awesome. It integrates with tons of other services, including Dropbox, GitHub, IFTTT, Jira, Google Drive, RSS, Nagios, Yo, Twitter and Pingdom are just some of the services it supports (as of today they seem to support ca 75 integrations). There are also generic connectors for incoming and outgoing webhooks, especially the incoming webhook feature will be interesting from a systems monitoring perspective – it will allows us to post messages to Slack by just calling a certain URL.

In this post we will look how we can use this to both monitor the various QlikView services, as well as monitor the transfer of files (e.g. data files used by QV) to a QV server, and a as a generic way of sending notifications from QlikView Management Console (QMC).

Most of the concepts below also apply to Qlik Sense, of course.

Continue reading “Using Slack to monitor Qlikview and Qlik Sense”