Qlik Sense data model of the year

Qlik-Sense-Logo1Long overdue, but during last few months I have finally started to play around with the various APIs available on the Qlik Sense platform.

It has been said before, but it is worth repeating: those APIs are really, really powerful. They are one of the main reasons why I think Qlik have some good years ahead of them.

Anyway – a recent experiment involved serialising apps to JSON text files, so they can be stored in Git in an effective way. Works really well and is surprisingly simple to achieve thanks to the groundwork done by members of the Qlik developer community.
So far so good, but the JSON blob describing a Sense app is extremely large and hard to navigate. Continue reading

Qlik Sense REST connector log file overload

Qlik-Sense-Logo1I recently investigated a strange behaviour of the Qlik Sense REST connector, where it would become slower and slower executing REST queries, the longer the server had been running. RAM or CPU consumption was not unusually high, but something was clearly eating away some kind of resources from the system. Turned out it was disk access that was the bottleneck.

Continue reading

Butler for Qlik Sense – connecting Sense to the world

QlikLogoThe goal of the previously described Slack proxy project was to allow posting of messages from Qlik Sense load scripts to Slack (great instant messaging platform!). The project has however developed into something larger and slightly more ambitious, integrating other connections than Slack. The original project name “Slack proxy” has become less and less relevant, and the project has thus been renamed to “Butler”.

As of right now (May 2016), Butler’s feature set include

  • Send messages from Sense load script to Slack
  • Send/publish MQTT messages from Sense load script (i.e. outbound MQTT).
  • Sense reload failures as emails, and to Slack.
  • Sense audit events (session start/stop and connection open/close) to Slack and to MQTT messages.
  • Create new directories on the Sense server’s (where Butler is running) disks.
  • Get disk space info, for disks on the Sense server where Butler is running.

Butler version 1.1 was just released, please refer to the Butler GitHub repository for further information. Feel free to fork it and contribute if you feel some feature is missing –  the node.js app should be pretty easy to understand and extend upon.

ESP8266 Over The Air updating – what are the options?

1950's TV link dish - a bit overkill for ESP8266, maybe..Over the air convenience

A real pain point when developing sensor nodes that are scattered around a building (or a country or the world!) is the updating part. How do you get new firmware onto the devices?

The cell phone manufacturers were early to investigate the options, but from my days in that industry I know first-hand how challenging it was to send new firmware to tens or hundreds of thousands of devices. Lot’s have happened since, today a system update for tens of millions of Android or IOS devices is no big deal, you just make sure your phone is plugged into power before starting the upgrade. Leave it overnight, and next morning you have the latest (and hopefully greatest) version installed.

Looking specifically at the ESP8266, it does have drawbacks in terms of power consumption (it’s hard to get the power consumption down for a device that needs 3-4 seconds to wake up from sleep, connecting to a wifi network and send its sensor readings – but improvements are possible). On the other hand, its wifi connectivity means Over The Air (OTA) updating is possible.

Having a bunch of ESP8266’s lying around, I looked into the options for doing OTA on ESP8266.

Continue reading

Microscope software for OSX

andonstar USB microscopeGot myself an Andonstar 2 megapixel USB microscope about a year ago, mainly just for testing it out, and having it readily available when some repair project demanded it. The microscope is actually pretty nice given its ca USD 60 price point, with decent picture quality, a sturdy stand (even though it’s somewhat of a pain adjusting it) and a built-in adjustable light source.

To be honest it hasn’t been used much (it did come in handy during the repair of the BK Precision LCR meter though).. but this is in part due to the lack of good OS X software for use with generic USB microscopes. Over the past year I have tried some video software, and still have a few on the try-these-out-list.

Continue reading

URL encoding in Qlik Sense and QlikView

URLWith so much cool data available online, tools like QlikView and Sense becomes real Swiss Army knives of data integration.

Pull in some data from company internal databases, some data from previously created QVDs, and more and more commonly also from various online sources, both public and private. Depending on how you call those online APIs you might get away with just sending in query parameters as they are, but in other cases – and this is especially true if you need to send more complex text strings to the API – you need to URL encode the query parameters.

Over the years I have run into this numerous times, but the other weekend I realised it’s actually very easy. Just create a mapping table using an online source for the utf8-to-URL encoded mapping, then use MapSubstring to convert each character in the URL parameter to its hex counterpart.

Code for this found over at GitHub, it’s just 10 or so lines of code.

Occam’s razor holds true again – the easy solutions prevail and are usually preferred!

 

Posting to Slack – and more – from Qlik Sense load scripts (part 3)

Quick update to the previous post on this topic.

There is a new version of slack_proxy available on GitHub. It adds a new endpoint for creating directories on the server where slack_proxy is running.

This obviously means that the apps name – “slack_proxy” – is not really that relevant or correct any longer… And with more end points considered, the name will become even less correct…. Oh well – I’ll keep the name for now – maybe it will change sometime up ahead.

The endpoints currently supported are:

  • /slack for posting to Slack
  • /createDir for creating directories on local disk

Other endpoints on the radar are for sending tweets, sending messages to Pushover, and controlling Blink(1) USB lights.

There is also now reasonably complete documentation on GitHub.