The lazy Qlik developer #2: Setting up Butler SOS
Butler SOS provides real-time monitoring for client-managed Qlik Sense. Key metrics for both Sense and Windows are monitored together with errors and warnings from the Sense logs. In this article you find a complete recipe for how to set up Butler SOS.
The single most challenging aspect of the Butler tools - Butler SOS included - seems to be getting started.
In this post we'll therefore look at how to set up and configure a complete monitoring solution for client-managed Qlik Sense:
- Butler SOS collects metrics, logs and events from Sense.
- Data is stored in InfluxDB.
- Grafana is used to visualise the data.
The three tools above will all run in Docker containers. They can also run as native Windows applications, that however involves installing each of those tools on your computer or a server. By using Docker we can test Butler SOS and then delete the Docker containers when we are done.
If you then want to put Butler SOS into production you can do so using Docker (or Kubernetes) or as native Windows/Linux/macOS applications. The end result is identical in both cases.
Butler SOS is open source and as such free to use.
- More info and documentation at https://butler-sos.ptarmiganlabs.com
- Feedback, questions, bug reports etc are best reported at https://github.com/ptarmiganlabs/butler-sos/discussions
- Windows/macOS/Linux binaries are available at https://github.com/ptarmiganlabs/butler-sos/releases
TODO list
Here are the things we will work on and deploy. Don't worry, we'll go through each of them in detail:
- Customise and deploy XML log appender files on Sense server(s)
- Add Sense certificates
- Edit Butler SOS config file
- Configure and start Butler SOS, InfluxDB and Grafana in Docker containers
- Create a datasource in Grafana
- Import a sample SenseOps dashboard into Grafana
- Generate some events and errors in Sense, verify they show up in Grafana
- Generate some user activity in Sense, verify it shows up in Grafana
Assumptions & general instructions
The demo in this text assumes a few things.
- The log database in Qlik Sense is turned off. Butler SOS can retrieve data from log db if it's present, but as log db was deprecated some time ago we will assume it's not used.
Most of the configuration needed is actually doing find-and-replace in the config files we will look at later in this text:
- Replace all instances of my Sense servers' host names (pro2-win2.lab.ptarmiganlabs.net and pro2-win2.lab.ptarmiganlabs.net) with your ditto. This applies to Butler SOS' config file. In my case I monitor two Sense servers, you can monitor one or more as you like.
- Replace all instances of my laptop's IP address (192.168.1.168) with the IP address of the computer where you run the Docker containers.
Template files & GitHub
The "part 2" directory of The lazy Qlik developer repository on GitHub contains all template files mentioned in this blog post.
Action: Clone that repository to the computer where you want to run Butler SOS, InfluxDB and Grafana.
If you are not comfortable with Git the repository can be downloaded as a zip file.
➜ git clone https://github.com/ptarmiganlabs/the-lazy-qlik-developer.git
Cloning into 'the-lazy-qlik-developer'...
remote: Enumerating objects: 20, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 20 (delta 5), reused 16 (delta 4), pack-reused 0
Receiving objects: 100% (20/20), 4.15 KiB | 4.15 MiB/s, done.
Resolving deltas: 100% (5/5), done.
➜
If downloading the zip file, expand it into suitable location on the server where you want to run Butler SOS, InfluxDB and Grafana.
The repository may over time include many different root level directories, right now we are interested in the one named "part 2-setting up butler sos".
Its strcucture looks like this:
.
├── config
│ └── production.yaml
├── docker-compose.yaml
├── grafana-dashboard.json
└── log_appenders
├── engine
│ └── LocalLogConfig.xml
├── proxy
│ └── LocalLogConfig.xml
├── repository
│ └── LocalLogConfig.xml
└── scheduler
└── LocalLogConfig.xml
Customise and deploy XML log appender files
Qlik Sense uses the Log4Net logging framework for all its logs. This framework is extendable, something we use with Butler SOS.
By adding a "log appender file" called "LocalLogConfig.xml" to select Sense services we can hook into any log events for those services and then forward desired log events to Butler SOS.
In this example we will hook into the logs of the Engine, Proxy, Repository and Scheduler services. There is one LocalLogConfig.xml file for each service.
Each XML file also needs to be customised to your situation. Specifically, you must enter the IP address of the computer where Butler SOS is running. That way Log4Net will know where to send the log messages we are interested in.
Action: Customise each of the four LocalLogConfig.xml files. Replace the 192.168.1.168 address with the address of your Docker host. Template files are available in this GitHub repository, and in the "log_appenders" directory.
Action: Deploy the LocalLogConfig.xml files in the locations below on each Sense server that should be monitored by Butler SOS.
| Sense service | Location of LocalLogConfig.xml |
|---|---|
| Engine | C:\ProgramData\Qlik\Sense\Engine |
| Proxy | C:\ProgramData\Qlik\Sense\Proxy |
| Repository | C:\ProgramData\Qlik\Sense\Repository |
| Scheduler | C:\ProgramData\Qlik\Sense\Scheduler |
Action: Make sure there are no firewalls or similar that block outbound UDP ports 9996 and 9997 on the Sense server(s) and same ports inbound on the computer where the Butler SOS Docker container runs.
When the above actions are done Sense will forward to Butler SOS
- log events (warnings, errors ) from the 4 services.
- user events, which are happening when users log in/out of Sense, or open/close Sense apps.
Add Sense certificates
Butler SOS uses certificates to authenticate with Sense.
These certs are exported from the QMC (link 1, link 2). For testing purposes you can also use the default certificates in "C:\ProgramData\Qlik\Sense\Repository\Exported Certificates.Local Certificates".
If deploying Butler SOS in production you should create new certificates for that purpose.
Action: Place the certificates in the "config/certificate" directory under where the docker-compose.yaml file is stored.
Your file structure now looks like this:
.
├── config
│ ├── certificate
│ │ ├── client.pem
│ │ ├── client_key.pem
│ │ └── root.pem
│ └── production.yaml
├── docker-compose.yaml
├── grafana-dashboard.json
└── log_appenders
├── engine
│ └── LocalLogConfig.xml
├── proxy
│ └── LocalLogConfig.xml
├── repository
│ └── LocalLogConfig.xml
└── scheduler
└── LocalLogConfig.xml
Edit Butler SOS config file
Butler SOS is extremely flexible and can be configured do to many kinds of monitoring, send the metrics to different destinations etc.
All configuration is done via the YAML config file. A sample file production.yaml is available in the GitHub repo. Edit the production.yaml file so it matches your situation.
Action: Save the config file as "production.yaml" in a directory called "config" under the directory where the "docker-compose.yaml" file is stored. See file tree above.
Action: In the config file, replace "192.168.1.168" with the IP address of the computer where Butler SOS is running.
Action: In the config file, replace all instances of pro2-win2.lab.ptarmiganlabs.net and pro2-win2.lab.ptarmiganlabs.net with the host names of your Sense server(s). If you have a single Sense server, just remove the lines for the 2nd server from the config file.
Configure and start Butler SOS, InfluxDB and Grafana in Docker containers
A single docker-compose.yaml file is used to start Butler SOS, InfluxDB and Grafana. It should work as-is.
# docker-compose.yml
version: "3.3"
services:
butler-sos:
image: ptarmiganlabs/butler-sos:9.3.1
container_name: butler-sos
restart: always
volumes:
# Make config file and log files accessible outside of container
- ./config:/nodeapp/config
- ./log:/nodeapp/log
ports:
- 9997:9997/udp # User events from Sense
- 9996:9996/udp # Log events from Sense
environment:
- NODE_ENV=production # Means that Butler SOS will read config data from production.yaml
logging:
driver: "json-file"
options:
max-file: "5"
max-size: "5m"
networks:
- senseops
influxdb:
image: influxdb:1.8.10
container_name: influxdb
restart: always
volumes:
- ./influxdb/data:/var/lib/influxdb # Mount for influxdb data directory
- ./influxdb/config/:/etc/influxdb/ # Mount for influxdb configuration
ports:
# The API for InfluxDB is served on port 8086
- 8086:8086
- 8082:8082
environment:
# Disable usage reporting
- INFLUXDB_REPORTING_DISABLED=true
networks:
- senseops
grafana:
image: grafana/grafana:9.1.0
container_name: grafana
restart: always
ports:
- 3000:3000
volumes:
- ./grafana/data:/var/lib/grafana
environment:
- GF_INSTALL_PLUGINS=grafana-clock-panel
networks:
- senseops
networks:
senseops:
driver: bridge
Action: Start the tools from the command line. Here we start the tools on macOS. Check for errors. If you have followed along above there should be none.
The first lines after starting Butler SOS/InfluxDB/Grafana can look something like below. There will be LOTS of info coming from especially InfluxDB and Grafana!
➜ docker-compose up
[+] Running 30/30
⠿ grafana Pulled
⠿ 9621f1afde84 Pull complete
⠿ 8a979fdf9b56 Pull complete
⠿ 1dfb2bc044fd Pull complete
⠿ 83f5d14e4bf0 Pull complete
⠿ b6745f3b63b1 Pull complete
⠿ c57092a7aaa6 Pull complete
⠿ 94139446967c Pull complete
⠿ 3406d8746525 Pull complete
⠿ 51ac91216bc8 Pull complete
⠿ butler-sos Pulled
⠿ 8740c948ffd4 Already exists
⠿ 29fd6e358874 Already exists
⠿ 687a59b7d3ab Pull complete
⠿ 6824e34d9639 Pull complete
⠿ e5cf09adc7bb Pull complete
⠿ a5ae9ae32c3a Pull complete
⠿ 26d7b373d9a9 Pull complete
⠿ cd641975beb1 Pull complete
⠿ 79b04efc006b Pull complete
⠿ c882717470ce Pull complete
⠿ influxdb Pulled
⠿ bbeef03cda1f Pull complete
⠿ f049f75f014e Pull complete
⠿ 56261d0e6b05 Pull complete
⠿ 9e6095c9511b Pull complete
⠿ e8aae834f50d Pull complete
⠿ f4ebf85949cb Pull complete
⠿ 6ba417bd43bf Pull complete
⠿ da1d9ab0ac81 Pull complete
[+] Running 4/4
⠿ Network part2-settingupbutlersos_senseops Created
⠿ Container butler-sos Created
⠿ Container grafana Created
⠿ Container influxdb Created
Attaching to butler-sos, grafana, influxdb
influxdb | grep: /etc/influxdb/influxdb.conf: No such file or directory
influxdb | ts=2023-03-15T09:26:12.379631Z lvl=info msg="InfluxDB starting" log_id=0ga9J8bl000 version=1.8.10 branch=1.8 commit=688e697c51fd
influxdb | ts=2023-03-15T09:26:12.379669Z lvl=info msg="Go runtime" log_id=0ga9J8bl000 version=go1.13.8 maxprocs=4
influxdb | ts=2023-03-15T09:26:12.481811Z lvl=info msg="Using data dir" log_id=0ga9J8bl000 service=store path=/root/.influxdb/data
influxdb | ts=2023-03-15T09:26:12.482151Z lvl=info msg="Compaction settings" log_id=0ga9J8bl000 service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
...
...
Butler SOS, InfluxDB and Grafana are now running in their own respective Docker containers.
Create a datasource in Grafana
Some basic configuration needs to be done in Grafana.
Action: Change password 0f Grafana's default admin account.
Open http://localhost:3000 > login using admin/admin > Enter new password (or skip) > Grafana main page shows.
Action: Create InfluxDB data source in Grafana.
Click Configuration icon in lower left > Data sources > Add data source > InfluxDB > Name = Butler SOS, URL = http://influxdb:8086, Database = senseops > Save & test > "datasource is working. 11 measurements found".
We now have a working datasource in Grafana, connected to the Sense metrics stored in InfluxDB.
Import a sample SenseOps dashboard into Grafana
There is a sample Grafana dashboard in the GitHub repo (link).
Action: Copy the dashboard's JSON to the clipboard > Click on Dashboards icon on left side in Grafana > Import > Paste JSON into the "Import via panel json" text box > Load > Import > Dashboard show be visible > Click Save in dashboard's top right menu.
If you'd rather upload the "grafana-dashboard.json" file instead of pasting it into Grafana that's fine too, of course.
At this point you should see a Grafana dashboard with some metrics in it. The CPU, free RAM, user sessions and a few other charts will show data.
Generate some warnings in Sense, verify they show up in Grafana
The Grafana dashboard also visualise various aspects on events comging from Sense. This can be warning or error log messages, or users logging in/out or opening/closing apps.
Action: Stop the Qlik Sense scheduler service on the Sense server. This will trigger warnings/errors in the Sense logs, which will be forwarded to Butler SOS/InfluxDB/Grafana.
Wait a while (the dashboard updates every 10 seconds), then something like this should show up:
Scroll down a bit, expanding the "Warnings & Errors" section of the dashbord and we see lots more details about the warnings:
Generate some user activity in Sense, verify it shows up in Grafana
Butler SOS also tracks user events. These are fired when users log in/out and when they open/close Sense apps.
Action: Log into the Sense hub, then open an app.
Wait a few seconds then the above actions should show up in Grafana. Opening the dashboard sections at the bottom will reveal more info about the user events.
Action: Reload the browser page of the opened app a couple of times.
The closing/reopening of the app will trigger "connection closed" and "connection opened" events, which are visualised in the dashboard.
