In a large Qlik Sense installations you are likely faced with a high number of reloads every hour.

But some apps are more important than others – those apps should reload immediately when triggered.
The alternative is that those important apps end up being queued after ongoing long running tasks – not good.

In the current Sense version (3.1.5) it is not possible to set priorities for app reload tasks. But we can achieve the same thing using security rules – let’s take a look at how.

The initial setup

Let us assume we have a Sense environment consisting of two Sense servers, using the shared persistence concept. Let us also assume there is some kind of shared storage for shared persistence storage, QVDs etc.

Servers are referenced to by using their roles in the Sense cluster, i.e. CENTRAL is the central node, DEV/RELOAD is the development/reload node, etc.

Finally, CENTRAL is the scheduling master and DEV/RELOAD is the only scheduling slave (i.e. the server where reloads execute).

Competing for resources

In this scenario DEV/RELOAD handles both development work and scheduled reloads. App development will thus compete with scheduled reloads for server resources.

This may be fine – but it may also result in app developers experiencing poor performance when scheduled app reloads are running on the server. Ask any developer, and they will tell you how frustrating it is waiting for slowly reloading Sense apps…

Let’s make some assumptions:

• We are using Sense Enterprise 3.1 SR3 or higher, configured for shared persistence.
• DEV/RELOAD has 5 reload slots (configurable in the QMC).
• All reload slots are busy with long running reload tasks (tasks may take hours to complete).
• There are several other reload tasks queued behind the currently running reloads.

At this point an important app reload task triggers. Important data is now available in a source system and should be extracted into QVDs on the Sense server ASAP.

Given the current situation, it might however be hours before the important app is reloaded – not good.

The solution: priority reloads

The best solution would be if Sense provided a way to assign priorities to tasks. Both a high/low option for each task, or a priority level of 1 to 3 would for example work well.

Lacking this, we must take another approach. The solution is easy though – add another node to the Sense cluster – call it PRIO – and the cluster now looks like this:

The new node should be a scheduling slave just like DEV/RELOAD.

Now, is there a way we can tell Sense to use the PRIO node for reloading only the high priority apps?
Of course there is…

Please note that while the solution described below does work well in my tests, security rules can be tricky.
Your existing rules may very well conflict with the ones suggested below – your milage may thus vary. And do not test this on a live production system – you can really mess things up by creating “bad” security rules..

Custom properties

Custom properties are very useful when it comes to tweaking Sense into doing new things. They can control local admin rights within Sense streams, define what Sense apps should be used as template apps and much more.
In our current scenario we define two new custom properties:

• “ReloadWhere” is available on apps and has a single possible value of “PrioNode”
• “NodeType” is available on nodes and has a single possible value of “PriorityReloads”

Security rules

Next we need to add the security rules that will make use of these custom properties.
Starting with Sense 3.1 SR3 (?) there is a section in the QMC called “Load balancing rules”. This feature is well documented and does exactly what it sounds like – determining on what server apps open.

But the concept also works for defining priority reloads. Priority reloads is really just load balancing applied to certain apps.

We need to create one new rule and adjust another. Let’s start with the new rule:

The most interesting parts here are the actions:

1. The CENTRAL node must never reload priority apps
2. The rule should only be used on nodes that has NodeType = PriorityReloads
3. Only apps having ReloadWhere = PrioNode should be considered

Bottom line is that if we set the “ReloadWhere” custom property of an app to “PrioNode”, that app will reload on any of the nodes identified by their “NodeType” custom property being set to “PriorityReloads”. That’s right – you can if needed use this concept to balance priority reloads between several nodes defined as PrioNodes.

Almost at least… there is one more thing we need to take care of.

Changing pre-installed rule

There is a security rule called “ResourcesOnNonCentralReloadNodes” that is created as part of the standard Sense installation. It basically ensures that all apps are available on all rim nodes.

The default version of the rule also prevents the the standard operations monitoring apps from being accessible on rim nodes. In my case I do want the ops monitor to be available everywhere, so I have removed that part of the rule.

So, we need to change the default rule a bit.
Specifically, apps that should be reloaded with priority should not be available on all rim nodes.
If they were, they would also be scheduled for execution on any rim node acting as a scheduling slave, with an associated risk of ending up queued behind long running tasks.

Conclusion

I have tried the concept described above and it works well also in practice.

One thing to keep in mind when adding more reload servers is that QVDs should probably be stored on a shared disk, rather than on for example DEV/RELOAD’s local disk. But if you set that up, you are then free to use the priority reload concept above to control which app is reloaded on what server.