Qlik Sense REST connector log file overload

I recently investigated a strange behaviour of the Qlik Sense REST connector, where it would become slower and slower executing REST queries, the longer the server had been running. RAM or CPU consumption was not unusually high, but something was clearly eating away some kind of resources from the system. Turned out it was disk access that was the bottleneck.

EDIT: The issue described here is a confirmed bug relating to the REST connector

The solution suggested in this post is sound and does improve things as suggested, but there is a deeper issue too. Looks like each call to the REST connector indeed eat up some resources in the Sense engine process. One small step at a time, the engine becomes slower and slower..

The effect is really slow, but after 40k-50k calls to the REST connector each call takes 8-10 times longer than it did when the engine process was started.

The only known workaround at this time is to restart the engine service periodically… which of course is less than ideal.

Disk access also takes time

It was really quite strange – why would a call via the REST connector take 3 seconds right after a server restart, and several minutes after the server had been up and running for a few weeks?

Some digging around revealed the cause: The connector seems to create a single log file each time it is invoked. Calling it thousands of times per day during the past months had caused a significant number of log files to accumulate in C:\ProgramData\Qlik\Custom Data\QvRestConnector\Log, which is where those log files live.

It kind of makes sense – each time the connector were to write to a log file, it would have to deal with hundreds of thousands of files in the log directory – while Windows Server can handle this, it is bound to take time.

Setting a monthly calendar reminder to clean that folder seemed like a recipe for further troubles, instead a tiny batch file, scheduled using the standard Windows task scheduler, solved the problem:

forfiles -p “C:\ProgramData\Qlik\Custom Data\QvRestConnector\Log” -s -m *.log -d -2 -c “cmd /c del @path”

Schedule this to run daily, and you will never have more than 2 days worth of log files in that directory (that command should be on a single line in the .bat file, btw).

Nice and easy!