When relying on various Node.js services (e.g. Butler SOS, Butler, App Duplicator etc), you quickly run into the challenge to ensure all services are always up and running.
A failing service might be fine, as long as it is quickly restarted in a predictable way.

A concrete example could be Qlik Sense or QlikView apps that send status messages to Slack during the execution of their reload scripts. Those messages will fail if Butler is for some reason not running.

This leads us to the conclusion that the services must automatically be

a) started when a server is rebooted, and
b) restarted if they for some reason terminate/die.

Enter process monitors.

At their core, process monitors ensure that the desired processes are always running, i.e. bullet b) above. Some process monitors also offer additional features such as zero-downtime restart of services, memory and performance profiling of the monitored services, being able to monitor different kinds of processes (not only Node.js ditto).

Adding to the pain is the fact that Sense and QlikView runs on Windows servers, meaning that all those great tools available on Linux cannot be used.

Looking at Node.js specifially, I have found two process monitors to work well on Windows: Forever and PM2.

Forever

Forever works well and relies on config files and a CLI for configuration and controlling. That works fine, but forever also feels a bit dated. Looking at the Github repository, little work has been done to project during the past several years. This of course is not necessarily a problem, but still leaves me feeling a bit uneasy using it in a production setting. On the other hand you could argue that forever is stable and works – and therefore exactly what is needed in a production environment… Tricky one.

Forever is not limited to handling Node apps, it can monitor pretty much any script or app.
While primarily targeting Linux environments, forever runs reasonably well on Windows too. By applying some manual tricks (google “run forever as windows service”), forever can be turned into a Windows service that autostarts when Windows boots.

Alternatively, you can use Windows Scheduler to create a task that runs at system startup, the task doing nothing but starting forever – which in turn starts and monitors the desired processes and scripts

PM2

PM2 is an open source tool, backed by a company (Keymetrics). Keymetrics offer various add-on services on top of the core PM2 app. The monitoring UI offered by Keymetrics is very nice indeed, but using it also means that you will be sending data to that company… which might not be ok for sensitive production environments. The base, open source PM2 product available on GitHub is however likely to do all you need and then some – I suggest you try it first and decide on additional monitoring later.

A nice thing about PM2 is that its command line interface is easy to use and understand. PM2 also collect quite comprehensive statistics (memory use, CPU load etc) for each monitored process. These metrics can then be easily viewed from a command line prompt.

As an example,  let us start a couple of Node.js apps using PM2:

pm2 start path/to/app1.js --name="App 1" pm2 start path/to/app2.js --name="App 2"

Assuming the apps started ok, you can now use “pm2 list” to view stats about the apps.

Let’s say we always want to start those two apps when the server boots.
Just run “pm2 save” from the command line, and PM2 will create a snapshot of what apps are currently being monitored.

Running “pm2 kill” (the kill command is not listed in the online documentation, but exists in recent PM2 versions) will shut down PM2 as well as the monitored apps.

Finally, running “pm2 resurrect” will start PM2 and also start the processes that were running when the “pm2 save” command was executed.

In short, PM2 does everything we need to monitor and keep our Qlik Sense support services alive and well.

There is only one catch.. PM2 just refuses to auto-start on server boot.
Which of course is a show stopper.

Starting PM2 at Windows server boot

It took me almost a year to find the solution to this problem, but once I did it is extremely easy to implement.

I have not found any documentation about this, but it seems PM2 relies on there being a system variable called “PM2_HOME”. Or rather, PM2 will start just fine in interactive mode (i.e. from a cmd.exe command prompt) even without this variable existing.

But when running the exact same command (“pm2 resurrect”) from an at-startup task in Windows Scheduler, PM2 fails to start properly. In fact it just fails silently, which is extremely annoying…

The solution is simple:

1. Create a c:\pm2 directory (the directory can be called anything and be placed anywhere)
2. Create a system wide environment variable called PM2_HOME, and assign it a value of “c:\pm2” (or whatever directory you created in the first step)
3. Log out and in again to make the new variable take effect.
4. From a command prompt, run “pm2 start path/to/app1” etc for all apps that should be monitored.
5. From that same command prompt, run “pm2 save”. You now have a snapshot file in c:\pm2.
6. Run “pm2 kill” to stop pm2 itself as well as monitored processes, then “pm2 resurrect” to make sure the desired apps are started by pm2.
7. Create a task in Windows Scheduler, set it to trigger on system startup, and simply do a “pm2 resurrect”. Depending on your path settings, you might have to replace “pm2” with the full path to the pm2 executable.
   In addition to trigger on system startup, the task should also trigger (for example) every 5 minutes. This ensures that PM2 itself will be restarted if it dies.
   In other words: maximum down time will be 5 minutes.
8. Reboot. PM2 should then launch after the reboot, and also start the apps that were monitored by PM2 when the “pm2 save” was issued.

Mission accomplished.

Run PM2 as a Windows service

The trick of using Windows scheduler to start and keep PM2 alive works, but is not really very elegant.

A better solution would be to run PM2 as a Windows service.
This might actually not be very hard to achieve – the very nice NSSM package makes it easy to convert almost any app into a Windows service.

I have not yet tested PM2 together with NSSM, but would love to hear from those who have. Anyone?