Monitoring home networks with NEMS Part 2: Monitoring Server Processes
This is Part 2 of a 2-part post on using NEMS to monitor servers on a home network. You can find Part 1 here.
In Part 1 of this series of posts, I went through the process of setting up NEMS to monitor servers on a home network. I demonstrated setting up some basic monitoring intended to check that the servers were up and responding on some standard ports. One thing that we were not yet able to do after this initial setup was to monitor specific processes on a Linux server to make sure that they were currently running. In my case, I have an Ethereum mining server and I’d like NEMS to monitor that the mining process is up and running. To do this, we’ll need to install some additional software on our server and then set up NEMS to monitor the process via that software.
The first thing that we need to do is to install the Nagios Remote Plugin Executor (NRPE) on the server that we wish to monitor. Doing so is rather straightforward and is already detailed in the NEMS documentation here. First we install NRPE. Via ssh, login to your server that you wish to monitor and execute the following to download and install NRPE from github.
wget -O - https://raw.githubusercontent.com/Cat5TV/nems-admin/master/build/047-nrpe | bash
Next, we need to edit the NRPE configuration file to allow our NEMS server to communicate with the NRPE daemon. Using your preferred text editor, open the /usr/local/nagios/etc/nrpe.cfg file. Look for the line that reads: allowed_hosts=127.0.0.1,::1
and add a comma followed by the private IP address of the NEMS server on your network, e.g.
allowed_hosts=127.0.0.1,::1,192.168.1.227
When done, save and close the file. Then restart NRPE:
systemctl restart nrpe
That is all that needs to be done on the monitored server so go ahead and logout of your ssh session. Now we can configure the new service in NEMS to monitor a process. In your web browser, navigate back to nems.local and on the main screen, click the “Configuration” tab and select “NEMS Configurator (NConf)” to reach the configuration page. Look for “Hosts” on the left-hand side menu and click on “Show” just to the right.
Next, select the machine that you set up in Part 1 of this post. Your screen will look a bit different dependent upon what you set up. In my case, I am going to select my “miner” host. To the right click on the cog wheel symbol:
On the next screen, you will need to select the service to add from the drop-down menu. In our case we want to select “check_nrpe (Linux Monitoring)”.
Then click the “add” button. Your new service should now appear directly below the drop-down menu. Select the pencil icon to edit the new service.
There are a number of options we’ll need to set here. Some of these are my own preferences. Feel free to set them to suit your particular needs.
- For service name you can leave the name as the default if you wish, but I changed mine to the more descriptive “check-ethminer”.
- For max check attempts I chose 5.
- For check inteval I chose 1.
- For retry interval I chose 1.
- For first notification delay I chose 0.
- For notification intercal I chose 30.
- For notification options I chose w,u,c,r,f,s.
- You can leave everything else as the default or blank.
You can find the definitions of all of these options here.
Finally, the most important step is to provide the command used for checking in the “params for check command box” at the bottom. In my case, I entered “check_procs -a -c 1:1 -C ethminer” (without the quotes). We start with check_procs, which is the name of the command. The -c argument allows us to set the parameter range for critical messages. In my case I use 1:1, indicating that if the number of processes called “ethminer” is something other than 1, I will get a critical message. The -C flag takes the exact name of the process you wish to monitor. In my case I am monitoring my Ethereum mining server, which has the process name “ethminer”. You could also use the -w argument, which lets you specify a parameter range for warning messages. Since my mining process is either running or its not, I don’t need a warning message.
Then hit Submit. The final thing that we need to do is to generate and deploy a new Nagios configuration file with our new setting. On the main NConf page, select the “Generate Nagios config” link near the top left:
If you have done everything correctly until this point, you should see something very similar to the following, with no error messages:
If so, go ahead and click that Deploy button. That’s it, we have now added our first Linux process check to be monitored by NEMS. If you return to your Tactical Overview reporting screen, you should see something like the following:
Finally, let’s test to see if it is actually working. Go ahead and kill whatever process you were monitoring. In my case I ssh into my machine and kill the “ethminer” process.
After a few minutes, you should get a critical message indicating that the process has gone down. If so, you are done. Restart your process and enjoy your new NEMS monitoring setup.
In this and the previous post we have gone through the process of setting up NEMS for home networking. We started with installing the NEMS image on a Raspberry Pi and then configured NEMS to monitor a number of hosts and processes. However, we have only scratched the surface of what NEMS can do. I encourage you to use this as a springboard to explore further and add additional hosts and processes.