Creating alerts for system shutdown and boot

sysadmin

I help run a number of servers for friends and family. It's a win-win for them and me. I get practice as a sysadmin, they get more peace of mind. Of course, these servers aren't ones that I solely run -- other people log in often and can do various actions.

After one particularly harried night, I realize that a friend's server was shut down by their VPS provider. Not too surprising for the super cheap price, but it's not nice either way. I'm still working on setting up a monitoring server for all of the servers I have access to, but in the mean time I wanted a solution to tell me when the server was shutdown and when it finished booting.

I love using Pushover for things like this too. Super easy API, and generous free terms -- it's not likely that I'll need more than 7,500 alerts that the servers are booting and shutting down in a month!

Since every server is running systemd, I can take advantage of it and greatly simplify the implementation. To start with, I create a basic shell script at /usr/local/sbin/bootup-pushover-alert.sh:

#!/bin/sh

# This script is a basic 'service' launched and stopped by systemd. This will
# send an alert out via Pushover to let me know that the server is shutting
# down or finished booting.

if [ "$1" = "boot" ]; then
  message="$(hostname -f) has finished booting"
  sound=bugle
elif [ "$1" = "shutdown" ]; then
  message="$(hostname -f) is shutting down"
  sound=falling
fi

curl -s \
  --form-string "token=token123" \
  --form-string "user=user456" \
  --form-string "message=$message" \
  --form-string "sound=$sound" \
  https://api.pushover.net/1/messages.json

I then set up the systemd service unit at /etc/systemd/system/boot-alert-pushover.service:

[Unit]
Description=Send an alert via pushover that the system has finished booting
After=network-online.target

[Service]
# Idle means that systemd will wait until the system is idle before executing
Type=idle
ExecStart=/usr/local/sbin/bootup-pushover-alert.sh boot
ExecStop=/usr/local/sbin/bootup-pushover-alert.sh shutdown
RemainAfterExit=yes
Restart=no

[Install]
WantedBy=multi-user.target

And simply enabling the service: systemctl enable boot-alert-pushover

A few interesting notes about this:

  • This will not work for the virtual equivalent of pulling the power cord. That's where some sort of external monitoring would be useful. Nodeping and others are great at that.
  • Setting the service type to idle means that systemd will delay executing the script until the server has (mostly?) finished booting.
  • For simple shell scripts like this, you must set RemainAfterExit to yes otherwise ExecStop won't be called.
  • Systemd does support calling a series of scripts in a directory when the system shuts down. See systemd-shutdown(8) for details about that. However, in my experience it didn't work.

And what kind of alert system would it be without some sort of image? Here's a very simple one I'm using already based on a couple of icons I found on the Noun Project:

a transparent icon of a server with a yellow sun on top

|