Feature request: Built-in high CPU watchdog with Virtualmin-aware diagnostic email report

Hi Virtualmin team (long time no see) hope you are all enjoying summer :sun_with_face:,

I would like to suggest a feature for Virtualmin/Webmin: a lightweight built-in high CPU watchdog that can send an email alert only after sustained high CPU usage, including a useful diagnostic report.

Background

On a Virtualmin server with multiple domains and separate PHP-FPM pools, I recently had a situation where the server CPU/load spiked heavily. The issue was caused by bot/scanner traffic hitting one virtual host, which then passed many requests into WordPress/PHP-FPM.

The existing monitoring options are useful, but I could not find a simple built-in feature that sends a detailed “what is happening right now” report when CPU is high for more than a few minutes.

Requested feature

A Virtualmin/Webmin module or option that can:

  • Monitor total CPU usage every minute.

  • Trigger only after CPU has been above a configurable threshold for a configurable duration, for example:

    • CPU above 95%

    • for 5 consecutive checks

  • Send an email alert to one or more recipients.

  • Apply a cooldown period to avoid repeated alerts, for example one alert every 30 minutes.

  • Include Virtualmin-aware diagnostics in the alert email.

Suggested alert report contents

The email report could include:

  • Current uptime/load average.

  • Top CPU-consuming processes.

  • Active PHP-FPM pools and worker counts.

  • Mapping between PHP-FPM pool names and Virtualmin domains.

  • Relevant PHP-FPM pool settings:

    • pm.max_children

    • pm.start_servers

    • pm.min_spare_servers

    • pm.max_spare_servers

    • memory_limit

    • max_execution_time

    • max_input_vars

    • request_slowlog_timeout

    • slowlog

    • request_terminate_timeout

  • Recent PHP-FPM slowlog entries if configured.

  • Recent Virtualmin access/error log summaries for the affected domains:

    • Top requests

    • Top IP addresses

    • Top user agents

    • Top HTTP status/request combinations

    • Scanner-like requests such as .env, .git/config, xmlrpc.php, wp-login.php, suspicious /api/ paths, etc.

  • MariaDB/MySQL processlist if local credentials are available.

  • Recent journalctl entries related to:

    • PHP-FPM

    • Nginx/Apache

    • MariaDB/MySQL

    • OOM kills

    • segfaults

    • service failures

  • Optional check for Nginx default server configuration, for example whether unknown HTTPS hostnames/IP-host requests are falling into the first SSL virtual host.

Why this would help

When a shared Virtualmin server suddenly reaches high CPU, it is often not enough to know that CPU is high. The administrator needs to know immediately:

  • Which domain or PHP-FPM pool is responsible.

  • Whether the cause is bots, WordPress cron, admin-ajax, XML-RPC, a plugin, database queries, backups, AWStats, or something else.

  • Which log files show the cause.

  • Whether the issue is isolated to one virtual server or system-wide.

Having this information in the first alert email would reduce troubleshooting time significantly.

Example of a custom script we created

Below is a simplified and anonymized version of the shell script we created. It runs every minute from cron and sends a diagnostic email only after sustained high CPU usage.

#!/usr/bin/env bash
set -u

ALERT_FROM="server-watchdog@example.com”
ALERT_RECIPIENTS=(
  "admin-alerts@example.com”
  "backup-admin@example.net”
)

HOSTNAME_FQDN="$(hostname -f 2>/dev/null || hostname)”
STATE_DIR="/var/lib/server-watchdog”
STATE_FILE="$STATE_DIR/cpu.state”
LAST_ALERT_FILE="$STATE_DIR/last-alert”
LAST_REPORT_FILE="$STATE_DIR/last-report.txt”
LOCK_FILE="/run/server-cpu-watchdog.lock”

CPU_THRESHOLD=95
REQUIRED_HITS=5
ALERT_COOLDOWN_SECONDS=1800

mkdir -p “$STATE_DIR”

exec 9>”$LOCK_FILE”
flock -n 9 || exit 0

read_cpu() {
    awk '/^cpu / {
        idle=$5
        total=0
        for (i=2; i<=NF; i++) total += $i
        print total, idle
    }' /proc/stat
}

read -r total1 idle1 < <(read_cpu)
sleep 2
read -r total2 idle2 < <(read_cpu)

total_delta=$((total2 - total1))
idle_delta=$((idle2 - idle1))

if [ "$total_delta" -le 0 ]; then
    exit 0
fi

cpu_usage=$(( (100 * (total_delta - idle_delta)) / total_delta ))

old_hits=0
[ -f "$STATE_FILE" ] && old_hits="$(cat "$STATE_FILE" 2>/dev/null || echo 0)”

if [ "$cpu_usage" -ge "$CPU_THRESHOLD" ]; then
    hits=$((old_hits + 1))
else
    hits=0
fi

echo "$hits" > “$STATE_FILE”

if [ "$hits" -lt "$REQUIRED_HITS" ]; then
    exit 0
fi

now="$(date +%s)”
last_alert=0
[ -f "$LAST_ALERT_FILE" ] && last_alert="$(cat "$LAST_ALERT_FILE" 2>/dev/null || echo 0)”

if [ $((now - last_alert)) -lt "$ALERT_COOLDOWN_SECONDS" ]; then
    exit 0
fi

echo "$now" > “$LAST_ALERT_FILE”

REPORT="$(mktemp /tmp/server-cpu-report.XXXXXX)”
trap 'rm -f "$REPORT"’ EXIT

{
    echo "CPU Watchdog Alert”
    echo "Host: $HOSTNAME_FQDN”
    echo "Date: $(date -Is)”
    echo "CPU measured: ${cpu_usage}%”
    echo "Threshold: ${CPU_THRESHOLD}%”
    echo "Required hits: ${REQUIRED_HITS}”
    echo

    echo "===== UPTIME =====“
    uptime
    echo

    echo "===== TOP PROCESSES =====“
    ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head -70
    echo

    echo "===== PHP-FPM POOL COUNTS =====“
    ps -eo cmd | grep "php-fpm: pool" | grep -v grep | awk '{print $3}' | sort | uniq -c | sort -nr
    echo

    echo "===== ACTIVE PHP-FPM PROCESSES =====“
    ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | grep "php-fpm: pool" | grep -v grep | head -100
    echo

    echo "===== PHP-FPM POOL CONFIG MAPPING =====“
    pools="$(ps -eo cmd | grep "php-fpm: pool" | grep -v grep | awk '{print $3}' | sort | uniq -c | sort -nr | awk '{print $2}' | head -12)”

    for pool in $pools; do
        echo "--- Pool: $pool —"

        conf="$(find /etc/php -type f -path "*/fpm/pool.d/${pool}.conf" 2>/dev/null | head -1)”

        if [ -n "$conf" ]; then
            echo "Config: $conf”
            grep -E "^(user|group|listen|pm\.max_children|pm\.start_servers|pm\.min_spare_servers|pm\.max_spare_servers|php_value\[upload_tmp_dir\]|php_value\[session.save_path\]|php_value\[error_log\]|php_value\[memory_limit\]|php_value\[max_execution_time\]|php_admin_value\[max_input_time\]|php_admin_value\[max_input_vars\]|slowlog|request_slowlog_timeout|request_terminate_timeout|pm.max_requests)" "$conf" 2>/dev/null
        else
            echo "Config: not found”
        fi

        echo
    done

    echo "===== MARIADB / MYSQL PROCESS =====“
    ps -eo pid,user,%cpu,%mem,etime,cmd --sort=-%cpu | grep -E "mariadbd|mysqld" | grep -v grep || true
    echo

    echo "===== TOP VIRTUALMIN LOG ACTIVITY FOR ACTIVE PHP DOMAINS =====“

    for pool in $pools; do
        conf="$(find /etc/php -type f -path "*/fpm/pool.d/${pool}.conf" 2>/dev/null | head -1)”
        [ -z "$conf" ] && continue

        domain_name="$(grep -E "php_value\[upload_tmp_dir\]|php_value\[session.save_path\]|php_value\[error_log\]" "$conf" 2>/dev/null | sed -n 's#.*= /home/[^/]*/domains/\([^/]*\)/.*#\1#p' | head -1)”
        [ -z "$domain_name" ] && continue

        access_log="/var/log/virtualmin/${domain_name}_access_log”
        error_log="/var/log/virtualmin/${domain_name}_error_log”
        slow_log="$(grep -E "^slowlog[[:space:]]*=" "$conf" 2>/dev/null | awk -F= '{gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2}' | head -1)”

        echo "——————————————————————————————"

        echo "Pool: $pool”
        echo "Domain from pool config: $domain_name”
        echo "Access log: $access_log”
        echo "Error log: $error_log”
        [ -n "$slow_log" ] && echo "Slowlog: $slow_log”
        echo "——————————————————————————————"
 


        echo

        if [ -f "$access_log" ]; then
            echo "Top requests:”
            LC_ALL=C tail -50000 "$access_log" 2>/dev/null \
            | grep -aE "GET|POST” \
            | awk -F\" '{print $2}’ \
            | awk '{print $1, $2}’ \
            | sort | uniq -c | sort -nr | head -50
            echo

            echo "Top user agents:”
            LC_ALL=C tail -50000 "$access_log" 2>/dev/null \
            | awk -F\" '{print $6}’ \
            | sort | uniq -c | sort -nr | head -40
            echo

            echo "Top IPs:”
            LC_ALL=C tail -50000 "$access_log" 2>/dev/null \
            | awk '{print $1}’ \
            | sort | uniq -c | sort -nr | head -40
            echo

            echo "Top status + request:”
            LC_ALL=C tail -50000 "$access_log" 2>/dev/null \
            | awk -F\" '{split($3,s," "); print s[1], $2}’ \
            | sort | uniq -c | sort -nr | head -50
            echo

            echo "Scanner-like recent lines:”
            LC_ALL=C tail -50000 "$access_log" 2>/dev/null \
            | grep -aE "/api/|/swagger|/storybook|/teamcity|/supabase|/system|/stub_status|wp-admin/install.php|\.env|\.git/config|xmlrpc.php|wp-login.php” \
            | tail -80
            echo
        fi

        if [ -f "$error_log" ]; then
            echo "Recent error log:”
            tail -100 "$error_log" 2>/dev/null
            echo
        fi

        if [ -n "${slow_log:-}" ] && [ -f "$slow_log" ]; then
            echo "Recent PHP-FPM slowlog:”
            tail -120 "$slow_log" 2>/dev/null
            echo
        fi
    done

    echo "===== NGINX DEFAULT SERVER CHECK =====“
    nginx -T 2>/dev/null | grep -nE "default_server|ssl_reject_handshake|server_name _" | head -50 || true
    echo

    echo "===== RECENT JOURNAL SIGNALS =====“
    journalctl --since "15 minutes ago" --no-pager 2>/dev/null \
    | grep -Ei "oom|killed process|php-fpm|mariadb|mysql|nginx|segfault|out of memory|failed|error” \
    | tail -150 || true
    echo

} > “$REPORT”

cp "$REPORT" “$LAST_REPORT_FILE”

SUBJECT="[CPU ALERT] ${HOSTNAME_FQDN} CPU ${cpu_usage}% for ${REQUIRED_HITS} checks”

if command -v mail >/dev/null 2>&1; then
    mail -r "$ALERT_FROM" -s "$SUBJECT" "${ALERT_RECIPIENTS[@]}" < “$REPORT”
elif command -v mailx >/dev/null 2>&1; then
    mailx -r "$ALERT_FROM" -s "$SUBJECT" "${ALERT_RECIPIENTS[@]}" < “$REPORT”
elif command -v sendmail >/dev/null 2>&1; then
    {
        echo "From: $ALERT_FROM”
        echo "To: $(IFS=, ; echo "${ALERT_RECIPIENTS[*]}”)”
        echo "Subject: $SUBJECT”
        echo
        cat “$REPORT”
    } | sendmail -t
fi

Example cron:

* * * * * root /usr/local/sbin/server-cpu-watchdog.sh >/dev/null 2>&1

Possible implementation idea

This could be added as an advanced option in Webmin/Virtualmin monitoring:

  • CPU threshold

  • Number of consecutive checks

  • Email recipients

  • Cooldown period

  • Include PHP-FPM pool diagnostics

  • Include Virtualmin domain log summaries

  • Include MySQL processlist

  • Include recent service/journal errors

This would be very useful for shared hosting servers where one virtual server, WordPress site, PHP-FPM pool, plugin, bot attack, or scheduled task can suddenly affect the whole machine.

Thanks for considering this feature.

Here’s something we use: https://www.nagios.org/ (free, open-source)

I don’t know how big or busy your Virtualmin server is, but you might get a significant performance improvement by reducing these:

pm = dynamic
pm.max_children = 6
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

At least try it before discarding the idea.
For reference I have 56 domains, mostly running WordPress.

I already have great values for those :wink: thanks for the reply!

What does this do differently?

My current solution also sends a follow-up email when the CPU issue has recovered by itself. I really like that part, because it prevents unnecessary panic. If I receive a high CPU alert but then also get a “CPU recovered” email a few minutes later, I know the server stabilized without me immediately having to log in and investigate.

The script is intentionally simple, small, and lightweight. It does not try to be a full monitoring platform. It only checks sustained high CPU usage, sends a diagnostic alert, and then sends one recovery notification when the CPU has stayed below the threshold again.

I would personally prefer something like this over an overengineered solution, especially for smaller Virtualmin servers where the goal is simply:

  • detect sustained high CPU usage;
  • send a useful diagnostic report;
  • avoid repeated spam alerts;
  • notify when the situation has recovered.

Below is a privacy-safe example of the script and all steps needed to make it work.


Lightweight CPU watchdog with recovery email

This script checks CPU usage once per minute using cron.

It sends:

  1. A CPU alert email when CPU usage is above the configured threshold for several consecutive checks.
  2. A recovery email when CPU usage has gone back below the threshold for several consecutive checks.

The recovery email is useful because it tells the administrator that the issue has resolved itself.


Features

  • Lightweight Bash script.
  • No external monitoring stack required.
  • Uses /proc/stat for CPU measurement.
  • Uses cron for scheduling.
  • Sends email via mail, mailx, or sendmail.
  • Sends alert only after sustained high CPU.
  • Sends one recovery email after the CPU has recovered.
  • Uses a lock file to avoid overlapping runs.
  • Uses state files to remember alert/recovery status.
  • Includes basic diagnostics:
    • uptime/load average;
    • top CPU processes;
    • active PHP-FPM pools;
    • PHP-FPM pool counts;
    • MariaDB/MySQL process if present;
    • recent relevant journal entries.

1. Create the script

Create the file:

nano /usr/local/sbin/server-cpu-watchdog.sh

Paste this script:

#!/usr/bin/env bash
set -u

ALERT_FROM="server-watchdog@example.com"
ALERT_RECIPIENTS=(
  "admin-alerts@example.com"
  "backup-admin@example.net"
)

HOSTNAME_FQDN="$(hostname -f 2>/dev/null || hostname)"

STATE_DIR="/var/lib/server-watchdog"
STATE_FILE="$STATE_DIR/cpu.state"
RECOVERY_STATE_FILE="$STATE_DIR/cpu-recovery.state"
ACTIVE_ALERT_FILE="$STATE_DIR/active-alert"
LAST_ALERT_FILE="$STATE_DIR/last-alert"
LAST_RECOVERY_FILE="$STATE_DIR/last-recovery"
LAST_REPORT_FILE="$STATE_DIR/last-report.txt"
LAST_RECOVERY_REPORT_FILE="$STATE_DIR/last-recovery-report.txt"
LOCK_FILE="/run/server-cpu-watchdog.lock"

CPU_THRESHOLD=95
REQUIRED_HITS=5
RECOVERY_REQUIRED_OK=3
ALERT_COOLDOWN_SECONDS=1800

mkdir -p "$STATE_DIR"

exec 9>"$LOCK_FILE"
flock -n 9 || exit 0

read_cpu() {
    awk '/^cpu / {
        idle=$5
        total=0
        for (i=2; i<=NF; i++) total += $i
        print total, idle
    }' /proc/stat
}

send_mail() {
    local subject="$1"
    local report="$2"

    if command -v mail >/dev/null 2>&1; then
        mail -r "$ALERT_FROM" -s "$subject" "${ALERT_RECIPIENTS[@]}" < "$report"
    elif command -v mailx >/dev/null 2>&1; then
        mailx -r "$ALERT_FROM" -s "$subject" "${ALERT_RECIPIENTS[@]}" < "$report"
    elif command -v sendmail >/dev/null 2>&1; then
        {
            echo "From: $ALERT_FROM"
            echo "To: $(IFS=, ; echo "${ALERT_RECIPIENTS[*]}")"
            echo "Subject: $subject"
            echo
            cat "$report"
        } | sendmail -t
    else
        logger -t server-cpu-watchdog "No mail, mailx, or sendmail found. Report not sent."
    fi
}

build_alert_report() {
    local report="$1"
    local cpu_usage="$2"

    {
        echo "CPU Watchdog Alert"
        echo "Host: $HOSTNAME_FQDN"
        echo "Date: $(date -Is)"
        echo "CPU measured: ${cpu_usage}%"
        echo "Threshold: ${CPU_THRESHOLD}%"
        echo "Required alert hits: ${REQUIRED_HITS}"
        echo "Required recovery checks: ${RECOVERY_REQUIRED_OK}"
        echo "Cooldown seconds: ${ALERT_COOLDOWN_SECONDS}"
        echo

        echo "===== UPTIME ====="
        uptime
        echo

        echo "===== TOP PROCESSES ====="
        ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head -70
        echo

        echo "===== PHP-FPM POOL COUNTS ====="
        ps -eo cmd | grep "php-fpm: pool" | grep -v grep | awk '{print $3}' | sort | uniq -c | sort -nr
        echo

        echo "===== ACTIVE PHP-FPM PROCESSES ====="
        ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | grep "php-fpm: pool" | grep -v grep | head -100
        echo

        echo "===== PHP-FPM POOL CONFIG MAPPING ====="
        pools="$(ps -eo cmd | grep "php-fpm: pool" | grep -v grep | awk '{print $3}' | sort | uniq -c | sort -nr | awk '{print $2}' | head -12)"

        if [ -z "$pools" ]; then
            echo "No active PHP-FPM pools found."
        fi

        for pool in $pools; do
            echo "--- Pool: $pool ---"
            conf="$(find /etc/php -type f -path "*/fpm/pool.d/${pool}.conf" 2>/dev/null | head -1)"

            if [ -n "$conf" ]; then
                echo "Config: $conf"
                grep -E "^(user|group|listen|pm\.max_children|pm\.start_servers|pm\.min_spare_servers|pm\.max_spare_servers|php_value\[upload_tmp_dir\]|php_value\[session.save_path\]|php_value\[error_log\]|php_value\[memory_limit\]|php_value\[max_execution_time\]|php_value\[upload_max_filesize\]|php_value\[post_max_size\]|php_admin_value\[max_input_time\]|php_admin_value\[max_input_vars\]|slowlog|request_slowlog_timeout|request_terminate_timeout|pm.max_requests)" "$conf" 2>/dev/null
            else
                echo "Config: not found"
            fi

            echo
        done

        echo "===== MYSQL / MARIADB PROCESS ====="
        ps -eo pid,user,%cpu,%mem,etime,cmd --sort=-%cpu | grep -E "mariadbd|mysqld" | grep -v grep || true
        echo

        echo "===== RECENT JOURNAL SIGNALS ====="
        journalctl --since "15 minutes ago" --no-pager 2>/dev/null \
        | grep -Ei "oom|killed process|php-fpm|mariadb|mysql|nginx|apache|segfault|out of memory|failed|error" \
        | tail -150 || true
        echo

    } > "$report"
}

build_recovery_report() {
    local report="$1"
    local cpu_usage="$2"

    {
        echo "CPU Watchdog Recovery"
        echo "Host: $HOSTNAME_FQDN"
        echo "Date: $(date -Is)"
        echo "CPU measured now: ${cpu_usage}%"
        echo "Threshold: ${CPU_THRESHOLD}%"
        echo "Recovery condition: CPU below threshold for ${RECOVERY_REQUIRED_OK} checks"
        echo

        echo "===== ORIGINAL ACTIVE ALERT ====="
        if [ -f "$ACTIVE_ALERT_FILE" ]; then
            cat "$ACTIVE_ALERT_FILE"
        else
            echo "No active alert file found."
        fi
        echo

        echo "===== CURRENT UPTIME ====="
        uptime
        echo

        echo "===== CURRENT TOP PROCESSES ====="
        ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head -40
        echo

        echo "===== CURRENT PHP-FPM POOL COUNTS ====="
        ps -eo cmd | grep "php-fpm: pool" | grep -v grep | awk '{print $3}' | sort | uniq -c | sort -nr
        echo

        echo "===== CURRENT ACTIVE PHP-FPM PROCESSES ====="
        ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | grep "php-fpm: pool" | grep -v grep | head -60
        echo

        echo "===== RECENT JOURNAL SIGNALS ====="
        journalctl --since "15 minutes ago" --no-pager 2>/dev/null \
        | grep -Ei "oom|killed process|php-fpm|mariadb|mysql|nginx|apache|segfault|out of memory|failed|error" \
        | tail -100 || true
        echo

    } > "$report"
}

read -r total1 idle1 < <(read_cpu)
sleep 2
read -r total2 idle2 < <(read_cpu)

total_delta=$((total2 - total1))
idle_delta=$((idle2 - idle1))

if [ "$total_delta" -le 0 ]; then
    exit 0
fi

cpu_usage=$(( (100 * (total_delta - idle_delta)) / total_delta ))

old_hits=0
[ -f "$STATE_FILE" ] && old_hits="$(cat "$STATE_FILE" 2>/dev/null || echo 0)"

old_recovery_hits=0
[ -f "$RECOVERY_STATE_FILE" ] && old_recovery_hits="$(cat "$RECOVERY_STATE_FILE" 2>/dev/null || echo 0)"

if [ "$cpu_usage" -ge "$CPU_THRESHOLD" ]; then
    hits=$((old_hits + 1))
    echo "$hits" > "$STATE_FILE"
    echo 0 > "$RECOVERY_STATE_FILE"

    if [ "$hits" -lt "$REQUIRED_HITS" ]; then
        exit 0
    fi

    if [ -f "$ACTIVE_ALERT_FILE" ]; then
        exit 0
    fi

    now="$(date +%s)"
    last_alert=0
    [ -f "$LAST_ALERT_FILE" ] && last_alert="$(cat "$LAST_ALERT_FILE" 2>/dev/null || echo 0)"

    if [ $((now - last_alert)) -lt "$ALERT_COOLDOWN_SECONDS" ]; then
        exit 0
    fi

    echo "$now" > "$LAST_ALERT_FILE"

    REPORT="$(mktemp /tmp/server-cpu-alert-report.XXXXXX)"
    trap 'rm -f "$REPORT"' EXIT

    build_alert_report "$REPORT" "$cpu_usage"
    cp "$REPORT" "$LAST_REPORT_FILE"

    {
        echo "Alert started: $(date -Is)"
        echo "CPU at alert: ${cpu_usage}%"
        echo "Threshold: ${CPU_THRESHOLD}%"
        echo "Required hits: ${REQUIRED_HITS}"
        echo "Host: $HOSTNAME_FQDN"
        echo "Subject: [CPU ALERT] ${HOSTNAME_FQDN} CPU ${cpu_usage}% for ${REQUIRED_HITS} checks"
    } > "$ACTIVE_ALERT_FILE"

    SUBJECT="[CPU ALERT] ${HOSTNAME_FQDN} CPU ${cpu_usage}% for ${REQUIRED_HITS} checks"
    send_mail "$SUBJECT" "$REPORT"

    exit 0
fi

echo 0 > "$STATE_FILE"

if [ -f "$ACTIVE_ALERT_FILE" ]; then
    recovery_hits=$((old_recovery_hits + 1))
    echo "$recovery_hits" > "$RECOVERY_STATE_FILE"

    if [ "$recovery_hits" -lt "$RECOVERY_REQUIRED_OK" ]; then
        exit 0
    fi

    now="$(date +%s)"
    echo "$now" > "$LAST_RECOVERY_FILE"

    REPORT="$(mktemp /tmp/server-cpu-recovery-report.XXXXXX)"
    trap 'rm -f "$REPORT"' EXIT

    build_recovery_report "$REPORT" "$cpu_usage"
    cp "$REPORT" "$LAST_RECOVERY_REPORT_FILE"

    SUBJECT="[CPU RECOVERED] ${HOSTNAME_FQDN} CPU back below ${CPU_THRESHOLD}%"
    send_mail "$SUBJECT" "$REPORT"

    rm -f "$ACTIVE_ALERT_FILE"
    rm -f "$RECOVERY_STATE_FILE"

    exit 0
fi

echo 0 > "$RECOVERY_STATE_FILE"
exit 0

2. Make the script executable

chmod 750 /usr/local/sbin/server-cpu-watchdog.sh

3. Create the cron job

Create:

nano /etc/cron.d/server-cpu-watchdog

Add:

* * * * * root /usr/local/sbin/server-cpu-watchdog.sh >/dev/null 2>&1

Set permissions:

chmod 644 /etc/cron.d/server-cpu-watchdog

4. Verify the configuration

grep -n "ALERT_RECIPIENTS\|CPU_THRESHOLD\|REQUIRED_HITS\|RECOVERY_REQUIRED_OK" /usr/local/sbin/server-cpu-watchdog.sh
cat /etc/cron.d/server-cpu-watchdog

Expected values:

CPU_THRESHOLD=95
REQUIRED_HITS=5
RECOVERY_REQUIRED_OK=3

5. Run a normal manual test

/usr/local/sbin/server-cpu-watchdog.sh
echo $?

If CPU is normal, this should not send an alert.


6. Force a full alert and recovery test

This test temporarily changes the threshold so that an alert is forced, then changes the threshold again so that a recovery email is forced.

cp /usr/local/sbin/server-cpu-watchdog.sh /usr/local/sbin/server-cpu-watchdog.sh.bak.test

rm -f /var/lib/server-watchdog/cpu.state
rm -f /var/lib/server-watchdog/cpu-recovery.state
rm -f /var/lib/server-watchdog/active-alert
rm -f /var/lib/server-watchdog/last-alert
rm -f /var/lib/server-watchdog/last-recovery

sed -i 's/^CPU_THRESHOLD=.*/CPU_THRESHOLD=0/' /usr/local/sbin/server-cpu-watchdog.sh
sed -i 's/^REQUIRED_HITS=.*/REQUIRED_HITS=1/' /usr/local/sbin/server-cpu-watchdog.sh
sed -i 's/^RECOVERY_REQUIRED_OK=.*/RECOVERY_REQUIRED_OK=1/' /usr/local/sbin/server-cpu-watchdog.sh

/usr/local/sbin/server-cpu-watchdog.sh

sed -i 's/^CPU_THRESHOLD=.*/CPU_THRESHOLD=100/' /usr/local/sbin/server-cpu-watchdog.sh

/usr/local/sbin/server-cpu-watchdog.sh

mv /usr/local/sbin/server-cpu-watchdog.sh.bak.test /usr/local/sbin/server-cpu-watchdog.sh
chmod 750 /usr/local/sbin/server-cpu-watchdog.sh

rm -f /var/lib/server-watchdog/cpu.state
rm -f /var/lib/server-watchdog/cpu-recovery.state
rm -f /var/lib/server-watchdog/active-alert
rm -f /var/lib/server-watchdog/last-alert
rm -f /var/lib/server-watchdog/last-recovery

After this test, two emails should be received:

[CPU ALERT] hostname CPU ... for ... checks
[CPU RECOVERED] hostname CPU back below 95%

7. Confirm production settings are restored

grep -n "CPU_THRESHOLD\|REQUIRED_HITS\|RECOVERY_REQUIRED_OK" /usr/local/sbin/server-cpu-watchdog.sh
ls -lah /var/lib/server-watchdog/

Expected:

CPU_THRESHOLD=95
REQUIRED_HITS=5
RECOVERY_REQUIRED_OK=3

Notes

The script is deliberately simple. It is not meant to replace a complete monitoring platform. It is meant to solve one practical problem:

“When my server is at high CPU for several minutes, send me a useful report — and also tell me if the server recovered by itself.”

That recovery email is the part I find especially useful. It prevents me from assuming the worst when the issue has already passed.

This already exists in webmin.

In the “System and Server Status” module, you can create a “load average” monitor

We test when the load is for 1 minute average over 1,5 cpu’s and it fails after 1 failure, we then run the command:

cd /root/httpddump && wget http://localhost:7080/server-status && /bin/systemctl restart httpd.service

This means that on a server with 20 cpu’s, the load is over 30 in 1 minute average, then a status is saved and httpd is restarted. In a server-status dump there are the ip addresses and the websites. You can then investigate if a website was scanned/attacked/just busy/ or whatever.

It doesn’t have to be this command, you can run your own script, or whatever command you want.
It doesn’t have to be at 1 failure or load 1 minute, you can set 5 failures when load is above x at load 5 minute load average.

We find the “System and Server Status” module a very powerful monitoring tool and use it for a lot.

Not only to check the load, but also to check things like disk space and inodes in /tmp and /backup. /tmp is a seperate partition mounted non-exec on our servers. It hardly ever goes wrong, but a website that is scanned/attacked and doesn’t clean sessions properly or always creates a 0 bytes session file can very quickly use up millions of inodes.

We also have a “Execute Command” monitor that tests “find /var/spool/cron -type f -mmin -10 -ls” on /var/spool/cron. This way we always know immediately a cron was added by a hacker. They do that to recreate there hack/backdoor when you find it and delete it.

regards
Jan

Hi Jan (Dat sounds very much Dutch :netherlands: to my ears haha),

Thanks for the detailed explanation. That is very useful.

I see your point, and I agree that the “System and Server Status” module is more powerful than I initially realized. I had mainly looked at it as a basic status/alerting tool, but your examples with load average monitoring, server-status dumps, inode checks, and cron change detection make it clear that it can be used much more flexibly.

One thing I especially like in my current script is that it also sends a recovery email when the CPU/load has returned to normal. That helps me avoid going into panic mode when the issue has already resolved itself by the time I see the first alert.

So my question is: can Webmin’s System and Server Status module also send a follow-up notification when a monitor returns to OK/recovered state? Or does it only alert when the check fails?

For my use case, the recovery notification is actually quite important. I want to know both:

  • when the server has been under high load for several minutes;
  • and when the server has recovered by itself.

That said, I really appreciate your reply. Your examples gave me a few good ideas, especially saving diagnostic output at the moment of failure and monitoring cron changes as an early warning sign for compromise.

Regards,
WF

The ConfigServer Security & Firewall will also send alerts.

From the dutch speaking part of the world, yes.

You can choose from the following in the setup

| Send report when |
When a service changes status
When a service goes down
As long as the service is up
Any time service is down
Never send report

If you choose “When a service changes status” i presume you will get 2 mails: 1 when it goes down and 1 when it comes backup up.

We set it to “Any time service is down”. I want to know when a problem occurs, not when there is/was no problem (anymore), a mail when it is solved would be clutter. And i always know when it was resolved: when the down mails stop.

But is it resolved then? A hacked website stays hacked, an out-of-date wordpress stays the focus of scanners, etc… And it doesn’t have to be something illegal, if someone runs a facebook campagne, that can cause extra load as well.

If it happens at night and, lets say, it failed an hour long, then we get 12 mails, it restarted 12 times and when the mails stopped, it was ok again. A long time, but on the plus side: we have 12 server-statuses to examine. If it was the same website, maybe its not up to date or needs cloudflare. If it was the same ip address on different websites, the range goes into the firewall.

If it happens in the day you see the mails immediately and you can block the offending ip address or limit the website immediately.

After owning and running a hosting company for more then 27 years, my “panic mode” was disabled a long time ago.

After that many years, you have seen it all and by then you have learned the definition of a true emergency: When there is no more beer i the fridge.

Regards
Jan