Monitor OPNsense internet connection

!README.md

Monitor OPNsense internet connection

OPNsense sometimes "loses" the WAN facing connection when that is "reinitialised" (modem reboot/WAN restart).

Monitoring the WAN facing connection and taking corrective action is the solution proposed by this script.

opnsense_ping_check.sh will ping IP addresses to determine if the WAN connection is still working, and try to restore the WAN if needed by bringing the interface down and up, or through a system reboot (OPNsense) if that failed.

So to install this script:

1. Copy it to your OPNsense instance.
2. Update values in the script (network interface name, IPs you want to ping).
3. Execute it once interactively (copies script, and adds action)

To enable monitoring, a cron job needs to be added in OPNsense that calls the action created by opnsense_ping_check.sh.

Using the OPNsense UI, the generated action can be enabled as a cron job. Cron jobs are added under System>Settings>Cron.
Add an entry:

1. Tick Enable;
2. Set minutes to */5 (or other delay - be reasonable);
3. Select "ping_check" as the command;
4. Set a description such as "Ping check and recover connection"
5. Click save.

Screenshot while adding action to the cron jobs:

.pre-commit-config.yaml

---
files: ^(.*\.(py|json|md|sh|yaml|cfg|txt))$
exclude: ^(\.[^/]*cache/.*)$
repos:
  - repo: https://github.com/verhovsky/pyupgrade-docs
    rev: v0.3.0
    hooks:
      - id: pyupgrade-docs
  - repo: https://github.com/executablebooks/mdformat
    # Do this before other tools "fixing" the line endings
    rev: 0.7.17
    hooks:
      - id: mdformat
        name: Format Markdown
        entry: mdformat  # Executable to run, with fixed options
        language: python
        types: [markdown]
        args: [--wrap, '75', --number]
        additional_dependencies:
          - mdformat-toc
          - mdformat-beautysh
          # -mdformat-shfmt
          # -mdformat-tables
          - mdformat-config
          - mdformat-black
          - mdformat-web
          - mdformat-gfm
  - repo: https://github.com/asottile/blacken-docs
    rev: 1.16.0
    hooks:
      - id: blacken-docs
        additional_dependencies: [black==22.6.0]
        stages: [manual]   # Manual because already done by mdformat-black
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      # - id: no-commit-to-branch
      #   args: [--branch, main]
      # - id: check-yaml
      #   args: [--unsafe]
      # - id: debug-statements
      - id: end-of-file-fixer
      - id: trailing-whitespace
        exclude: \.md$
      # - id: check-json
      - id: mixed-line-ending
      # - id: check-builtin-literals
      # - id: check-ast
      - id: check-merge-conflict
      - id: check-executables-have-shebangs
      - id: check-shebang-scripts-are-executable
      # - id: check-docstring-first
      - id: fix-byte-order-marker
      - id: check-case-conflict
      # - id: check-toml

#   - repo: https://github.com/adrienverge/yamllint.git
#     rev: v1.23.0
#     hooks:
#       - id: yamllint
#         args:
#           - --no-warnings
#           - -d
#           - '{extends: relaxed, rules: {line-length: {max: 90}}}'
#   - repo: https://github.com/asottile/pyupgrade
#     rev: v2.31.0
#     hooks:
#       - id: pyupgrade
#         args:
#           - --py37-plus
#   - repo: https://github.com/psf/black
#     rev: 22.1.0
#     hooks:
#       - id: black
#         args:
#           - --safe
#           - --quiet
#           - -l 79
#   - repo: https://github.com/Lucas-C/pre-commit-hooks-bandit
#     rev: v1.0.5
#     hooks:
#       - id: python-bandit-vulnerability-check
#         args: [--skip, 'B101,B311', --recursive, .]
#
#   - repo: https://github.com/fsouza/autoflake8
#     rev: v0.3.1
#     hooks:
#       - id: autoflake8
#         args:
#           - -i
#           - -r
#           - --expand-star-imports
#           - custom_components
#   - repo: https://github.com/PyCQA/flake8
#     rev: 4.0.1
#     hooks:
#       - id: flake8
#         additional_dependencies:
#           - pyproject-flake8==0.0.1a2
#           - flake8-bugbear==22.1.11
#           - flake8-comprehensions==3.8.0
#           - flake8_2020==1.6.1
#           - mccabe==0.6.1
#           - pycodestyle==2.8.0
#           - pyflakes==2.4.0
#   - repo: https://github.com/PyCQA/isort
#     rev: 5.10.1
#     hooks:
#       - id: isort

  - repo: https://github.com/codespell-project/codespell
    rev: v2.2.6
    hooks:
      - id: codespell
        args:
          # - --builtin=clear,rare,informal,usage,code,names,en-GB_to_en-US
          - --builtin=clear,rare,informal,usage,code,names
          - --ignore-words-list=wan
          - --skip="./.*"
          - --quiet-level=2

#   - repo: https://github.com/pre-commit/mirrors-mypy
#     rev: v0.931
#     hooks:
#       - id: mypy
#         args:
#           - --ignore-missing-imports
#           - --install-types
#           - --non-interactive
#           - --check-untyped-defs
#           - --show-error-codes
#           - --show-error-context
#         additional_dependencies:
#           - zigpy==0.43.0

opnsense_ping_check.sh

#!/bin/sh
# Script for OPNsense to monitor WAN facing connection
# - When failing:
#   1. down and up the interface, check again
#   2. When still failing, reboot OPNsense

# The second part of this script will also install
# an action on opnsense and copy the script to a system
# location.
#
# Therefore, to install this script:
#  a. Copy it to your OPNsense instance.
#  b. Execute it once interactively (copies script, and adds action)
#
# Using the OPNsense UI, this action can be enabled as a cron
# job.  Cron jobs are added under System>Settings>Cron.
# Add an entry that you:
#  1. Enable
#  2. Set minutes to "*/5"
#  3. Select "ping_check" as the command
#  4. Set a description such as "Ping check and recover connection"
#  5. Click save.
#
# Based on a script that was likely adapted from
#   http://blog.martinshouse.com/2014/06/pfsense-auto-reboot-if-internet.html
#


# First IP to ping to check if connection is up
IP1=8.8.8.8  # Google DNS Server 1
# Second IP to ping to check if connection is up
IP2=8.8.4.4  # Google DNS Server 2
# Minimum uptime
MIN_UPTIME=120

# Testing uptime to run script only xx seconds after boot
# Current time
curtime=$(date +%s)
# Boottime in seconds
uptime=$(sysctl kern.boottime | awk -F'sec = ' '{print $2}' | awk -F',' '{print $1}')
# Uptime in seconds
uptime=$((curtime - uptime))

# If boot is longer than 120 seconds ago...
if [ $uptime -gt $MIN_UPTIME ]; then

  # A message to the console (If you want feedback)
  # echo "Testing Connection at" `date +%Y-%m-%d.%H:%M:%S` "uptime:" $uptime "seconds" >> file.txt
  # wall file.txt
  # rm file.txt

  # Try 1 or 2 minutes worth of very short pings to the selected servers.
  # Quit immediately if we get a single frame back.
  # If neither server responds at all then reboot the firewall.

  counting=$(ping -o -s 0 -c 10 $IP1 | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')

  if [ "$counting" -eq 0 ]; then

    counting=$(ping -o -s 0 -c 10 $IP2 | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')

    if [ "$counting" -eq 0 ]; then

      # trying to just restart NIC

      ifconfig igb0 down
      ifconfig igb0 up

      counting=$(ping -o -s 0 -c 10 $IP1 | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }' )

      if [ "$counting" -eq 0 ]; then

        # network down
        # Save RRD data

        /usr/local/etc/rc.reboot
      fi
    fi
  fi
fi


# Optional, add opnsense action for this script, which can then be added
# as a cron job in the UI:
TARGET_ACTION=/usr/local/opnsense/service/conf/actions.d/actions_ping_check.conf
TARGET_LOCATION=/usr/local/sbin/ping_check.sh

# Copy this script to target location if needed
if [ ! -r "$TARGET_LOCATION" ] ; then
  cp "$0" "$TARGET_LOCATION"
  chmod +x "$TARGET_LOCATION"
fi

# Add action if needed
if [ ! -r "$TARGET_ACTION" ] ; then
  cat > "$TARGET_ACTION" <<EOACTION
[load]
command:$TARGET_LOCATION
parameters:
type:script
message:starting ping check
description:ping_check
EOACTION
  # Restart configd service to have action appear in the menus
  service configd restart
fi

OPNsense_ping_cron.png

Hello, I have this script running perfectly. Thanks! One question. I sometimes have a ISP outage (ie. 5 hours long) which won't be fixed by a reboot. Will this be rebooting every 2 minutes until it comes back up? If I change the MIN_UPTIME to 3600 then it would only reboot once a hour correct?

The script would reboot at most every 5 minutes because that is the periodicity of the cron job which is higher than the 2 minutes.

Setting MIN_UPTIME to 3600 will limit the reboots to once every hour.

The script would reboot at most every 5 minutes because that is the periodicity of the cron job which is higher than the 2 minutes.

Setting MIN_UPTIME to 3600 will limit the reboots to once every hour.

Perfect! Thanks again.

Does this leave an entry in the logs, or will I get an e-mail if this triggers? Is there a good way to set that up? It'd be nice to know how often this is popping off. By the way, thank you for creating this much needed tool!

@ericbakeragilix There currently isn't anything in the script that logs something.
A logger -t TAG "MESSAGE could be added whenever the reboot is triggered.

A quick does does not reveal any "OPNsense method" to trigger an email notification from a script.
Further, as this script triggers when the network is absent, the mail would need to be added to a local mail queue or sent later from the script.
The script could be extended to append the date it requests a reboot to a file. Then - on each execution - it could check if that file exists. Ifs, try to notify about the reboot through a mail server with the contents of that file and delete it when the notification could be sent. That could be a simple https API call to some webhook that takes care of the rest.

Great script! As of note, one more step to do is to restart the configd service so the cron job will show on the GUI:

service configd restart

@edominicci Thank you for the feedback - I added this to the script.