Script to automatically update hosts file from a filter list

README.md

Minimal script to quickly update hosts file from a filter list.

Note that this does not replace the entirety of the hosts file, so previous customizations will remain intact.

Installation

Download the update_hosts script below or run:

$ curl --proto "=https" --tlsv1.2 https://gist.githubusercontent.com/metametapod/4c4a7a9c888343fc6e722f8ed77aa763/raw/update_hosts -o update_hosts

Run as root:

# chown root:"$(id -gn root)" update_hosts
# chmod 544 update_hosts

Via anacron

Run as root:

# mv update_hosts /etc/cron.daily/

This will run the script once a day, or when waking your computer if powered off.

Via cron

Run as root:

# mv update_hosts /usr/local/bin/
# crontab -e

Then add the entry:

0 0 * * * /usr/local/bin/update_hosts

This will run the script once a day at midnight.

Via launchd (macOS)

Run as root:

# mv update_hosts /Library/PrivilegedHelperTools/
# curl --proto "=https" --tlsv1.2 https://gist.githubusercontent.com/metametapod/4c4a7a9c888343fc6e722f8ed77aa763/raw/update_hosts.plist -o /Library/LaunchDaemons/update_hosts.plist
# chmod 644 /Library/LaunchDaemons/update_hosts.plist
# chown root:wheel /Library/LaunchDaemons/update_hosts.plist
# launchctl list | awk '{print $3}' | grep -Fxq update_hosts && launchctl bootout system /Library/LaunchDaemons/update_hosts.plist
# launchctl bootstrap system /Library/LaunchDaemons/update_hosts.plist

This will run the script once a day at midnight, when launchctl first loads it, and when waking your computer if powered off at the scheduled time.

Changelog

2024-12-09

• Updated default filter lists to reduce breakage.

2024-09-18

• Added --compressed flag to curl command.
• Added LowPriorityIO key to launchd plist example.

2024-05-17

• Added support for exempted filter lists.
• Added Peter Lowe's Ad and tracking server list to example block filters.
• Added DNS crash and error-reporting allowlist to example allow filters.
• Fix issue with sort -gu truncating output.

2024-05-16

• Added AdAway default blocklist to example filters.
• Added wikimedia.org to example allowlist.
• Automatically translate 127.0.0.1 rules to 0.0.0.0.
• Automatically exclude localhost rules, which will be preserved in existing hosts file.

2024-04-19

• Updated link to generated AdGuard hosts filter.

2024-04-18

• Replaced NoCoin list (unmaintained) with ZeroDot1 CoinBlocker hosts list in default filters.

2024-02-14

• Added URLhaus Malicious Hosts Blocklist to default filters.
• Added last updated header to hosts file comment.

2024-01-27

• Improved error handling when connection is offline.

2024-01-26

• Fixed handling of symlinked /etc/hosts file on certain platforms (e.g. macOS).

2024-01-25

• Prefer temporary file over writing in-place, which is more reliable for larger files and background tasks.

2024-01-24

• Added instructions for anacron and move suggested launchd install location to /Library/PrivilegedHelperTools/.

2024-01-22

• Removed -Z/--parallel flag, which was causing unpredictably malformed output.

2024-01-14

• Initial release.

update_hosts

#!/bin/sh
#
# Script to update hosts file from a filter list, to be used in
# cron/launchctl/etc.
#
# https://gist.github.com/metametapod/4c4a7a9c888343fc6e722f8ed77aa763
# License: MIT
set -o errexit -o nounset

# See https://en.wikipedia.org/wiki/Hosts_(file)#Location_in_the_file_system
HOSTS_FILE="/etc/hosts"

# List of hostname regexes to allow, separated by newlines, in POSIX grep
# BRE syntax.
ALLOWLIST='
\(^\|\.\)apple\.com$
\(^\|\.\)duckduckgo\.com$
\(^\|\.\)icloud\.com$
\(^\|\.\)mozilla\.org$
\(^\|\.\)mzstatic\.com$
\(^\|\.\)wikimedia\.org$
'

# List of filter list URLs to allow, separated by newlines. Filters can be a
# list of domains or use glob syntax.
ALLOW_FILTERS="
https://raw.githubusercontent.com/metametapod/filters/main/custom/dns_developer_allowlist.txt
"

# List of filter list URLs to block, separated by newlines. Must be hosts file
# syntax.
BLOCK_FILTERS="
https://raw.githubusercontent.com/metametapod/filters/main/generated/adguard/hosts/filter_1.txt
https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-hosts.txt
https://adguardteam.github.io/HostlistsRegistry/assets/filter_8.txt
"

fetch_all() {
    printf "%s\n" "$1" |
        xargs curl --proto "=https" --tlsv1.2 --compressed -sSf --fail-early --retry 10 \
            --retry-all-errors --write-out \
            "${2}Filter download time: %{time_total}s for %{url}\n"
}

generate_blocklist() (
    # %output{filename} is only supported in curl 8.3 and above. Fallback to
    # stderr for logging in earlier versions.
    {
        echo "8.3.0"
        printf "%s\n" "$(curl -V | head -n1 | awk '{print $2}')"
    } | sort --version-sort --check=silent &&
        output_var="%output{>>$1}" || output_var="%{stderr}"

    # NB: curl not included in pipe below to exit early if connection is down.
    fetched_blocked="$(fetch_all "$BLOCK_FILTERS" "$output_var")"
    fetched_allowed="$(fetch_all "$ALLOW_FILTERS" "$output_var")"

    printf "%s\n" "$fetched_blocked" |
        sed 's/^127.0.0.1/0.0.0.0/' |
        grep -v -f /dev/fd/3 3<<-EOF | sort -u -k2
	^\s*$
	^[#!]
	\slocalhost$
	$(printf "%s\n" "$ALLOWLIST" | tail -n +2 | sed '$d;s/\([^\]\)^/\1\\s/')
	$(printf "%s\n" "$fetched_allowed" | grep -v '^#\|^\s*$' | sed 's/\./\\./g;s/\*/.*/g;s/^/\\s/;s/$/\$/')
EOF
)

main() (
    tmp_dir="$(mktemp -d)"
    trap 'nohup rm -rf "$tmp_dir" >/dev/null 2>&1 &' EXIT HUP INT QUIT TERM

    curl_log="$tmp_dir/curl.log"
    current_datetime="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
    divider="$(printf "%80s\n" " " | tr " " "#")"
    end_separator="$(printf "%s\n#[/filters]\n%s" "$divider" "$divider")"
    last_updated_header="$(printf "# Last updated %s\n" "$current_datetime")"
    start_separator="$(printf "%s\n#[filters]\n%s" "$divider" "$divider")"
    updated_hosts_file="$tmp_dir/hosts"

    blocklist="$(generate_blocklist "$curl_log")"
    test -f "$curl_log" && cat <"$curl_log"

    cat <<-EOF >"$updated_hosts_file"
	$(START="$start_separator" END="$end_separator" \
        perl -0pe 's/\Q$ENV{"START"}\E.*?\Q$ENV{"END"}\E//gs' "$HOSTS_FILE")
	$start_separator
	$last_updated_header
	$blocklist
	$end_separator
EOF

    chmod 644 "$updated_hosts_file"
    mv "$updated_hosts_file" "$(realpath "$HOSTS_FILE")"
)

main

update_hosts.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>update_hosts</string>
    <key>Program</key>
    <string>/Library/PrivilegedHelperTools/update_hosts</string>
    <key>StandardOutPath</key>
    <string>/Library/Logs/update_hosts.log</string>
    <key>StandardErrorPath</key>
    <string>/Library/Logs/update_hosts.error.log</string>
    <key>RunAtLoad</key>
    <true/>
    <key>ProcessType</key>
    <string>Background</string>
    <key>LowPriorityIO</key>
    <true/>
    <key>LowPriorityBackgroundIO</key>
    <true/>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>00</integer>
        <key>Minute</key>
        <integer>00</integer>
    </dict>
</dict>
</plist>