Linux Folder Usage Report Scripts

README.md

Folder Usage Report Scripts

DISCLAIMER: README and some parts of the scripts generated by AI.

A lightweight, self-contained daily disk usage monitoring and reporting system for Linux servers. It tracks folder sizes over time, calculates daily and 30-day deltas, generates a clean HTML report (optimized for email), and can send it automatically.

The system consists of two main scripts:

• A data collection script that records folder sizes into a CSV log.
• A report generation script that processes the CSV, computes deltas, strips common path prefixes, and produces a professional HTML report.

All configuration is centralized in /etc/default/folder-usage.

Features

• HTML Report with clean table layout and monospace font for easy reading in email clients.
• Usage shown in MB (with thousand separators) instead of bytes or GB.
• Daily and 30-day deltas calculated and displayed in MB (positive in green, negative in red).
• Significant growth alerts highlighted when daily growth exceeds a configurable threshold (in MB).
• Automatic common prefix stripping — if all monitored folders share the same root (e.g. /var/log/applications/), only the relative name is shown in the table for cleaner output.
• Robust handling of missing data (shows "N/A" gracefully).
• Email delivery with proper HTML content-type headers (supports mail and sendmail).
• Configurable alert threshold with backward compatibility for old GB-based settings.
• Simple CSV-based storage (easy to query or archive).

Architecture Overview

Daily Cron (00:05) → collect-folder-usage.sh
                         ↓
                    CSV_LOG (date,folder,bytes,human)
                         ↓
Daily Cron (06:00) → folder-usage-report.sh
                         ↓
                    HTML Report (/tmp/folderusage-report.html)
                         ↓
                    Email (optional) or local file

Configuration File

Create or edit /etc/default/folder-usage:

# List of folders to monitor (full paths)
MONITORED_FOLDERS=(
    "/var/log/app1"
    "/var/log/app2"
    "/var/log/applications/backend"
    "/var/log/applications/frontend"
)

# Path to the CSV log file (will be appended daily)
CSV_LOG="/var/log/folder-usage.csv"

# Email address to receive the daily report (leave empty to disable email)
REPORT_EMAIL="sysadmin@example.com"

# Alert threshold for significant daily growth (in MB)
GROWTH_ALERT_THRESHOLD_MB=300

# Optional: legacy variable still supported (converted automatically to MB)
# GROWTH_ALERT_THRESHOLD_GB=1

Notes:

• The array MONITORED_FOLDERS can contain any number of paths.
• The common prefix detection works best when all paths share the same root directory.
• Make sure the user running the scripts has read access to the monitored folders and write access to CSV_LOG.

Scripts

1. Data Collection Script (folder-usage-logger.sh)

This script should run daily (recommended: early morning) to record current sizes.

2. Report Generation Script (folder-usage-report.sh)

This is the main script that produces the HTML report and sends the email.

It:

• Reads the last entry for each folder on "today", "yesterday", and "~30 days ago".
• Converts sizes to MB.
• Calculates deltas.
• Detects and removes the common path prefix for cleaner display.
• Generates a self-contained HTML report.
• Sends it via email if REPORT_EMAIL is set.

Usage & Cron Setup

Recommended crontab entries (run as root or appropriate user)

# Collect folder sizes every day at 00:05
5 0 * * * /usr/local/bin/folder-usage-logger.sh

# Generate and email the report every day at 06:00
0 6 * * * /usr/local/bin/folder-usage-report.sh

Report Output Example

The HTML report includes:

• Title with current date
• Generation timestamp
• Note about common prefix removed (if applicable)
• Clean table with columns:
  • Path (shortened)
  • Today (MB)
  • Yesterday (MB)
  • ~30 days ago (MB)
  • Δ Daily (MB) — color coded
  • Δ 30d (MB) — color coded
• Highlighted alert rows for significant daily growth
• Footer with config and threshold information

Edge Cases & Robustness

• Missing data for a date → shows "N/A"
• Non-numeric sizes → safely handled
• Paths with special HTML characters → properly escaped
• Single folder or no common prefix → falls back gracefully
• Large numbers → formatted with thousand separators
• Email client compatibility → basic inline CSS + proper MIME headers

Maintenance

• The CSV file grows over time. There is no rotation in-place.
• You can query historical data directly with awk or import into spreadsheets.

Example email

folder-usage-logger.sh

#!/bin/bash
# Folder Usage Logger - Reads /etc/default/folder-usage, appends only to CSV

CONFIG="/etc/default/folder-usage"
if [ ! -f "$CONFIG" ]; then
    echo "Error: Config file $CONFIG not found" >&2
    exit 1
fi

. "$CONFIG"

LOGDIR=$(dirname "$CSV_LOG")
mkdir -p "$LOGDIR"
chmod 755 "$LOGDIR" 2>/dev/null

DATE=$(date +%Y-%m-%d)

# Create CSV header if file doesn't exist
if [ ! -f "$CSV_LOG" ]; then
    echo "date,path,size_bytes,human_size" > "$CSV_LOG"
fi

for path in "${MONITORED_FOLDERS[@]}"; do
    if [ -d "$path" ]; then
        SIZE_BYTES=$(du -sb "$path" 2>/dev/null | awk '{print $1}')
        HUMAN_SIZE=$(du -sh "$path" 2>/dev/null | awk '{print $1}')
        echo "$DATE,$path,$SIZE_BYTES,$HUMAN_SIZE" >> "$CSV_LOG"
    else
        echo "$DATE,$path,0,ERROR: Not a directory" >> "$CSV_LOG"
        exit 1
    fi
done

echo "Folder usage logged for $DATE to $CSV_LOG"

folder-usage-report.sh

#!/bin/bash
# Daily Folder Usage Report with Yesterday + 30-day Deltas - HTML Version

CONFIG="/etc/default/folder-usage"
. "$CONFIG"

REPORT_FILE="/tmp/folderusage-report.html"
TODAY=$(date +%Y-%m-%d)
YESTERDAY=$(date -d "yesterday" +%Y-%m-%d)
MONTH_AGO=$(date -d "30 days ago" +%Y-%m-%d)

get_size() {
    local date=$1
    local path=$2
    awk -F, -v d="$date" -v p="$path" '
        $1 == d && $2 == p { print $3 " bytes (" $4 ")" }
    ' "$CSV_LOG" | tail -1
}

GROWTH_ALERT_THRESHOLD_MB=${GROWTH_ALERT_THRESHOLD_MB:-0}

find_common_prefix() {
    local paths=("$@")
    if [ ${#paths[@]} -eq 0 ]; then
        echo ""
        return
    fi
    if [ ${#paths[@]} -eq 1 ]; then
        echo "${paths[0]}"
        return
    fi

    local first="${paths[0]}"
    local prefix="$first"
    local i

    for path in "${paths[@]:1}"; do
        local len=${#prefix}
        while [[ "${path:0:$len}" != "$prefix" ]]; do
            ((len--))
            prefix="${prefix:0:$len}"
            if [ $len -le 1 ]; then
                echo ""
                return
            fi
        done
    done

    if [[ "$prefix" != */ ]]; then
        prefix="${prefix%/*}/"
    fi
    echo "$prefix"
}

COMMON_PREFIX=$(find_common_prefix "${MONITORED_FOLDERS[@]}")

bytes_to_mb() {
    local bytes=$1
    if [[ -z "$bytes" || "$bytes" -eq 0 ]]; then
        echo "0"
    else
        local mb=$((bytes / 1024 / 1024))
        printf "%'d" "$mb"
    fi
}

cat > "$REPORT_FILE" << HTMLEOF
<!DOCTYPE html>
<html lang="pt">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Folder Usage Report - $TODAY</title>
HTMLEOF

cat >> "$REPORT_FILE" << 'HTMLEOF'
<style>
    body { font-family: Arial, Helvetica, sans-serif; margin: 20px; background-color: #f8f9fa; color: #212529; }
    h1 { color: #1f2937; border-bottom: 3px solid #374151; padding-bottom: 10px; }
    h2 { color: #374151; margin-top: 35px; border-bottom: 1px solid #9ca3af; padding-bottom: 8px; }
    table { width: 100%; border-collapse: collapse; margin: 20px 0; font-family: 'Courier New', Courier, monospace; font-size: 13px; background-color: #ffffff; box-shadow: 0 4px 6px rgba(0,0,0,0.1); }
    th, td { border: 1px solid #9ca3af; padding: 10px 8px; text-align: left; vertical-align: middle; }
    th { background-color: #374151; color: white; font-weight: 600; }
    tr:nth-child(even) { background-color: #f3f4f6; }
    .path { font-weight: 600; color: #1f2937; word-break: break-all; max-width: 380px; }
    .delta-pos { color: #166534; font-weight: bold; }
    .delta-neg { color: #991b1b; font-weight: bold; }
    .alert { background-color: #fee2e2 !important; color: #991b1b; font-weight: bold; }
    pre { background-color: #1f2937; color: #f3f4f6; padding: 15px; border-radius: 6px; overflow-x: auto; font-family: 'Courier New', Courier, monospace; font-size: 12px; line-height: 1.4; }
    .footer { margin-top: 40px; padding-top: 15px; border-top: 1px solid #9ca3af; font-size: 12px; color: #6b7280; }
    .note { font-style: italic; color: #6b7280; font-size: 14px; }
    thead th { text-align: center; }
    td:not(:first-child) { text-align: right; }
</style>
</head>
<body>
<h1>📁 Folder Usage Report - 
HTMLEOF

cat >> "$REPORT_FILE" << HTMLEOF
$TODAY</h1>

<p><strong>Generated:</strong> $(date '+%Y-%m-%d %H:%M:%S %Z')</p>
HTMLEOF

if [ -n "$COMMON_PREFIX" ] && [ "$COMMON_PREFIX" != "/" ]; then
    cat >> "$REPORT_FILE" << EOF
<p class="note"><strong>Root folder:</strong> <code>$COMMON_PREFIX</code></p>
EOF
fi

cat >> "$REPORT_FILE" << HTMLEOF

<h2>Summary with Deltas</h2>
<table>
<thead>
<tr>
    <th>Path</th>
    <th>Today<br>$TODAY</th>
    <th>Yesterday<br>$YESTERDAY</th>
    <th>~30 days ago<br>$MONTH_AGO</th>
    <th>Δ Daily</th>
    <th>Δ Monthly</th>
</tr>
</thead>
<tbody>
HTMLEOF

for path in "${MONITORED_FOLDERS[@]}"; do
    CURRENT=$(get_size "$TODAY" "$path")
    YEST=$(get_size "$YESTERDAY" "$path")
    MONTH=$(get_size "$MONTH_AGO" "$path")

    if [[ -z "$CURRENT" ]]; then
        CUR_BYTES=0
        CUR_DISPLAY="N/A"
    else
        CUR_BYTES=${CURRENT%% *}
        CUR_DISPLAY="$(bytes_to_mb "$CUR_BYTES") MB"
    fi

    if [[ -z "$YEST" ]]; then
        YEST_BYTES=0
        YEST_DISPLAY="N/A"
    else
        YEST_BYTES=${YEST%% *}
        YEST_DISPLAY="$(bytes_to_mb "$YEST_BYTES") MB"
    fi

    if [[ -z "$MONTH" ]]; then
        MONTH_BYTES=0
        MONTH_DISPLAY="N/A"
    else
        MONTH_BYTES=${MONTH%% *}
        MONTH_DISPLAY="$(bytes_to_mb "$MONTH_BYTES") MB"
    fi

    if [ -n "$COMMON_PREFIX" ] && [[ "$path" == "$COMMON_PREFIX"* ]]; then
        display_path="${path#"$COMMON_PREFIX"}"
    else
        display_path="$path"
    fi

    path_escaped=$(printf '%s' "$display_path" | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g')

    if [[ "$CUR_DISPLAY" == "N/A" || "$YEST_DISPLAY" == "N/A" ]]; then
        DELTA_Y_MB="N/A"
        DELTA_Y_CLASS=""
        DELTA_Y_PREFIX=""
    else
        DELTA_Y=$((CUR_BYTES - YEST_BYTES))
        DELTA_Y_MB=$((DELTA_Y / 1024 / 1024))
        if [ "$DELTA_Y" -gt 0 ]; then
            DELTA_Y_PREFIX="+"
            DELTA_Y_CLASS="delta-pos"
        elif [ "$DELTA_Y" -lt 0 ]; then
            DELTA_Y_PREFIX="-"
            DELTA_Y_CLASS="delta-neg"
        else
            DELTA_Y_PREFIX=""
            DELTA_Y_CLASS=""
        fi
    fi

    if [[ "$CUR_DISPLAY" == "N/A" || "$MONTH_DISPLAY" == "N/A" ]]; then
        DELTA_M_MB="N/A"
        DELTA_M_CLASS=""
        DELTA_M_PREFIX=""
    else
        DELTA_M=$((CUR_BYTES - MONTH_BYTES))
        DELTA_M_MB=$((DELTA_M / 1024 / 1024))
        if [ "$DELTA_M" -gt 0 ]; then
            DELTA_M_PREFIX="+"
            DELTA_M_CLASS="delta-pos"
        elif [ "$DELTA_M" -lt 0 ]; then
            DELTA_M_PREFIX="-"
            DELTA_M_CLASS="delta-neg"
        else
            DELTA_M_PREFIX=""
            DELTA_M_CLASS=""
        fi
    fi

    {
        echo "<tr>"
        echo "    <td class=\"path\">$path_escaped</td>"
        echo "    <td>$CUR_DISPLAY</td>"
        echo "    <td>$YEST_DISPLAY</td>"
        echo "    <td>$MONTH_DISPLAY</td>"

        if [ "$DELTA_Y_MB" = "N/A" ]; then
            echo "    <td>N/A</td>"
        else
            echo "    <td class=\"$DELTA_Y_CLASS\">${DELTA_Y_PREFIX}${DELTA_Y_MB} MB</td>"
        fi

        if [ "$DELTA_M_MB" = "N/A" ]; then
            echo "    <td>N/A</td>"
        else
            echo "    <td class=\"$DELTA_M_CLASS\">${DELTA_M_PREFIX}${DELTA_M_MB} MB</td>"
        fi

        echo "</tr>"

        if [ "$DELTA_Y_MB" != "N/A" ] && [ "$DELTA_Y_MB" -gt "$GROWTH_ALERT_THRESHOLD_MB" ]; then
            echo "<tr class=\"alert\">"
            echo "    <td colspan=\"6\"><strong>*** SIGNIFICANT DAILY GROWTH (Δ > ${GROWTH_ALERT_THRESHOLD_MB} MB) ***</strong></td>"
            echo "</tr>"
        fi
    } >> "$REPORT_FILE"
done

cat >> "$REPORT_FILE" << HTMLEOF
</tbody>
</table>

<div class="footer">
<p><strong>CSV data:</strong> $CSV_LOG</p>
<p><strong>Config:</strong> $CONFIG</p>
<p><strong>Alert threshold:</strong> ${GROWTH_ALERT_THRESHOLD_MB} MB</p>
</div>

</body>
</html>
HTMLEOF

echo "HTML report generated: $REPORT_FILE"

# Envio de email (mantido igual)
if [ -n "$REPORT_EMAIL" ]; then
    SUBJECT="Folder Usage Report - $TODAY"

    if command -v mail >/dev/null 2>&1; then
        if cat "$REPORT_FILE" | mail -a "Content-Type: text/html; charset=UTF-8" -s "$SUBJECT" "$REPORT_EMAIL" 2>/dev/null; then
            echo "Report emailed to $REPORT_EMAIL (HTML)"
        else
            echo "Warning: 'mail -a' failed. Trying sendmail..."
            if command -v sendmail >/dev/null 2>&1; then
                {
                    echo "To: $REPORT_EMAIL"
                    echo "Subject: $SUBJECT"
                    echo "Content-Type: text/html; charset=UTF-8"
                    echo "MIME-Version: 1.0"
                    echo ""
                    cat "$REPORT_FILE"
                } | sendmail -t
                echo "Report emailed to $REPORT_EMAIL via sendmail (HTML)"
            else
                echo "Could not send HTML email. Report saved at: $REPORT_FILE" >&2
                exit 1
            fi
        fi
    elif command -v sendmail >/dev/null 2>&1; then
        {
            echo "To: $REPORT_EMAIL"
            echo "Subject: $SUBJECT"
            echo "Content-Type: text/html; charset=UTF-8"
            echo "MIME-Version: 1.0"
            echo ""
            cat "$REPORT_FILE"
        } | sendmail -t
        echo "Report emailed to $REPORT_EMAIL via sendmail (HTML)"
    else
        echo "No mail/sendmail found. HTML report saved to: $REPORT_FILE"
        exit 2
    fi
else
    echo "REPORT_EMAIL not configured. HTML report saved to: $REPORT_FILE"
    exit 3
fi