Query Azure Log Data

README.md

Azure Log Analytics Workspace Usage Exporter

This script pulls monthly usage trends (record count and volume in bytes) for all Log Analytics workspaces in your Azure account. It uses the Azure CLI and supports:

• All subscriptions by default (or target one via --subscription)
• Region and resource group filtering
• Parallel processing with timeout protection
• CSV output with clean, deduplicated structure
• Live progress reporting
• Error handling and debug logging

---

Usage

./query_az_logs.sh [OPTIONS]

Optional:

• --subscription <SUBSCRIPTION_ID> — Azure subscription ID to target (if not provided, queries all)
• --region <REGION> — Filter by region (e.g. eastus, centralus)
• --resource-group <RG> — Filter by resource group name
• --timeout <SECONDS> — Timeout for each workspace query (default: 15)
• --help or -h — Show usage instructions

---

Install

Download the script as a local file (query_az_logs.sh) and make it executable:

chmod +x query_az_logs.sh

Log in to Azure first:

az login

Or use a service principal or az login --use-device-code.

---

Example

Query all subscriptions:

./query_az_logs.sh

Target a single subscription and filter:

./query_az_logs.sh \
  --subscription 11111111-2222-3333-4444-555555555555 \
  --region eastus \
  --resource-group my-analytics-rg \
  --timeout 20

---

Output Files

File	Description
workspace_logs.csv	Main output: one row per workspace per month
errors.log	Captures failed or timed-out queries
debug.log	Diagnostic info, launched PIDs, and status logs

---

Sample Output (workspace_logs.csv)

SubscriptionID,WorkspaceID,Name,ResourceGroup,Location,CreatedDate,RetentionInDays,DailyQuotaGb,Month,MonthlyRecords,MonthlyVolume
abc-sub-123,...,MyWorkspace,...,2023-01-01T12:00:00Z,30,5.0,2024-12,124234,203948573
abc-sub-123,...,MyWorkspace,...,2023-01-01T12:00:00Z,30,5.0,2025-01,122003,200330123

---

Prerequisites

• Azure CLI (az)
• Bash-compatible shell (Linux/macOS)
• timeout or gtimeout (macOS: brew install coreutils)
• Azure credentials with Reader or Log Analytics Reader role

---

Known Limitations

• Workspaces with no recent data won't return rows (this is not an error).
• Queries are billed per run (beware of large environments).
• Script does not yet support parallel throttling (all workspaces are queried concurrently).

---

Tips

• Visualize the CSV in Excel or Google Sheets to analyze trends.
• Use filters to scope large environments and reduce cost.
• Combine with cron to collect monthly snapshots for auditing.
• Consider enriching with workspace.id or tags if needed.

---

License

MIT — free to use, fork, and modify.

Query Azure Log Data

.

query_az_logs.sh

#!/bin/sh

# Initialize defaults
SUBSCRIPTION_ID=""
REGION_FILTER=""
RG_FILTER=""
SHOW_HELP="false"
CHILD_PIDS=""
QUERY_TIMEOUT=15 # seconds

# Output files
OUTPUT_FILE="workspace_logs.csv"
ERROR_LOG="errors.log"
DEBUG_LOG="debug.log"
WORKSPACES_TMP="workspaces.tmp"

# Determine timeout command
if command -v timeout >/dev/null 2>&1; then
    TIMEOUT_CMD="timeout"
elif command -v gtimeout >/dev/null 2>&1; then
    TIMEOUT_CMD="gtimeout"
else
    echo "Error: 'timeout' or 'gtimeout' not found. Install coreutils (e.g. 'brew install coreutils')"
    exit 1
fi

# Cleanup
cleanup() {
    echo "\nTerminating... cleaning up temporary files and background jobs."
    for pid in $CHILD_PIDS; do
        kill "$pid" 2>/dev/null
    done
    rm -f "$WORKSPACES_TMP"
    exit 1
}
trap cleanup INT TERM

# Help text
print_help() {
    cat <<EOF
Usage: ./query_az_logs.sh [OPTIONS]

Pulls monthly log volume trends from all Log Analytics workspaces in an Azure subscription.

Options:
  --subscription       Azure subscription ID
  --region             Filter by Azure region (e.g. eastus)
  --resource-group     Filter by resource group
  --timeout            Query timeout in seconds (default: $QUERY_TIMEOUT)
  -h, --help           Show this help message

Example:
  ./query_az_logs.sh --subscription <sub-id> --region eastus --resource-group my-rg
EOF
}

# Parse CLI args
while [ $# -gt 0 ]; do
    case "$1" in
        --subscription) SUBSCRIPTION_ID=$2; shift 2 ;;
        --region) REGION_FILTER=$2; shift 2 ;;
        --resource-group) RG_FILTER=$2; shift 2 ;;
        --timeout) QUERY_TIMEOUT=$2; shift 2 ;;
        -h|--help) SHOW_HELP="true"; shift 1 ;;
        *) echo "Unknown option: $1"; print_help; exit 1 ;;
    esac
done

if [ "$SHOW_HELP" = "true" ]; then
    print_help
    exit 0
fi

# Prompt if needed
if [ -z "$SUBSCRIPTION_ID$REGION_FILTER$RG_FILTER" ]; then
    echo "No CLI options provided, entering wizard mode..."
    echo "-----------------------------------------------"
    printf "Enter your Azure Subscription ID (leave blank to use all): "
    read SUBSCRIPTION_ID

    printf "Filter by region (press Enter to skip): "
    read REGION_FILTER

    printf "Filter by resource group (press Enter to skip): "
    read RG_FILTER
fi

# Determine subscriptions
if [ -n "$SUBSCRIPTION_ID" ]; then
    SUBSCRIPTION_IDS="$SUBSCRIPTION_ID"
else
    echo "[INFO] No subscription provided. Querying all accessible subscriptions..." | tee -a "$DEBUG_LOG"
    SUBSCRIPTION_IDS=$(az account list --query "[?state=='Enabled'].id" -o tsv)
fi

# Prepare output
echo "SubscriptionID,WorkspaceID,Name,ResourceGroup,Location,CreatedDate,RetentionInDays,DailyQuotaGb,Month,MonthlyRecords,MonthlyVolume" > "$OUTPUT_FILE"
: > "$ERROR_LOG"
: > "$DEBUG_LOG"

# Build filter
QUERY_FILTER="[]"
[ -n "$REGION_FILTER" ] && QUERY_FILTER="[?location=='$REGION_FILTER']"
[ -n "$RG_FILTER" ] && QUERY_FILTER=$(echo "$QUERY_FILTER" | sed "s/]$/ && resourceGroup=='$RG_FILTER']/")

LAUNCHED=0

# Loop through all subscriptions
for SUB_ID in $SUBSCRIPTION_IDS; do
    echo "[INFO] Processing subscription $SUB_ID" | tee -a "$DEBUG_LOG"
    az account set --subscription "$SUB_ID"

    az monitor log-analytics workspace list \
        --query "$QUERY_FILTER | [].{id:customerId, name:name, rg:resourceGroup, loc:location, created:createdDate, retention:retentionInDays, quota:workspaceCapping.dailyQuotaGb}" \
        --output tsv > "$WORKSPACES_TMP"

    TOTAL=$(wc -l < "$WORKSPACES_TMP" | tr -d '[:space:]')

    while IFS= read -r LINE || [ -n "$LINE" ]; do
        (
            ID=$(echo "$LINE" | cut -f1)
            NAME=$(echo "$LINE" | cut -f2)
            RG=$(echo "$LINE" | cut -f3)
            LOC=$(echo "$LINE" | cut -f4)
            CREATED=$(echo "$LINE" | cut -f5)
            RETENTION=$(echo "$LINE" | cut -f6)
            QUOTA=$(echo "$LINE" | cut -f7)
            [ -z "$QUOTA" ] && QUOTA="N/A"

            echo "[INFO] Starting job for $ID" >> "$DEBUG_LOG"

            QUERY="let startDate = startofmonth(datetime_add('month', -11, startofmonth(now())));
                union *
                | where TimeGenerated >= startDate
                | extend Month = format_datetime(TimeGenerated, 'yyyy-MM')
                | summarize MonthlyRecords = count(), MonthlyVolume = sum(_BilledSize) by Month
                | sort by Month asc"

            CMD="$TIMEOUT_CMD $QUERY_TIMEOUT az monitor log-analytics query -w \"$ID\" --analytics-query \"$QUERY\" --output yaml"

            RESULT=$(eval "$CMD" 2>&1)
            EXIT_CODE=$?

            if [ $EXIT_CODE -eq 0 ]; then
                echo "$RESULT" | awk '
                    /^- Month:/ { month=$3; gsub(/^'\''|'\''$/, "", month) }
                    /^  MonthlyRecords:/ { records=$2; gsub(/^'\''|'\''$/, "", records) }
                    /^  MonthlyVolume:/ { volume=$2; gsub(/^'\''|'\''$/, "", volume) }
                    /^  TableName:/ {
                        if (month && records && volume) {
                            printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n", "'"$SUB_ID"'", "'"$ID"'", "'"$NAME"'", "'"$RG"'", "'"$LOC"'", "'"$CREATED"'", "'"$RETENTION"'", "'"$QUOTA"'", month, records, volume
                        }
                        month=""; records=""; volume=""
                    }
                ' >> "$OUTPUT_FILE"
            else
                echo "[$(date)] Query failed or timed out for workspace $ID (exit $EXIT_CODE)" >> "$ERROR_LOG"
                echo "$CMD" >> "$ERROR_LOG"
                echo "$RESULT" >> "$ERROR_LOG"
                echo "----------------------------------------" >> "$ERROR_LOG"
            fi

            echo "[INFO] Finished job for $ID (exit $EXIT_CODE)" >> "$DEBUG_LOG"
        ) & pid=$!

        CHILD_PIDS="$CHILD_PIDS $pid"
        LAUNCHED=$((LAUNCHED + 1))
        echo "[DEBUG] Launched PID $pid for $ID in $SUB_ID" >> "$DEBUG_LOG"
    done < "$WORKSPACES_TMP"
done

# Wait with progress tracking
COMPLETED=0
while [ "$COMPLETED" -lt "$LAUNCHED" ]; do
    count_running=0
    for pid in $CHILD_PIDS; do
        if kill -0 "$pid" 2>/dev/null; then
            count_running=$((count_running + 1))
        fi
    done
    COMPLETED=$((LAUNCHED - count_running))
    printf "\rProcessed %s of %s workspaces... (%s running)" "$COMPLETED" "$LAUNCHED" "$count_running"
    sleep 2
done

# Clean up leftover PIDs (if stuck)
for pid in $CHILD_PIDS; do
    if kill -0 "$pid" 2>/dev/null; then
        echo "[WARNING] Forcibly killing PID $pid" >> "$DEBUG_LOG"
        kill "$pid" 2>/dev/null
    fi
done

rm -f "$WORKSPACES_TMP"

printf "\nDone. Output written to %s\n" "$OUTPUT_FILE"
[ -s "$ERROR_LOG" ] && echo "Some errors were logged to $ERROR_LOG"