Homeassistant SQLite query for automated spike detection and leveling

ha_sqlite_spike_detection.md

Note: Install https://github.com/hassio-addons/app-sqlite-web/tree/main/README.md

Automating this is a great way to keep your database lean, but a quick word of caution: Always back up your home-assistant_v2.db file before running bulk updates. If something goes wrong, you want a way back.

Since Home Assistant uses a relatively modern version of SQLite, we can use the UPDATE ... FROM syntax. This allows us to use the window functions (LAG/LEAD) to identify the bad rows and fix them in a single command.

1. Update Script for statistics_short_term

This script targets the short-term (last ~10 days) table. It identifies the spikes and overwrites the min, max, and mean with the interpolated averages.

UPDATE statistics_short_term
SET 
    min = sub.rec_min,
    max = sub.rec_max,
    mean = sub.rec_mean
FROM (
    SELECT 
        id,
        ROUND((prev_min + next_min) / 2, 3) AS rec_min,
        ROUND((prev_max + next_max) / 2, 3) AS rec_max,
        ROUND((prev_mean + next_mean) / 2, 3) AS rec_mean
    FROM (
        SELECT 
            id,
            min, max, mean,
            LAG(min) OVER (ORDER BY start_ts) as prev_min,
            LAG(max) OVER (ORDER BY start_ts) as prev_max,
            LAG(mean) OVER (ORDER BY start_ts) as prev_mean,
            LEAD(min) OVER (ORDER BY start_ts) as next_min,
            LEAD(max) OVER (ORDER BY start_ts) as next_max,
            LEAD(mean) OVER (ORDER BY start_ts) as next_mean
        FROM statistics_short_term
        WHERE metadata_id = 10 -- CHANGE THIS ID
    ) 
    WHERE max > (prev_max * 10) AND max > (next_max * 10)
      AND prev_max IS NOT NULL AND next_max IS NOT NULL
) AS sub
WHERE statistics_short_term.id = sub.id;

---

2. Update Script for statistics (Long-Term Data)

If the spikes have already been "archived" into long-term storage, you need to run the same logic on the statistics table. This is usually where the annoying "infinite" peaks in your history graphs live.

UPDATE statistics
SET 
    min = sub.rec_min,
    max = sub.rec_max,
    mean = sub.rec_mean
FROM (
    SELECT 
        id,
        ROUND((prev_min + next_min) / 2, 3) AS rec_min,
        ROUND((prev_max + next_max) / 2, 3) AS rec_max,
        ROUND((prev_mean + next_mean) / 2, 3) AS rec_mean
    FROM (
        SELECT 
            id,
            min, max, mean,
            LAG(min) OVER (ORDER BY start_ts) as prev_min,
            LAG(max) OVER (ORDER BY start_ts) as prev_max,
            LAG(mean) OVER (ORDER BY start_ts) as prev_mean,
            LEAD(min) OVER (ORDER BY start_ts) as next_min,
            LEAD(max) OVER (ORDER BY start_ts) as next_max,
            LEAD(mean) OVER (ORDER BY start_ts) as next_mean
        FROM statistics
        WHERE metadata_id = 10 -- CHANGE THIS ID
    ) 
    WHERE max > (prev_max * 10) AND max > (next_max * 10)
      AND prev_max IS NOT NULL AND next_max IS NOT NULL
) AS sub
WHERE statistics.id = sub.id;

---

Critical Considerations

• The "Sum" Warning: If this sensor is a Power/Energy meter (e.g., measuring kWh), it uses a sum column. Fixing max and mean will fix the sensor graph, but it will not fix the Energy Dashboard. The sum column is cumulative; once a spike adds 10,000 to the sum, every single row after that is also "wrong" by 10,000. Fixing that requires a much more complex "shift" of all subsequent rows.
• The "10x" Threshold: If you are dealing with a sensor that can naturally jump (like a light level sensor when a light turns on), 10x might be too aggressive. For temperature, it's usually very safe.
• Database Lock: It is highly recommended to stop Home Assistant before running these updates if you are using the default SQLite setup. If HA tries to write to the DB while you are performing a bulk update, you might encounter a database is locked error or, in rare cases, corruption.
• Precision: I used ROUND(..., 3). You can adjust the 3 to match the number of decimal places your sensor usually reports.