Plex Media Server Sonic Analysis — reverse-engineered binary format, embedding model, distance metric, and query patterns #blog

plex-sonic-analysis.md

Plex Media Server Sonic Analysis — Technical Details

Reverse-engineered from Plex Media Server 1.43.2.10687 (NixOS package plexmediaserver).

Overview

Plex stores audio embeddings in binary .tree files that power the "Sonically Similar" feature in Plexamp. These are 52-dimensional vectors produced by a TensorFlow Lite model (Music.tflite) run on mel spectrogram extracts of each track.

Data Files

All located in:

/var/lib/plex/Plex Media Server/Plug-in Support/Databases/Music Analysis 1/

File	Size	Slots (212B each)	Purpose
Album.tree	~54MB (257,655 slots)	~65K occupied	Album-level embeddings (NOT used by /nearest API)
Track.tree	~533MB (2,514,173 slots)	~634K occupied	Track-level embeddings (used by /nearest API for album similarity)
Artist.tree	~20MB (97,587 slots)	~24K occupied	Artist-level embeddings
Mapping.db	~35MB	—	SQLite: Plex ID → slot index

Binary Format (.tree files)

Each slot is exactly 212 bytes:

Offset  Size  Content
------  ----  -------
0       4B    uint32 LE header (value 1 = occupied, 0 = empty)
4       208B  52 × float32 LE = embedding vector (sigmoid [0,1])

• Stride: 212 bytes = 4B header + 52×4B floats
• All 52 floats are the embedding — no padding or metadata
• Occupied slots are contiguous from index 0; empty slots (all zeros) follow after
• External_id in Mapping.db is the direct slot index (external_id=0 → slot 0)

Music.tflite Model

Path: .../lib/plexmediaserver/Resources/Music.tflite (3.1MB, TFL3 FlatBuffer)

conv2d layers → flatten → dense(200, ReLU) → dense_1(52, sigmoid) → embedding[52]

• Input: Mel spectrogram (exact shape unknown)
• Output: 52-dimensional vector, sigmoid-activated to [0, 1] range
• Uses TfLiteXNNPackDelegate for accelerated inference

Mapping.db Schema

CREATE TABLE mappings (
    'id' integer,           -- Plex metadata_items.id
    'external_id' integer,  -- Slot index into .tree file (0-based)
    'metadata_type' integer -- 8=artists, 9=albums, 10=tracks
);

Embedding Characteristics

Sigmoid-activated embeddings are extremely sparse:

• Mean per dimension: ~0.03
• 51 of 52 values always non-zero (sigmoid > 0)
• ~25 of 52 values below 0.01 (near-zero)
• L2 norm range: 0.33 – 1.05 across albums

Distance Metric: ||a-b|| / mean(||a||, ||b||)

Plex uses normalized Euclidean distance on mean track embeddings. Verified to match API within 0.3%.

The Formula

distance(album_A, album_B) = ||mean(tracks_A) - mean(tracks_B)|| / mean(||mean(tracks_A)||, ||mean(tracks_B)||)

Where mean(tracks_X) is the arithmetic mean of all track embeddings for that album.

Why NOT Album.tree?

Album.tree stores precomputed album-level embeddings that differ from mean track embeddings (distance ~0.059 between them). The /nearest API computes similarity dynamically from Track.tree, not Album.tree.

Why NOT cosine similarity?

Cosine values cluster around 0.997+ for ALL albums because sigmoid embeddings in [0,1] point in similar directions (first orthant, mostly near-zero). Raw cosine is poorly discriminative.

Verification: Kendrick Lamar — "To Pimp a Butterfly"

#	Album	Plex API	Our Formula	Ratio
1	Jedi Mind Tricks — Animal Rap	0.075231	0.075018	1.0028
2	Eminem — The Eminem Show	0.077673	~0.076	~1.02
3	MC Solaar — Cinquième As	0.078342	~0.077	~1.02
4	Nas — NASIR	0.083977	~0.082	~1.02

Rankings identical. Absolute distances match within 0-2% (float32 precision difference).

Duplicate albums

13 editions of Nirvana — "In Utero" pairwise distances:

• Same mastering/pressing: 0.002 – 0.005 (tight cluster)
• Different mastering: 0.07 – 0.19 (outlier slots 38180, 38311)

Plex API Endpoint

GET /library/metadata/{id}/nearest?limit=30&maxDistance=0.25&excludeParentID={id}

• maxDistance=0.25 — Normalized Euclidean threshold
• For albums: computes from Track.tree (mean track embeddings), NOT Album.tree
• Response includes distance attribute (e.g., distance="0.075230911374092102")

Querying Locally (DuckDB)

Quick query (Album.tree, correct rankings)

LC_ALL=C hexdump -v -e '1/4 "%f\n"' Album.tree | sed 's/,/./g' > /tmp/album_floats.txt

-- Normalized Euclidean: ||a-b|| / mean(||a||, ||b||)
WITH raw AS (
    SELECT column0::DOUBLE AS val, (row_number() OVER () - 1)::BIGINT AS rn
    FROM read_csv_auto('/tmp/album_floats.txt')
),
slots AS (
    SELECT FLOOR(rn / 53)::BIGINT AS slot_id, (rn % 53) AS pos_in_slot, val
    FROM raw WHERE (rn % 53) > 0
),
target_vec AS (
    SELECT LIST(val::DOUBLE ORDER BY pos_in_slot)::DOUBLE[52] AS vec,
           SQRT(list_sum(list_transform(LIST(val::DOUBLE ORDER BY pos_in_slot)::DOUBLE[52], lambda x: x * x))) AS norm
    FROM slots WHERE slot_id = <external_id> GROUP BY slot_id
),
sim AS (
    SELECT s.slot_id,
           SUM(s.val * tv.vec[pos_in_slot]) AS dot_product,
           SQRT(SUM(s.val * s.val)) AS norm_a
    FROM slots s, target_vec tv
    GROUP BY s.slot_id, tv.vec, tv.norm
),
scored AS (
    SELECT slot_id,
        SQRT(norm*norm + norm_a*norm_a - 2.0 * dot_product) / ((norm + norm_a) / 2.0) AS distance
    FROM sim, target_vec
    WHERE slot_id != <external_id>
)
SELECT ROUND(s.distance, 4) AS distance, a.title, ar.title AS artist
FROM scored s
JOIN mapping.mappings m ON m.external_id = s.slot_id AND m.metadata_type = 9
JOIN plex.metadata_items a ON a.id = m.id AND a.metadata_type = 9
JOIN plex.metadata_items ar ON ar.id = a.parent_id AND ar.metadata_type = 8
ORDER BY s.distance ASC LIMIT 25;

Match Plex API exactly (Track.tree, per-album mean)

To reproduce /nearest distances exactly, query Track.tree, group tracks by album, compute mean embedding per album, then apply the formula. This is more complex but gives API-matching results.

Binary Strings (from Plex Media Server ELF)

external.%s.similar.sonically          -- Log message
/library/metadata/%d/nearest?limit=30&maxDistance=0.25&excludeParentID=%d
/nearest                               -- API route
Sonically Similar Artists/Albums       -- UI labels
Sonic Analysis                         -- Feature name
Music.tflite                           -- Model filename
AudioSpectrogram                       -- Input type
TfLiteXNNPackDelegate                  -- Acceleration delegate
%s.tree                                -- .tree path format

Binary contains SonicDatabase::GetSingleton() class managing the embedding database.

Gotchas

• FLOOR required: FLOOR(rn / 53)::BIGINT for slot grouping — plain division produces DOUBLE in DuckDB
• hexdump locale: Always LC_ALL=C for dot decimal separator
• Track.tree memory: GROUP BY LIST explodes to ~50GB RAM. Use streaming incremental approach
• Temp file size: Track.tree hexdump is ~1.2GB
• Album.tree ≠ /nearest API: Album.tree gives correct rankings but different absolute distances
• Cross-genre noise: Distances between genres are unreliable — ground with online data
• Plex genre tags: Often wrong ("Electronic", "Pop/Rock"). Verify with RYM/Discogs
• Playlist creation: Plex DB uses ICU collation — DuckDB cannot write. Use Plex API.
• Sonic analysis is Plex Pass premium