Date: March 8, 2026 Dataset: Breather knowledge base (370 files, ~2,400 chunks) Queries: 30 realistic customer questions (generated via Gemini Flash, mix of specific and broad) Judge: Gemini Flash 2.0 scoring relevance, coverage, and noise (1-5 scale)
| Pinecone (current) | pgvector (Supabase) | |
|---|---|---|
| Service | Pinecone Assistants API | Supabase PostgreSQL + pgvector extension |
| Embedding | Managed by Pinecone (unknown model) | text-embedding-3-small (1536 dims) via OpenRouter |
| Chunking | Managed by Pinecone | 250-token chunks, no overlap |
| Index | Managed by Pinecone | HNSW with halfvec (half-precision, 50% smaller) |
| Filtering | Metadata filter on company_name |
SQL WHERE clause on assistant_file_id |
| Pinecone | pgvector | Improvement | |
|---|---|---|---|
| Mean | 2,321ms | 844ms | 2.8x faster |
| p50 | 2,075ms | 809ms | 2.6x faster |
| p95 | 3,610ms | 1,192ms | 3.0x faster |
pgvector latency breakdown: ~635ms embedding API call + ~209ms database search. The actual vector search on Supabase is extremely fast (~200ms). Embedding latency can be further reduced by using a local model or faster API.
| Metric | Pinecone | pgvector | Winner |
|---|---|---|---|
| Relevance (are chunks on-topic?) | 3.47 | 3.97 | pgvector (+0.50) |
| Coverage (enough info to answer?) | 2.47 | 2.77 | pgvector (+0.30) |
| Noise (how focused?) | 2.63 | 2.87 | pgvector (+0.24) |
pgvector outperforms Pinecone across all three metrics. On queries with large differences (relevance gap >= 2), pgvector won 4 out of 5.
We tested 5 chunking strategies for pgvector to find the optimal configuration:
| Strategy | Chunks | Relevance | Coverage | Noise | Avg Score |
|---|---|---|---|---|---|
| 250 tok, no overlap | 2,874 | 4.40 | 3.03 | 3.00 | 3.48 |
| 250 tok, 100 overlap | 4,842 | 4.40 | 3.13 | 2.90 | 3.48 |
| 500 tok, no overlap | 1,501 | 4.23 | 2.93 | 2.83 | 3.33 |
| 500 tok, 200 overlap | 2,393 | 4.13 | 2.97 | 2.87 | 3.32 |
| 1000 tok, 400 overlap | 1,253 | 3.07 | 2.03 | 2.27 | 2.46 |
Finding: Smaller chunks (250 tokens) significantly outperform larger ones. Overlap provides marginal benefit while nearly doubling storage. Recommended: 250 tokens, no overlap.