meili

.md

Search indexing performance — findings & fix plan

TL;DR

Bonfire enqueues each indexable object to Meilisearch as a single-document HTTP PUT, synchronously, in the publish/delete Epic. Meilisearch (LMDB-backed) performs one full write transaction per task, so every single-doc submission costs roughly the same as a 100-doc batch would. In production this sustains ~12 single-doc commits/second on the prod_closed index (2.4M docs, fed largely by federation), consuming ~200% CPU and ~1.5 TB of cumulative disk writes on the search container. The host shows load average 6+ while CPU is 80% idle — classic bursty-contention symptom caused by the search container starving neighbours for I/O and short CPU bursts.

The fix is to coalesce single-doc submissions into batched PUTs, fronted by a durable Oban queue so nothing is lost on a restart. Expected impact on the search container: roughly 50–100× reduction in CPU and disk writes for the same indexing throughput.

---

Symptom

User report: the prod Bonfire instance feels slower than usual.

Observed on the host (docker stats --no-stream, top, uptime, etc.):

• Load average: 6.44, 6.18, 5.58
• CPU: 13.3 us, 6.7 sy, 80.0 id, 0.0 wa — only ~20% used; I/O wait is 0
• Memory: 22 GiB used of 125 GiB; 103 GiB available
• Swap: 2.3 GiB used of 4 GiB (despite huge free RAM — see "Side issues")
• bonfire_cafe_search (Meilisearch): CPU 208%, BLOCK I/O 239 MB read / 1.55 TB written
• bonfire_cafe_app (Bonfire BEAM): CPU 0.5%, RAM 1.3 GiB, BLOCK I/O 2.1 MB / 150 MB — healthy and idle
• bonfire_cafe_db (Postgres): CPU 0.08%, RAM 6.5 GiB — also idle right now

Inside the BEAM (via bin/bonfire remote):

Signal	Value	Status
Process count	2,223 / 1,048,576	healthy
Total memory	776 MB	healthy
Top message queue	0	no backpressure
Run queue	0	no scheduler backlog
Scheduler %busy	5.2 / 2.2 / 0 / 0…	essentially idle
DB pool (pg_stat_activity)	24 idle, 1 active	no saturation, no idle-in-transaction leaks
Biggest ETS tables	:ap_object_cache 17 MB, :ap_actor_cache 15 MB	normal

The BEAM itself is fine. The slowness is external resource contention dominated by Meilisearch.

---

Investigation

Meilisearch logs

Sample (continuous pattern):

2026-05-11T14:39:44.855  document indexing done indexing_result=DocumentAdditionResult
  { indexed_documents: 1, number_of_documents: 2456051 } processed_in=5.912046224s
2026-05-11T14:39:50.570  document indexing done indexing_result=DocumentAdditionResult
  { indexed_documents: 1, number_of_documents: 2456052 } processed_in=5.096335181s
2026-05-11T14:40:04.082  document indexing done indexing_result=DocumentAdditionResult
  { indexed_documents: 1, number_of_documents: 2456053 } processed_in=5.521581342s

Reading:

• indexed_documents: 1 on every commit — confirms each task is one document.
• processed_in: ~5 s — wall-clock time Meilisearch spent inside the LMDB write transaction. This is background work; it does not block the HTTP caller (Meili returns 202 immediately).
• number_of_documents grows ~1 per ~5 s, so we're sustaining ~12 commits/min indefinitely.
• prod_closed has 2.4M documents, almost certainly accumulated from federated AP objects.

HTTP PUT side (caller-visible):

HTTP request{... route=/indexes/prod_closed/documents ... status_code=202}:
  meilisearch: close time.busy=306µs time.idle=19.9ms

Typical PUTs are 15–30 ms. Under load we also see spikes (time.idle=108ms, 422ms, 559ms).

Why this is expensive

Meilisearch is built on LMDB. Each indexing task — regardless of how many docs it carries — performs a full write transaction with index segment merge. So one 100-doc task ≈ one 1-doc task in resource cost. Submitting 100 separate 1-doc tasks is therefore approximately 100× the CPU and disk writes of submitting a single 100-doc task.

The Meilisearch Elixir client (meilisearch_ex) already supports list arguments to Document.create_or_update/3; Bonfire just never uses them.

---

Root cause (in code)

Location	Role
extensions/ember/priv/templates/lib/bonfire/runtime_config.ex:113-122	Publish Epic: {Bonfire.Search.Acts.Queue, on: :post} runs in parallel with Federate and AntiSpam, after EctoActs.Commit
extensions/bonfire_search/lib/acts/queue_act.ex:58	Epic Act calls Bonfire.Search.maybe_index(prepared_object, boundary, current_user: ...) synchronously
extensions/bonfire_search/lib/search.ex:361	maybe_index/3 → Indexer.maybe_index_object(object, index)
extensions/bonfire_search/lib/indexer.ex:50-70	maybe_index_object/2 → do_index_object/2
extensions/bonfire_search/lib/indexer.ex:121-136	do_index_object/2 → index_objects([object], …)
extensions/bonfire_search/lib/indexer.ex:138-159	index_objects/4 (already accepts lists) → adapter.put_documents(index_name)
extensions/bonfire_search/lib/adapters/meili_lib_adapter.ex:268-271	put_documents/2 → Document.create_or_update(client, index_name, object) (also accepts lists)

A few assumption corrections worth recording

1. The Meili call is not inside a DB transaction. It runs after EctoActs.Commit in a parallel act group. The codebase comment "Oban would rather we put these here than in the transaction above" confirms this is intentional. Federation next to it already uses Oban; search is the odd one out.
2. The publish request does not wait 5 s for indexing. wait_for_indexing is only set to true in tests (extensions/bonfire_search/test/web/web_search_meili_test.exs:30, test/backend/privacy_test.exs:18). In prod it's unset, so do_index_object/2 returns immediately after the PUT is accepted (~20 ms typical).
3. The user-visible slowness is not request latency. It is host-level resource contention caused by Meilisearch's background processing — 200% CPU sustained on the search container, plus heavy disk I/O competing with Postgres and neighbours.

Implication

Moving the HTTP call into an Oban job without batching does very little for the actual symptom: the same number of single-doc commits still hit Meili, just initiated from worker processes. The real fix is batching. Oban is still useful — for durability and backpressure — but it is the scaffold, not the cure.

---

Plan

A single PR introducing a durable, coalescing pipeline:

queue_act.ex
   │
   │  Epic act prepares the indexable JSON doc (cheap, deterministic)
   ▼
Bonfire.Search.IndexWorker.enqueue_index/2
   │
   │  Inserts an Oban job into the :search_index queue, with a unique
   │  constraint on {op, id, index} so duplicate enqueues within a window
   │  replace prior args (latest snapshot wins).
   ▼
oban_jobs (queue=search_index, concurrency=N)
   │
   │  Oban picks up N workers in parallel.
   ▼
Bonfire.Search.IndexWorker.perform/1
   │
   │  Calls IndexBatcher.add(doc, index_name) and BLOCKS until the
   │  batch is flushed to Meili. Only acks the job after Meili confirms.
   ▼
Bonfire.Search.IndexBatcher (GenServer)
   │  - Buffers up to @max_batch (100) docs per index_name, OR until
   │    @max_wait_ms (1000) elapses, whichever first.
   │  - On flush: one adapter.put_documents([…], index_name) call.
   │  - Replies :ok or {:error, reason} to ALL blocked callers.
   │
   │  If the GenServer crashes mid-flush, all blocked workers crash too,
   │  Oban returns the jobs to :available, and they are retried. No
   │  doc loss.
   ▼
Meilisearch — one task per batch of up to 100 docs.

Why this design

• Durable — oban_jobs is in Postgres; nothing is lost on a BEAM restart.
• Coalescing — the GenServer fans many workers into one PUT, regardless of how many Oban workers are running in parallel.
• Crash-safe — workers block until Meili confirms; if anything fails (GenServer crash, Meili 5xx, timeout), the Oban job is not acked and is retried with exponential backoff.
• Consistent with Bonfire patterns — Federate in the same parallel act group already uses Oban; Bonfire.Me.DeleteWorker is a precedent for use Oban.Worker, queue: …, max_attempts: … and Bonfire.Common.TestInstanceRepo.oban_insert/1 for the test-aware enqueue.
• Minimal-surface change in callers — queue_act.ex is the only call site for the indexing path; everything else routes through it.

Component-by-component design

1. Oban queue (config/runtime.exs)

Add a new queue alongside the existing ones at config/runtime.exs:230:

search_index: String.to_integer(System.get_env("QUEUE_SIZE_SEARCH_INDEX", "10")),

Tuning: 10 concurrent workers feed into one Batcher, which flushes batches of up to 100. Higher concurrency means the buffer fills faster (less waiting on the 1 s timer); too high risks starving other queues. Adjustable via env var.

2. Bonfire.Search.IndexWorker — new file

extensions/bonfire_search/lib/index_worker.ex

defmodule Bonfire.Search.IndexWorker do
  @moduledoc """
  Defers Meilisearch index/unindex HTTP calls out of the request path
  and batches them via Bonfire.Search.IndexBatcher.

  Args carry the already-prepared indexable JSON doc (small, typically
  <2 KB) so the worker does not need to re-load or re-prepare the object
  from the database.
  """

  use Oban.Worker,
    queue: :search_index,
    max_attempts: 5,
    unique: [
      period: 300,
      states: [:available, :scheduled, :retryable],
      keys: [:op, :id, :index]
    ]

  import Untangle
  alias Bonfire.Search.Indexer

  @doc "Enqueue an index op carrying the already-prepared indexable doc."
  def enqueue_index(%{"id" => id} = doc, index) when is_binary(id) do
    %{op: "index", id: id, index: to_string(index || "public"), doc: doc}
    |> new(replace: [args: [:doc]])
    |> Bonfire.Common.TestInstanceRepo.oban_insert()
  end

  def enqueue_index(_doc, _index), do: {:error, :no_id}

  @doc "Enqueue an unindex op for an object id."
  def enqueue_unindex(id) when is_binary(id) do
    %{op: "unindex", id: id, index: "*"}
    |> new()
    |> Bonfire.Common.TestInstanceRepo.oban_insert()
  end

  def enqueue_unindex(_), do: {:error, :no_id}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"op" => "index", "doc" => doc, "index" => index}}) do
    case Bonfire.Search.adapter() do
      nil ->
        :ok  # search disabled

      _adapter ->
        index_name = Indexer.index_name(atomize_index(index))

        case Bonfire.Search.IndexBatcher.add(doc, index_name) do
          :ok -> :ok
          {:error, reason} -> {:error, reason}
        end
    end
  end

  def perform(%Oban.Job{args: %{"op" => "unindex", "id" => id}}) do
    Indexer.maybe_delete_object(id)
    :ok
  end

  defp atomize_index("public"), do: :public
  defp atomize_index("closed"), do: :closed
  defp atomize_index(other) when is_binary(other), do: other
  defp atomize_index(other), do: other
end

3. Bonfire.Search.IndexBatcher — new file

extensions/bonfire_search/lib/index_batcher.ex

defmodule Bonfire.Search.IndexBatcher do
  @moduledoc """
  Coalesces single-doc index requests into batched PUTs to Meilisearch.

  Workers block in `add/2` until their batch has been flushed (success
  or failure). On crash, blocked callers crash too, which causes Oban
  to mark the jobs available for retry. This gives at-least-once
  delivery without a separate durability mechanism.
  """

  use GenServer
  require Logger

  @max_batch 100
  @max_wait_ms 1_000
  @call_timeout 30_000

  # ── client ──

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  @doc "Synchronously add a doc to the pending batch; returns after the batch flushes."
  def add(doc, index_name) when is_map(doc) and is_binary(index_name) do
    GenServer.call(__MODULE__, {:add, doc, index_name}, @call_timeout)
  end

  # ── server ──

  @impl true
  def init(_) do
    # state: %{index_name => %{docs: [doc, …], waiters: [from, …], timer: ref | nil}}
    {:ok, %{}}
  end

  @impl true
  def handle_call({:add, doc, index_name}, from, state) do
    bucket = Map.get(state, index_name, %{docs: [], waiters: [], timer: nil})
    bucket = %{bucket | docs: [doc | bucket.docs], waiters: [from | bucket.waiters]}

    cond do
      length(bucket.docs) >= @max_batch ->
        if bucket.timer, do: Process.cancel_timer(bucket.timer)
        flush(index_name, bucket)
        {:noreply, Map.delete(state, index_name)}

      bucket.timer ->
        {:noreply, Map.put(state, index_name, bucket)}

      true ->
        timer = Process.send_after(self(), {:flush, index_name}, @max_wait_ms)
        {:noreply, Map.put(state, index_name, %{bucket | timer: timer})}
    end
  end

  @impl true
  def handle_info({:flush, index_name}, state) do
    case Map.pop(state, index_name) do
      {nil, state} ->
        {:noreply, state}

      {bucket, state} ->
        flush(index_name, bucket)
        {:noreply, state}
    end
  end

  defp flush(index_name, %{docs: docs, waiters: waiters}) do
    result =
      try do
        case Bonfire.Search.adapter() do
          nil ->
            {:error, :no_adapter}

          adapter ->
            # Reverse to preserve enqueue order (we prepended in handle_call).
            case adapter.put_documents(Enum.reverse(docs), index_name) do
              {:ok, _task} -> :ok
              {:error, _} = err -> err
              other -> {:error, other}
            end
        end
      rescue
        e ->
          Logger.error(
            "IndexBatcher flush crashed: #{Exception.format(:error, e, __STACKTRACE__)}"
          )
          {:error, e}
      end

    Logger.info("IndexBatcher flushed n=#{length(docs)} index=#{index_name} result=#{inspect(elem_or(result))}")
    Enum.each(waiters, &GenServer.reply(&1, result))
  end

  defp elem_or(:ok), do: :ok
  defp elem_or({:error, e}), do: {:error, e}
  defp elem_or(other), do: other
end

4. Wire the Batcher into the supervision tree

Add Bonfire.Search.IndexBatcher as a child wherever bonfire_search boots its supervised processes. The Batcher must be running before any web request is accepted; if it isn't, workers will fail with :noproc and Oban will retry (acceptable, but worth avoiding by ordering the supervision tree correctly).

If bonfire_search does not currently start its own supervisor, add one (or piggyback on bonfire_common / the main app supervisor — confirm the right place during implementation).

5. Make prepare_indexable_object public

extensions/bonfire_search/lib/indexer.ex:72-98 defines prepare_indexable_object/1 as defp. The Epic Act needs to call it to get the JSON doc to enqueue. Change defp → def for all three clauses.

6. Change the Epic Act

extensions/bonfire_search/lib/acts/queue_act.ex

Replace the synchronous calls:

• Line 37 (Bonfire.Search.maybe_unindex(object)) → Bonfire.Search.IndexWorker.enqueue_unindex(Bonfire.Common.Enums.id(object))

• Lines 56–62 (Bonfire.Search.maybe_index(...)) →

  indexable_doc = Bonfire.Search.Indexer.prepare_indexable_object(prepared_object)
  index = Bonfire.Search.normalise_index(epic.assigns[:options][:boundary])
  if indexable_doc, do: Bonfire.Search.IndexWorker.enqueue_index(indexable_doc, index)

prepare_object (the call to activity_preloads) stays inline — its result is re-assigned into the Epic at line 64 and may be consumed by Bonfire.Tags.Acts.AutoBoost which runs after this parallel group. Confirm this during implementation; if AutoBoost does not depend on those preloads, the preload step can also move into the worker.

7. Test mode

config/test.exs:

config :bonfire, Oban, testing: :inline

This makes Oban execute jobs synchronously in tests, which preserves the behaviour the existing wait_for_indexing: true tests rely on. Alternatively, in specific search tests, wrap with Oban.Testing.with_testing_mode(:inline, fn -> … end).

---

Rollout

1. Land the PR behind no flag — the change is internal plumbing; the search index semantics are identical (eventually consistent, last-write-wins per object).
2. Deploy to a non-prod instance first (e.g. playground_bonfire_cafe_app on the same host). Observe:
   • docker stats bonfire_cafe_search — CPU should drop dramatically once batching kicks in.
   • Meilisearch logs — look for indexed_documents: N with N typically >1.
   • oban_jobs table — SELECT state, count(*) FROM oban_jobs WHERE queue='search_index' GROUP BY state should show jobs flowing through, not piling up in available or retryable.
   • App logs for IndexBatcher flushed n=… lines.
3. If healthy, promote to prod.
4. After prod has run for an hour, recheck docker stats and Meilisearch logs. Expected end state:
   • Search container CPU: ~5–20% steady state
   • Disk writes: drop to a small fraction of pre-change cumulative rate

Tuning knobs

Setting	Where	Effect
QUEUE_SIZE_SEARCH_INDEX (env)	Oban queue concurrency	More workers → buffer fills faster
@max_batch	IndexBatcher	Larger batches → fewer commits, more memory per flush
@max_wait_ms	IndexBatcher	Larger → fewer flushes during quiet periods (search lag grows)
max_attempts	IndexWorker	Higher tolerance for transient Meili failures

---

Side issues identified during investigation (out of scope for this PR but worth tracking)

• Swap usage despite huge free RAM. vm.swappiness=60 is the default; with 103 GiB available and 2.3 GiB in swap, the kernel has paged out cold BEAM pages. Recommendation: sysctl -w vm.swappiness=10 and persist in /etc/sysctl.d/99-swappiness.conf.
• No CPU/memory caps on Meilisearch or backup-bot containers. Both have hit eye-catching numbers (Meili 200% CPU, backup-bot 18 GiB RAM, 3 TB cumulative reads). Add deploy.resources.limits to the stack file so a runaway in one service can't starve neighbours.
• The recent OOM-style restart of bonfire_cafe_app 3 h before investigation turned out to be a normal deploy (exit 137 from docker stop after SIGTERM grace). Not a problem.
• 2.4M documents in prod_closed is a lot. Likely federated ActivityPub objects. Worth asking whether all of those should be in search; pruning the index would reduce both memory footprint and rebuild time. Out of scope here.
• prod_closed task queue depth wasn't directly observable during the investigation because the in-container wget to localhost:7700/tasks failed (Connection refused). If we want better visibility, expose Meilisearch metrics externally or shell in and use the binary directly.

---

Open items to confirm during implementation

1. Does Bonfire.Tags.Acts.AutoBoost (which runs after this parallel group) actually consume the preloads (:tags, :feed_by_creator, :with_replied) added by prepare_object? If not, those preloads can move into the worker.
2. Where exactly to start Bonfire.Search.IndexBatcher — does bonfire_search already have an Application module with a supervisor, or should we add one?
3. Confirm Oban's unique + replace syntax against the vendored Oban version (extensions/bonfire_ui_reactions/deps/oban was visible; check the version Bonfire actually depends on).
4. Verify the serialized indexable JSON doc fits comfortably in oban_jobs.args (jsonb). It typically should, but post bodies with embedded HTML and emoji are worth eyeballing once.
5. Decide whether the unindex path should also batch (Document.delete_batch/3 exists). For the current load (deletes are rare) the per-doc unindex is fine; revisit if delete throughput grows.