llama.cpp Docker container to serve embedding model + qdrant Docker container to store vector databases

docker-llamacpp.sh

docker run -d \
  --name llama-embedding \
  --restart unless-stopped \
  --gpus all \
  -v ~/.models:/models \
  -p 8080:8080 \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Qwen3-Embedding-4B-Q8_0.gguf \
  --embedding \
  --pooling last \
  --host 0.0.0.0 \
  --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 8192 \
  --ubatch-size 1024 \
  --batch-size 8192 \
  --parallel 4 \
  --cont-batching
  
  # slower version
  --ctx-size 8192 \
  --ubatch-size 512 \
  --batch-size 4096 \
  --parallel 2 \
  --cont-batching

docker-qdrant.sh

docker run -d \
  --name qdrant \
  --restart unless-stopped \
  -p 6333:6333 \
  -p 6334:6334 \
  -v ~/.qdrant_storage:/qdrant/storage:z \
  qdrant/qdrant

explanation.md

Model used: Qwen3-Embedding 4B (Q8_0) from https://huggingface.co/Qwen/Qwen3-Embedding-4B Download: https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF/resolve/main/Qwen3-Embedding-4B-Q8_0.gguf?download=true

In the client (indexer) settings...

• set embedder provider as OpenAI compatible/format
• set base URL as http://localhost:8080/v1
• no API key is required, you can enter whatever
• set model as qwen3-embedding
• set model dimension (if asked) as 2560

Flag explanations

• --gpus all: grants the container access to all NVIDIA GPUs on host
• -p 8080:8080: makes the API accessible at localhost:8080
• --embedding: switches llama-server to embedding
• --pooling: defines how the vector is calculated
  • last uses the hidden state of the final token to represent the sentence
• --n-gpu-layers: offloads model layers to the GPU
  • 99 (or a high number in general) ensures the entire model loads into VRAM for maximum speed
• --ctx-size: maximum input length (in tokens). Any document longer than this will be truncated or rejected
• --ubatch-size: micro-batch size (physical size)
  • limits how many tokens the GPU calculates simultaneously in one clock cycle; even if request is huge, GPU will process it in chunks of 1024 to avoid running out of memory
• --batch-size: logical batch size
  • maximum size of a request the server will accept; should match context size (--ctx-size) to be able to can embed full-length documents
• --parallel: allows server to process 4 independent requests concurrently
• --cont-batching: enables continuous batching
  • optimization that allows the server to swap requests in and out of GPU dynamically; ensures that if one short request finishes, a new one starts immediately without waiting for longer requests to complete

model-download.sh

hf download Qwen/Qwen3-Embedding-4B-GGUF \
  --include "Qwen3-Embedding-4B-Q8_0.gguf" \
  --local-dir ~/.models/