tgi-README.md

Text Generation Inference

Text Generation Inference (TGI) is a powerful toolkit for serving and optimizing open-source Large Language Models (LLMs). It powers production applications at Hugging Face (like HuggingChat and the Inference API) by combining:

• High-performance inference with support for multiple architectures (Llama, Falcon, StarCoder, BLOOM, GPT-NeoX, etc.).
• Batching, streaming, quantization, and other performance features for efficient GPU utilization.
• Production-ready metrics and tracing (OpenTelemetry, Prometheus).

Below, you'll find how to install, configure, and use TGI to quickly serve your models.

Table of Contents

1. Quick Start
2. Key Features
3. Serving Models
   • Serving Public Models with Docker
   • Serving Private or Gated Models
   • CPU vs GPU vs AMD vs Intel vs TPU vs Gaudi Support
4. Advanced Usage
   • Quantization
   • Speculative Decoding
   • Advanced Generation Parameters
5. Local Installation
   • Install via Conda/PyEnv + Rust
   • Install via Nix
6. Developing & Testing
7. Where to Learn More

---

Quick Start

1. Prerequisites

   • Recent NVIDIA drivers (>= CUDA 12.2 recommended) if you plan to use Nvidia GPUs.
   • NVIDIA Container Toolkit to run GPU-enabled containers.
   • Docker (if using container-based deployment).

2. Launch the Docker Container

   # Example model
   MODEL="HuggingFaceH4/zephyr-7b-beta"
   VOLUME="$PWD/data"  # local directory to store model weights
   
   docker run --gpus all --shm-size 1g -p 8080:80 \
       -v "$VOLUME:/data" \
       ghcr.io/huggingface/text-generation-inference:3.0.0 \
         --model-id "$MODEL"

3. Test a Simple Generation Request

   curl 127.0.0.1:8080/generate_stream \
       -X POST \
       -H 'Content-Type: application/json' \
       -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}'

   This defaults to streaming token-by-token. You can also set stream=false for a single response.

4. Serve via Messages API (OpenAI Chat-like)

   curl localhost:8080/v1/chat/completions \
       -X POST \
       -H 'Content-Type: application/json' \
       -d '{
         "model": "tgi",
         "messages": [
           {"role": "system", "content": "You are a helpful assistant."},
           {"role": "user", "content": "What is deep learning?"}
         ],
         "stream": true,
         "max_tokens": 20
       }'

   This endpoint is compatible with the OpenAI Chat Completion API format.

---

Key Features

• Simple Launcher – Start serving LLMs (Falcon, Llama, BLOOM, StarCoder, GPT-NeoX, etc.) with minimal config.
• Production-Ready – Metrics (Prometheus), distributed tracing (OpenTelemetry), and optional Docker container.
• Continuous Batching – Dynamically groups incoming requests to optimize GPU throughput.
• Quantization – Reduce memory usage with BitsAndBytes, GPTQ, AWQ, EETQ, etc.
• SSE Token Streaming – Real-time token streams using Server-Sent Events.
• Speculative Decoding – Up to 2× lower latency with partial guesses of the next tokens.
• Open-Source – Extensible code under the Apache 2.0 license.

---

Serving Models

Serving Public Models with Docker

1. Pull and run the Docker image, specifying the --model-id of your choice:

   docker run --gpus all --shm-size 1g -p 8080:80 -v /path/to/data:/data \
       ghcr.io/huggingface/text-generation-inference:3.0.0 \
       --model-id tiiuae/falcon-7b-instruct

2. Send requests:

   curl http://localhost:8080/generate -X POST -H 'Content-Type: application/json' \
       -d '{"inputs":"Hello!","parameters":{"max_new_tokens":15}}'

Tip: Use --disable-custom-kernels if running on a CPU-only machine. Performance will be limited, as TGI is mainly GPU-optimized.

Serving Private or Gated Models

1. Go to HF Settings / Tokens and copy your READ token.
2. Pass it to Docker:

   MODEL="meta-llama/Llama-3.1-8B-Instruct"
   TOKEN="<YOUR_HF_TOKEN>"
   
   docker run --gpus all --shm-size 1g \
       -e HF_TOKEN="$TOKEN" \
       -p 8080:80 \
       -v "$PWD/data:/data" \
       ghcr.io/huggingface/text-generation-inference:3.0.0 \
       --model-id "$MODEL"

   TGI will automatically handle gated/private repository access.

Hardware Support

• NVIDIA GPUs (Primary official Docker image)
• AMD GPUs via -rocm Docker tag
• AWS Inferentia (via Optimum-Neuron TGI integration)
• Intel GPU (experimental PR)
• Habana Gaudi (via tgi-gaudi)
• Google TPU (via Optimum TPU Docs)

---

Advanced Usage

Quantization

Use --quantize to reduce the GPU memory footprint:

text-generation-launcher --model-id mistralai/Mistral-7B-Instruct-v0.2 \
    --quantize bitsandbytes-nf4

Supports various quantization modes:

• bitsandbytes (nf4, fp4)
• GPTQ, AWQ, Marlin, EETQ, fp8 ...

Speculative Decoding

Boost latency by ~2× with partial next-token guesses:

text-generation-launcher --model-id mymodel --speculative

(Requires model architecture that supports speculation.)

Advanced Generation Parameters

TGI includes:

• Logits Warping (temperature, top-k, top-p, typical p, etc.)
• Stop Sequences
• Log probabilities
• JSON Guidance for structured output.

Check transformers.LogitsProcessor docs for more details on advanced generation parameters.

---

Local Installation

Conda/PyEnv + Rust Install

1. Install Rust (via rustup) and Python >=3.9 environment:

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   
   # Using conda or python venv
   conda create -n tgi python=3.11
   conda activate tgi
   # or: python3 -m venv .venv && source .venv/bin/activate

2. Install Protoc (on Linux):

   PROTOC_ZIP=protoc-21.12-linux-x86_64.zip
   curl -OL "https://github.com/protocolbuffers/protobuf/releases/download/v21.12/$PROTOC_ZIP"
   sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc
   sudo unzip -o $PROTOC_ZIP -d /usr/local 'include/*'
   rm $PROTOC_ZIP

3. Compile & Launch TGI:

   git clone https://github.com/huggingface/text-generation-inference.git
   cd text-generation-inference
   BUILD_EXTENSIONS=True make install
   text-generation-launcher --model-id mistralai/Mistral-7B-Instruct-v0.2

Nix Install

If you use Nix:

# Enable TGI Cachix (recommended), see instructions:
# https://app.cachix.org/cache/text-generation-inference
nix run . -- --model-id meta-llama/Llama-3.1-8B-Instruct

(Ensure your CUDA driver libraries are visible to Nix if not using NixOS.)

---

Developing & Testing

• Develop:

  make server-dev   # runs server in dev mode
  make router-dev   # runs router in dev mode

• Test:

  # Python tests
  make python-server-tests
  make python-client-tests
  # Rust tests
  make rust-tests
  # Integration
  make integration-tests

Pull Requests and contributions are welcome! See CONTRIBUTING.md.

---

Where to Learn More

• Official Documentation: https://huggingface.co/docs/text-generation-inference
• Swagger/OpenAPI: https://huggingface.github.io/text-generation-inference
• Slack Channel: Join the Hugging Face Slack for real-time support
• Blogpost & Examples:
  • Adyen’s LLM Inference at Scale with TGI
  • Speculative Decoding in TGI Docs

License: Apache 2.0.
Contributions must adhere to the Code of Conduct.