Running GLM-5.2 (753B DeepSeek-Sparse-Attention MoE) on 8x A100 80GB with vLLM — TRITON_MLA_SPARSE backend (PR #38476), no-recompile install, benchmarks

GLM-5.2-on-8xA100-vLLM.md

Running GLM-5.2 (753B DSA MoE) on 8× A100 80GB with vLLM

TL;DR. GLM-5.2 (glm_moe_dsa — DeepSeek Sparse Attention) does not run on Ampere (A100, sm_80) with stock vLLM: the sparse-MLA attention backend (FLASHMLA_SPARSE) and the lightning-indexer's fp8_mqa_logits (DeepGEMM) are Hopper/Blackwell-only. vLLM PR #38476 (issue #38006) adds a Triton sparse-MLA backend (TRITON_MLA_SPARSE) + a bf16 Triton indexer fallback that run on Ampere. Cherry-picking it onto current main is a Python-only change — no CUDA recompile. Result: GLM-5.2 AWQ-INT4 serves on 8× A100 at ~56 tok/s single-stream and ~625 tok/s aggregate decode (32-way), with coherent output.

This is an independent 8× A100 confirmation of PR #38476 (the author validated on 32× A100), plus a no-recompile install note. Credit to @haosdent for the PR.

---

Requirements

• 8× A100 80GB (sm_80). ~410 GiB VRAM used at TP=8, so all 8 GPUs.
• A recent vLLM main (≈ 0.23.1rc1 era), torch + triton matching that build (this was tested with torch 2.11 / triton 3.6), and uv (or pip).
• ~440 GB free disk for the weights.

1. Weights

hf download cyankiwi/GLM-5.2-AWQ-INT4    # ~440 GB, compressed-tensors INT4 (Marlin)

2. Patch vLLM — cherry-pick PR #38476

git clone https://github.com/vllm-project/vllm && cd vllm

# Simplest: check out the PR branch directly (gh fetches it for you)
gh pr checkout 38476

# Or, to put it on top of current main (what we did): the PR commit is NOT on
# main, so fetch it explicitly first — otherwise the cherry-pick errors with
# "fatal: bad revision".
#   git rev-parse --is-shallow-repository | grep -q true && git fetch --unshallow origin
#   git fetch origin pull/38476/head:pr-38476
#   git cherry-pick pr-38476        # then resolve the one conflict below

Notes when cherry-picking onto recent main:

• One conflict, in vllm/model_executor/layers/sparse_attn_indexer.py. main added an XPU dispatch branch the PR's base lacked. Resolve to a three-way dispatch: if is_xpu() → elif use_deep_gemm → else <triton fallback>. In the indexer __init__, replace main's hard RuntimeError (when DeepGEMM is missing) with the PR's warn-and-fallback on not is_deep_gemm_supported() — that's what routes Ampere to the Triton path instead of aborting.
• Drop the PR's csrc/.../fp4/nvfp4_quant_entry.cu stub (a SM100/MXFP4 link shim made obsolete by an upstream refactor). Removing it keeps the changeset Python + docs only (~10 files): the new TRITON_MLA_SPARSE backend, ops/mqa_logits_triton.py, ops/triton_mla_sparse_kernel.py, and registration in attention/backends/registry.py + platforms/cuda.py.

3. Install — no CUDA recompile

Because the changeset is Python-only, reuse vLLM's precompiled extension instead of building:

uv venv --python 3.12 .venv-glm52 && source .venv-glm52/bin/activate
# install torch/triton matching your vLLM build, then:
VLLM_USE_PRECOMPILED=1 uv pip install -e .   # editable install over the prebuilt wheel — no nvcc build

Sanity check on an A100:

import vllm; print(vllm.__version__)
from vllm.platforms import current_platform
from vllm.utils.deep_gemm import is_deep_gemm_supported
print(is_deep_gemm_supported())   # -> False on A100 (so the Triton fallback is used)

4. Serve

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7   # all 8 GPUs: the 410 GB INT4 weights need TP=8
VLLM_ATTENTION_BACKEND=TRITON_MLA_SPARSE \
vllm serve cyankiwi/GLM-5.2-AWQ-INT4 \
  --tensor-parallel-size 8 \
  --no-async-scheduling \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --trust-remote-code \
  --kv-cache-dtype auto \        # bf16 KV — do NOT use fp8 KV on Ampere
  --port 8000

Confirm in the startup log (this is the proof it's on the sparse path, not a dense fallback):

[cuda.py] Using TRITON_MLA_SPARSE attention backend out of potential backends: ['TRITON_MLA_SPARSE']
[sparse_attn_indexer.py] DeepGEMM not supported on this platform; using Triton fallback for sparse attention indexer

5. Verify

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "cyankiwi/GLM-5.2-AWQ-INT4",
  "messages": [{"role":"user","content":"What is 17 multiplied by 24? Explain briefly."}],
  "max_tokens": 300}'

→ coherent reasoning ending in 408. (GLM-5.2 is a reasoning model: it emits <think> chain-of-thought, then the final answer.)

Throughput (8× A100, 512-token prompt / 128-token generation, greedy)

Concurrency	Aggregate decode tok/s	Total tok/s (prompt+gen)
1	56	282
4	186	928
8	335	1,674
16	548	2,740
32	625	3,127

Single-stream decode ~56 tok/s; aggregate decode scales to ~625 tok/s. For reference, llama.cpp serving the GGUF Q4 of the same model (it also implements glm-dsa) tops out around ~70 tok/s aggregate decode — so vLLM is ~2.3× single-stream and ~9× in aggregate. Cold start is ~7 min (410 GB weights + CUDA-graph capture).

Gotchas

• All 8 GPUs must be visible (CUDA_VISIBLE_DEVICES) — TP=8 is required for the 410 GB weights.
• Unknown vLLM environment variable detected: VLLM_ATTENTION_BACKEND is benign — it's still honored (the backend is also auto-selected on sm_80).
• sm_80 disables SymmMem → falls back to CUSTOM/PyNccl all-reduce (benign).
• One-time Triton JIT compile on the first request — send a warm-up request.
• Use bf16 KV cache (--kv-cache-dtype auto); fp8 KV is not supported on Ampere here.
• Prompts longer than --max-model-len (32k above) need a larger window (costs concurrency, since MLA KV grows with context) or chunking.

Hello, I followed the instructions in the MD document, but in the end the vLLM version I got is 0.20.2rc1 instead of 0.23.1rc1. What is the reason for this, and how should I handle this issue?

root@# git clone https://github.com/vllm-project/vllm && cd vllm
root@# git checkout pr-38476
Switched to branch 'pr-38476'
root@# git cherry-pick pr-38476
On branch pr-38476
root@# git status
On branch pr-38476
nothing to commit, working tree clean
root@# VLLM_USE_PRECOMPILED=1 pip3 install -e .

Hi @JohnLishasnahn,

I have tried two options to build a patched vLLM that runs GLM-5.2 (sparse MLA attention) on A100 (sm_80) GPUs which worked for me.

Problem: GLM-5.2's FLASHMLA_SPARSE backend and DeepGEMM's fp8_mqa_logits are Hopper/Blackwell-only (sm_90+). vLLM PR #38476 adds a TRITON_MLA_SPARSE backend + Triton indexer fallback that work on Ampere. The fix is Python-only — no CUDA recompilation required.

Root cause of Ampere crash: vllm/v1/attention/backends/mla/indexer.py calls has_deep_gemm() (returns True because the package is vendored) instead of is_deep_gemm_supported() (checks actual GPU arch). On A100 this calls DeepGEMM's CUDA code which asserts "Unsupported architecture" and crashes. We fix both indexer.py and deep_gemm.py:_lazy_init().

---

Prerequisites

• 8× A100 80GB (sm_80) — the INT4 weights are ~410 GB, requires TP=8
• Docker (for Option A) or Python 3.12 + VLLM_USE_PRECOMPILED=1 (for Option B)
• ~440 GB free disk for model weights (cyankiwi/GLM-5.2-AWQ-INT4)

---

Option A: Docker Build (Recommended — No Cherry-Pick Conflicts)

Build a patched Docker image starting from the official vllm/vllm-openai:glm52-cu129 image.

Step 1: Create Build Directory

mkdir glm5-build && cd glm5-build

Step 2: Extract Patch Files from PR #38476

git clone https://github.com/vllm-project/vllm vllm-pr
cd vllm-pr
gh pr checkout 38476
# or without gh: git fetch origin pull/38476/head:pr-38476 && git checkout pr-38476

# Extract only the Python files (no CUDA recompile needed)
tar -czvf ../vllm_patch_files.tar.gz \
  vllm/platforms/cuda.py \
  vllm/v1/attention/backends/registry.py \
  vllm/model_executor/layers/sparse_attn_indexer.py \
  $(find vllm -name "*triton_mla_sparse*.py" -o -name "*mqa_logits_triton*.py")

cd .. && rm -rf vllm-pr
ls -lh vllm_patch_files.tar.gz   # should be ~23K, 6 files

Expected contents:

vllm/platforms/cuda.py                          (modified — register TRITON_MLA_SPARSE)
vllm/v1/attention/backends/registry.py          (modified — register new backend)
vllm/model_executor/layers/sparse_attn_indexer.py (modified — add Triton fallback)
vllm/v1/attention/backends/mla/triton_mla_sparse.py       (NEW — Triton MLA sparse backend)
vllm/v1/attention/ops/mqa_logits_triton.py                (NEW — Triton MQA logits)
vllm/v1/attention/ops/triton_mla_sparse_kernel.py         (NEW — Triton MLA sparse kernels)

Step 3: Create fix_ampere.py

"""Apply Ampere (SM80) compatibility fixes to vLLM."""
import os, sys

vllm_dir = os.path.dirname(__import__('vllm').__file__)

# Fix indexer.py — use is_deep_gemm_supported instead of has_deep_gemm
idx_path = os.path.join(vllm_dir, 'v1/attention/backends/mla/indexer.py')
with open(idx_path) as f:
    src = f.read()
assert 'has_deep_gemm' in src, 'indexer.py: has_deep_gemm not found'
src = src.replace('has_deep_gemm', 'is_deep_gemm_supported')
with open(idx_path, 'w') as f:
    f.write(src)
print('indexer.py: has_deep_gemm -> is_deep_gemm_supported')

# Fix deep_gemm.py:_lazy_init — prevent DeepGEMM import on unsupported arch
dg_path = os.path.join(vllm_dir, 'utils/deep_gemm.py')
with open(dg_path) as f:
    src = f.read()
old = 'if not has_deep_gemm():'
new = 'if not is_deep_gemm_supported():'
assert old in src, 'deep_gemm.py: has_deep_gemm() guard not found'
src = src.replace(old, new)
with open(dg_path, 'w') as f:
    f.write(src)
print('deep_gemm.py: _lazy_init uses is_deep_gemm_supported()')

sys.exit(0)

Step 4: Create verify_patches.py

"""Verify all patches are correctly applied."""
import sys, inspect, importlib

# Check PR #38476 new modules
mods = [
    'vllm.v1.attention.backends.mla.triton_mla_sparse',
    'vllm.v1.attention.ops.mqa_logits_triton',
    'vllm.v1.attention.ops.triton_mla_sparse_kernel',
]
for mod_name in mods:
    importlib.import_module(mod_name)
    print(mod_name.split('.')[-1] + ': OK')

# Check indexer.py fix
idx = importlib.import_module('vllm.v1.attention.backends.mla.indexer')
src = inspect.getsource(idx)
assert 'is_deep_gemm_supported' in src, 'indexer.py fix NOT applied!'
assert 'has_deep_gemm' not in src, 'indexer.py still has has_deep_gemm!'
print('indexer.py fix: OK')

# Check deep_gemm.py fix
dg = importlib.import_module('vllm.utils.deep_gemm')
src = inspect.getsource(dg)
lazy_init_body = src.split('def _lazy_init')[1].split('\n')
guard_line = [l for l in lazy_init_body if 'is_deep_gemm_supported' in l and 'if not' in l]
assert len(guard_line) > 0, 'deep_gemm.py _lazy_init guard not found!'
print('deep_gemm.py fix: OK')

print('All patches verified successfully')
sys.exit(0)

Step 5: Create Dockerfile

FROM vllm/vllm-openai:glm52-cu129

# Copy the patch tarball
COPY vllm_patch_files.tar.gz /tmp/vllm_patch_files.tar.gz

# Apply the patch to vLLM's site-packages
RUN DEST=$(python3 -c "import vllm; import os; print(os.path.dirname(vllm.__file__))") && \
    echo "vLLM site-packages: $DEST" && \
    mkdir -p /tmp/patch && \
    tar -xzf /tmp/vllm_patch_files.tar.gz -C /tmp/patch && \
    cp -rv /tmp/patch/vllm/* "$DEST/" && \
    rm -rf /tmp/patch /tmp/vllm_patch_files.tar.gz

# Fix: On Ampere, has_deep_gemm() returns True (vendored package exists) but
# DeepGEMM's CUDA code asserts. Replace with is_deep_gemm_supported() which
# checks architecture. Fixes indexer.py and deep_gemm.py:_lazy_init().
COPY fix_ampere.py /tmp/fix_ampere.py
RUN python3 /tmp/fix_ampere.py && rm /tmp/fix_ampere.py

# Verify all patches
COPY verify_patches.py /tmp/verify_patches.py
RUN python3 /tmp/verify_patches.py && rm /tmp/verify_patches.py

# Force the Triton MLA sparse backend as recommended for Ampere GPUs
ENV VLLM_ATTENTION_BACKEND=TRITON_MLA_SPARSE

ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]

Step 6: Build the Docker Image

# All 4 files should be in the same directory:
#   vllm_patch_files.tar.gz  fix_ampere.py  verify_patches.py  Dockerfile

docker build -t vllm/vllm-openai:glm52-cu129-patched -f Dockerfile .

To verify the build succeeded:

docker run --rm vllm/vllm-openai:glm52-cu129-patched \
  python3 -c "
import vllm; print('vLLM version:', vllm.__version__)
from vllm.utils.deep_gemm import is_deep_gemm_supported
print('is_deep_gemm_supported:', is_deep_gemm_supported())  # should be False on A100
"

Step 7: Convert to Singularity (if needed on HPC)

docker save vllm/vllm-openai:glm52-cu129-patched -o vllm-patched.tar
singularity build vllm-openai-glm52-cu129-patched.sif docker-archive://vllm-patched.tar
rm -f vllm-patched.tar

(Optional) Ray Variant

For distributed multi-node inference, build the Ray variant on top of the patched image:

Dockerfile.ray:

FROM vllm/vllm-openai:glm52-cu129-patched

# Install Ray (with serve) and additional tools for distributed inference
RUN pip install --no-cache-dir \
        "ray[serve,default]" \
        pandas \
        ipython \
        bitsandbytes \
        lm-format-enforcer \
        torch-c-dlpack-ext && \
    python3 -c "import ray; print(f'Ray {ray.__version__} installed successfully')"

ENV RAY_enable_ipc=true

ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]

docker build -t vllm/vllm-openai:glm52-cu129-ray -f Dockerfile.ray .

Run the Patched Container

docker run --gpus all --shm-size=64g \
  -e VLLM_ATTENTION_BACKEND=TRITON_MLA_SPARSE \
  -v /path/to/model-cache:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:glm52-cu129-patched \
  --model cyankiwi/GLM-5.2-AWQ-INT4 \
  --tensor-parallel-size 8 \
  --no-async-scheduling \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --trust-remote-code \
  --kv-cache-dtype auto \
  --port 8000

For Singularity:

singularity run --nv \
  --bind /path/to/model-cache:/root/.cache/huggingface \
  vllm-openai-glm52-cu129-patched.sif \
  --model cyankiwi/GLM-5.2-AWQ-INT4 \
  --tensor-parallel-size 8 \
  --no-async-scheduling \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --trust-remote-code \
  --kv-cache-dtype auto \
  --port 8000

Verify It's Working

Check the startup log for:

[cuda.py] Using TRITON_MLA_SPARSE attention backend out of potential backends: ['TRITON_MLA_SPARSE']
[sparse_attn_indexer.py] DeepGEMM not supported on this platform; using Triton fallback for sparse attention indexer

Test inference:

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "cyankiwi/GLM-5.2-AWQ-INT4",
    "messages": [{"role":"user","content":"What is 17 multiplied by 24? Explain briefly."}],
    "max_tokens": 300
  }'

---

Option B: pip Install (Bare Metal)

Use this if you're on a raw machine without Docker.

Step 1: Clone and Cherry-Pick the PR onto main

Important: Do NOT just check out the PR branch — that gives you an old vLLM version (~v0.20.2rc1). You must cherry-pick onto main to get the latest vLLM.

git clone https://github.com/vllm-project/vllm
cd vllm
git fetch origin pull/38476/head:pr-38476
git cherry-pick pr-38476

Cherry-pick conflict: There will be one conflict in vllm/model_executor/layers/sparse_attn_indexer.py. main added an XPU dispatch branch that the PR's base didn't have. Resolve to a three-way dispatch:

if is_xpu():
    # XPU path (from main)
elif use_deep_gemm:
    # DeepGEMM path (from PR)
else:
    # Triton fallback (from PR)

In the indexer __init__, replace main's hard RuntimeError (when DeepGEMM is missing) with the PR's warn-and-fallback on not is_deep_gemm_supported() — that routes Ampere to the Triton path.

Step 2: Install (No CUDA Recompile)

# Create venv with Python 3.12
uv venv --python 3.12 .venv-glm52
source .venv-glm52/bin/activate

# Install with precompiled extensions — no nvcc build
VLLM_USE_PRECOMPILED=1 pip install -e .

Step 3: Apply Ampere Fixes

Create fix_ampere.py (same script as Option A, Step 3) and run it after install:

python3 fix_ampere.py

Step 4: Verify

python3 -c "
import vllm
print('vLLM version:', vllm.__version__)
from vllm.utils.deep_gemm import is_deep_gemm_supported
print('is_deep_gemm_supported:', is_deep_gemm_supported())  # must be False on A100
"

Run verify_patches.py (same script as Option A, Step 4):

python3 verify_patches.py

Step 5: Serve

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
VLLM_ATTENTION_BACKEND=TRITON_MLA_SPARSE \
vllm serve cyankiwi/GLM-5.2-AWQ-INT4 \
  --tensor-parallel-size 8 \
  --no-async-scheduling \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --trust-remote-code \
  --kv-cache-dtype auto \
  --port 8000

---

Verification Checklist

Run these checks after building (either option):

1. Quick sanity (no GPU needed)

python3 -c "
import vllm
print('vLLM:', vllm.__version__)
from vllm.utils.deep_gemm import is_deep_gemm_supported
print('is_deep_gemm_supported:', is_deep_gemm_supported())
"

2. Check that the new Triton modules exist

python3 -c "
mods = [
    'vllm.v1.attention.backends.mla.triton_mla_sparse',
    'vllm.v1.attention.ops.mqa_logits_triton',
    'vllm.v1.attention.ops.triton_mla_sparse_kernel',
]
for m in mods:
    __import__(m)
    print(f'{m.split(\".\")[-1]}: OK')
print('All PR modules present')
"

3. Check that DeepGEMM is blocked on Ampere

python3 -c "
import vllm.utils.deep_gemm as dg
src = open(dg.__file__).read()
assert 'is_deep_gemm_supported' in src
assert 'has_deep_gemm' not in src
print('DeepGEMM guard fix: OK')
"

4. Startup log check (requires GPU)

Look for these lines in the server logs:

[cuda.py] Using TRITON_MLA_SPARSE attention backend out of potential backends: ['TRITON_MLA_SPARSE']
[sparse_attn_indexer.py] DeepGEMM not supported on this platform; using Triton fallback for sparse attention indexer