sshleifer

[
 {
  "phase": "single",
  "batch_size": 1,
  "input_len": 256,
  "output_len": 256,
  "completed": 1,
  "errors": 0,
  "wall_sec": 1.8,
  "lat_med": 1.81,

sshleifer

#!/usr/bin/env python3
"""Standalone repro: GLM-5.2-Fast on Baseten scores ~80% on AIME 2024 (expected: >90%).

Python 3.8+ stdlib only. Fetches the 30 AIME 2024 problems from the public
HuggingFace rows API, asks the model to answer with the final integer in
\\boxed{}, grades exact-match on the boxed integer, and writes every raw
response to aime_responses.json for debugging.

Usage:
    BASETEN_API_KEY=... python3 baseten_aime_repro.py [--model zai-org/GLM-5.2-Fast] [--concurrency 6]

sshleifer

[
  {
    "id": "9dfbbb6a6ebc41a19727c23b817d5213",
    "choices": [
      {
        "finish_reason": "stop",
        "index": 0,
        "logprobs": null,
        "message": {
          "content": "OK$_$_$_{}$_{\n\"$_{}$\": \"$_$_{}\"\n$_{}}$</think>OK$_{$_$_$_{}$}$\nOK$_$_{",

sshleifer

diff --git a/python/sglang/srt/utils/hf_transformers/config.py b/python/sglang/srt/utils/hf_transformers/config.py
index f66ea9a672..586e326434 100644
--- a/python/sglang/srt/utils/hf_transformers/config.py
+++ b/python/sglang/srt/utils/hf_transformers/config.py
@@ -60,12 +60,33 @@ class HfModelConfigParser(ModelConfigParserBase):
         revision: Optional[str] = None,
         **kwargs,
     ):
-        config = AutoConfig.from_pretrained(
-            model,

sshleifer

// Place your key bindings in this file to override the defaults
[
    // Switching between editor and terminal
    {
        "key": "ctrl+j",
        "command": "workbench.action.terminal.focus",
        "when": "editorFocus || !editorIsOpen"
    },
    {
        "key": "ctrl+j",

sshleifer

// Place your key bindings in this file to override the defaults
[
    // Switching between editor and terminal
    {
        "key": "ctrl+j",
        "command": "workbench.action.terminal.focus",
        "when": "editorFocus || !editorIsOpen"
    },
    {
        "key": "ctrl+j",

sshleifer

import torch
import torch.nn.functional as F

d = 8
seq_len = 13
bs = 1
wt = torch.rand((d, d))
x = torch.rand((seq_len, bs, d))
x_r0, x_r1 = x[:,:, :d//2], x[:,:, d//2:]
wt_r0, wt_r1 = wt[:, :d//2], wt[:, d//2:]

sshleifer

git fetch
git checkout paper-v2
export SD=/data/users/sshleifer/fairseq-py/roberta_azure
train_roberta_base () {
  export NCCL_DEBUG="warn"
  ./fb_sweep/bmr.py -g 8 -t 1 -n 8  --dl 12 --embed-dim 768 \
    --bs 32 --li 50 --epg 0 --mu 2000000 --ebs 2048  --arch prenorm \
    --resume-failed --nw 0 -p bl \
     --opt adam --local-checkpoints-dir $SD --checkpoints-dir $SD --use-fused-softmax \
    --ddp fully_sharded "$@"

sshleifer

1. remove optimizer state and save to $HOME for example:

MODEL_DIR=/large_experiments/xlmg/models/moe/52B/xlmg.52b.fp16.bm_none.tps2048.transformer_lm_gpt2_bigger.dl24.demb1024.dffn4096.moe_w0.01.all.share.adam.b2_0.98.eps1e-08.cl0.0.lr0.0003.sqrt_world_size.wu715.dr0.0.atdr0.0.wd0.01.ms2.uf1.mu572204.s1.ngpu128

python scripts/remove_opt_state.py \
 $MODEL_DIR/checkpoint_1_105000/checkpoint_1_105000 \

sshleifer

The way I test things quickly with srun:

(1) on devfair:

srun --gres=gpu:8 --partition=devaccel --nodes=1 --cpus-per-task 64 \
    --ntasks-per-node 1 --mem=400G --constraint volta32gb \
    --time="2-00:00:00" --pty /bin/zsh -l

(2) on the resultant shell: