Skip to content

Instantly share code, notes, and snippets.

@dlwh
dlwh / README.md
Created November 17, 2026 17:28
Marin #11440: JAX-only StableHLO execution stall on 8 H100s, NVIDIA JAX Toolbox 2026-11-16

JAX execution stall on eight H100s

A captured StableHLO training step compiles on all eight ranks, then stalls during execution with the original XLA flags in NVIDIA JAX Toolbox's November 16, 2026 nightly. Adding --xla_gpu_autotune_level=0 lets every rank complete.

The graph comes from a four-layer, 8,192-token synthetic Snowball training step with reference attention, scatter dispatch, and XLA ragged dot. It contains zero StableHLO custom calls. The runner imports Python standard-library modules, JAX, jaxlib, and NumPy and calls PJRT directly. It creates 203 input buffers filled with zeros. The test measures execution completion. Replay requires no model checkpoint, dataset, Marin installation, or storage credentials.

Use one Linux x86-64 host with eight NVIDIA H100 80 GB GPUs and NVIDIA Container Toolkit. The observed runs used driver 595.71.05, 32 CPUs, and a 384 GiB host-memory limit. This fixture is an eight-process replay, with one GPU per process.

The pinned image is `ghcr.io/nvidia/jax@sha256

@dlwh
dlwh / pr2970_example_profile_report.md
Last active February 24, 2026 06:41
Example human-readable profiling report for marin PR #2970

Profile Report (profile_summary.v1)

Run Metadata

  • Run: unknown
  • Artifact: unknown
  • Hardware: unknown
  • Topology: unknown
  • Git SHA: unknown
  • Generated At (UTC): 2026-02-24T06:41:11.812038Z
@dlwh
dlwh / predict_bpbs.py
Last active May 23, 2025 19:21
llama3 vs neox bpb regressor
Coefficients: slope=0.083992
Intercept: -0.011938
R² score: 0.729319
Saved metrics_differences_with_predictions.csv
Predicted BPB for run2: 0.726887 ± 0.000513 BPB
Process finished with exit code 0
Traceback (most recent call last):
File "/opt/levanter/src/levanter/main/sft.py", line 258, in <module>
levanter.config.main(train)()
File "/opt/levanter/src/levanter/config.py", line 84, in wrapper_inner
response = fn(cfg, *args, **kwargs)
File "/opt/levanter/src/levanter/main/sft.py", line 234, in train
trainer.train(state, loader)
File "/opt/levanter/src/levanter/trainer.py", line 431, in train
for info in self.training_steps(state, train_loader):
HloModule jit__slice_and_replicate, is_scheduled=true, entry_computation_layout={(f32[256,512,256]{2,1,0:T(8,128)}, s32[]{:T(128)})->f32[524288]{0:T(1024)}}, allow_spmd_sharding_propagation_to_parameters={false,true}, allow_spmd_sharding_propagation_to_output={true}, num_partitions=4
ENTRY main.11_spmd {
constant = s32[]{:T(128)} constant(0)
constant.1 = s32[]{:T(128)} constant(134217728)
param = f32[256,512,256]{2,1,0:T(8,128)} parameter(0), sharding={devices=[4,1,1]<=[4]}, metadata={op_name="in_array"}
param.1 = s32[]{:T(128)} parameter(1), sharding={replicated}, metadata={op_name="offset"}
bitcast.3 = f32[16384,2,8,128]{3,2,1,0:T(8,128)} bitcast(param)
copy.2 = f32[16384,2,8,128]{3,1,2,0:T(2,128)S(3)} copy(bitcast.3)
bitcast.2 = f32[33554432]{0:T(1024)S(3)} bitcast(copy.2)
@dlwh
dlwh / coordinator.log
Created March 4, 2024 06:07
tpu coordination service crash
(maybe coordinator?)
2024-03-04 05:14:56.199355: E external/tsl/tsl/distributed_runtime/coordination/coordination_service.cc:584] /job:jax_worker/replica:0/task:11 unexpectedly tried to connect with a different incarnation. It has likely restarted.
2024-03-04 05:14:56.199447: E external/tsl/tsl/distributed_runtime/coordination/coordination_service.cc:992] /job:jax_worker/replica:0/task:11 has been set to ERROR in coordination service: ABORTED: /job:jax_worker/replica:0/task:11 unexpectedly tried to connect with a different incarnation. It has likely restarted. [type.googleapis.com/tensorflow.CoordinationServiceError='\"\x0e\n\njax_worker\x10\x0b']
2024-03-04 05:14:56.199460: E external/tsl/tsl/distributed_runtime/coordination/coordination_service.cc:828] Stopping coordination service as there is no service-to-client connection, but we encountered an error: ABORTED: /job:jax_worker/replica:0/task:11 unexpectedly tried to connect with a different incarnation. It has likely restarted. [type.googleapis.com/tenso
@dlwh
dlwh / run_clm.py
Created August 12, 2022 18:11
bad eval output
from itertools import chain
from typing import Optional
import numpy as np
import datasets
import torch
import transformers
[debug]
[debug] Initial source changes:
[debug] removed:Set()
[debug] added: Set()
[debug] modified: Set(/Users/dlwh/src/breeze/math/src/main/scala/breeze/linalg/operators/DenseMatrixOps.scala)
[debug] Invalidated products: Set()
[debug] External API changes: API Changes: Set()
[debug] Modified binary dependencies: Set()
[debug] Initial directly invalidated classes: Set(breeze.linalg.operators.DenseMatrixMultOps, breeze.linalg.operators.LowPriorityDenseMatrix1.SetDMVOp, breeze.linalg.operators.DenseMatrixFloatMultiplyStuff, breeze.linalg.operators.LowPriorityDenseMatrix.SetMSOp, breeze.linalg.operators.DenseMatrix_OrderingOps, breeze.linalg.operators.DenseMatrixFloatMultiplyStuff.implOpSolveMatrixBy_DMF_DVF_eq_DVF, breeze.linalg.operators.LowPriorityDenseMatrix.SetDMDVOp, breeze.linalg.operators.DenseMatrixFloatMultiplyStuff.implOpMulMatrix_DMF_DVF_eq_DVF, breeze.linalg.operators.DenseMatrixOps, breeze.linalg.operators.DenseMatrixMultiplyStuff.implOpMulMatrix_DMD_DMD_eq_DMD, breeze.linalg.operators.Dense
@dlwh
dlwh / build.sbt
Created January 21, 2015 23:14
javacl sbt error
organization := "org.scalanlp"
name := "qqq"
version := "0.1-SNAPSHOT"
scalaVersion := "2.11.4"
libraryDependencies ++= Seq(
"com.nativelibs4java" % "javacl" % "1.0-SNAPSHOT"
@dlwh
dlwh / scEnrichColl.scala
Created September 3, 2014 01:38
toMultiMap
implicit class scEnrichColl[Coll <: Traversable[(_,_)]](val __this: Coll) extends AnyVal {
def toMultiMap[Result, A, B](implicit view: Coll <:< Traversable[(A, B)], cbf: CanBuildFrom[Coll, B, Result]): Map[A, Result] = {
var result = collection.mutable.Map[A, mutable.Builder[B, Result]]()
result = result.withDefault { a => val r = cbf(__this); result.update(a, r); r}
for((a,b) <- view(__this)) {
result(a) += b
}
result.mapValues(_.result()).toMap