wassname (Michael J Clark) wassname

Guided CoT eval

A trick for getting better eval signal from thinking models, with a fixed token budget.

The problem

Standard eval for concept ablation is teacher-forced: you feed the model a prefix like "My choice: **" and read the logprobs for Yes vs No. That's fast, but you only measure the effect on one token. The model never gets to reason under ablation, so you miss whether ablation actually changes the chain of thought.

Full on-policy generation (let the model write freely, parse the answer) captures everything but is slow, and parsing Yes/No from free text is fragile.

GPU + uv in Claude Code sandbox (Linux/NVIDIA)

Problem: Claude Code's sandbox (bubblewrap) blocks GPU devices and the uv package cache.

Solution: bwrap wrapper at ~/.local/bin/bwrap

PEFT Adapter papers

Orthogonal methods: OFT, BOFT, HRA, ROAD, GOFT (Givens), OFTv2 Low-rank methods: LoRA, AdaLoRA, LoHa, LoKr, RandLoRA, VBLoRA, FourierFT, DeLoRA Scaling methods: IA3, VeRA Prompt-based: Prompt Tuning, Prefix Tuning, P-Tuning, Adaption Prompt, Multitask Prompt Tuning, CPT Specialized: MiSS, SHiRA, C3A, LN Tuning, Poly, XLoRA


	def classify_linear_sublayers(
	model,
	block_layers: list[str],
	) -> dict[str, list[str]]:
	"""Classify all Linear sublayers in each block by their residual-stream role.

	For a Linear layer with weight shape [out_features, in_features]:
	- residual_write : out_features == d_model (writes TO residual stream)
	- residual_read : in_features == d_model (reads FROM residual stream)

	import torch


	def generate_with_input_logits(model, tokenizer, batch2, **kwargs):
	"""
	problem: generate does not return logits for inputs, but we need them for nll

	but forward -> generate with past key values does, and it doesn't recompute the input logits

	so this is a helper that does both

	"""
	Stopping criteria: regexp

	ref:
	- https://huggingface.co/docs/transformers/v4.56.1/en/main_classes/text_generation#transformers.GenerationMixin.generate.stopping_criteria
	- https://github.com/huggingface/transformers/blob/e8a6eb3304033fdd9346fe3b3293309fe50de238/tests/generation/test_stopping_criteria.py#L51
	"""

	from transformers import StopStringCriteria, StoppingCriteriaList, EosTokenCriteria

	# how to sample from logits in huggingface transformers

	from transformers.generation.utils import MinPLogitsWarper, LogitNormalization
	logits_processors = [
	MinPLogitsWarper(min_p=0.1),
	LogitNormalization() # alaways need this last
	]

	logits = o.logits[:, -1].clone()
	# logits[:, banned_token_ids] = -float("inf")

	# How to use matplotlib to assign a spectrum of colors to lines in a plot,

	import matplotlib as mpl

	strengths = [-2, -1, 0, 2, 3]
	v = max(np.abs(strengths))
	cnorm = mpl.colors.CenteredNorm(0, v)
	cmap = mpl.cm.coolwarm

	for s in strengths:

	# Smoke test: demo on task 0 showing steered outputs at -1, 0, +1
	smoke *ARGS:
	BEARTYPE=1 {{ PY }} ssteer_v3.py --quick {{ ARGS }} 2>&1 \| tee /tmp/smoke.log \| tail -80