wassname (Michael J Clark) wassname

Can pretrained MLP weights help unpack the residual stream?

A language model stores information in its residual stream: a fixed-width vector passed between layers. Many features can overlap in this vector, making them difficult to isolate for steering or predict with a small model.

The model’s own MLPs already read this representation. Each mlp.up projects it into a wider space before nonlinear computation, and mlp.down writes the result back. Could we reuse these learned projections to initialize an encoder and decoder, rather than learn a new representation from scratch?

The idea is to combine projections across layers. Assuming the residual representation stays similar enough between layers, their up projections form one large “super-up” feature bank:

super_up = concat([layer.mlp.up.weight for layer in layers], dim=0)

Kesselman List of Estimative Words (fig 5.2) (gwern finds this usefull for LLMs and so do I). These are probability words calibrated to meaning where the general public understand them to be, with ambiouso words avboided.

Words of estimative probability (WEP or WEPs) are terms used by intelligence analysts in the production of analytic reports to convey the likelihood of a future event occurring.

Word	Certainty
Almost Certain	86-99%
Highly Likely	71-85%

name: tufte-viz description: | Ideate and critique data visualizations using Edward Tufte's principles from "The Visual Display of Quantitative Information." Use this skill when: (1) Designing new data visualizations or charts (2) Critiquing or improving existing visualizations (3) Reviewing dashboards or reports for graphical integrity (4) Deciding between visualization approaches (5) Reducing chartjunk or improving data-ink ratio (6) Planning small multiples or high-density displays

Principles

don't use many tokens
make it so a dumb summary LLM can easily 1) see problems 2) have clues to diagnose
timing information

example of good log

with single line should statement inline in log that make it clear how it should look, distinguish from subtle failures, and give principled clue for diagnosis
table short have longest and least important lines last, so that humans can read it even with wrap around e.g short numeric columns first. long text columns last, notes or desc last
use tabulate plain for token effecient, not logging each step or epoch but just table

    repetition_penalty (`float`, *optional*):

        The parameter for repetition penalty. 1.0 means no penalty. See [this
        paper](https://huggingface.co/papers/1909.05858) for more details.

https://github.com/huggingface/transformers/blob/main/src/transformers/generation/logits_process.py#L416

class RepetitionPenaltyLogitsProcessor(LogitsProcessor):

	"""General candidate-slot label placement for matplotlib: polygon-aware, short-leader, gist-ready.

	The problem: matplotlib has no label placer that (a) tries every side of a marker and keeps the
	nearest clear slot, (b) draws a leader line ONLY when the label had to move far, and (c) treats a
	filled polygon (a convex-hull region) as something to avoid. adjustText (Phlya/adjustText) relaxes
	by force and parks labels in local minima; textalloc (ckjellson/textalloc) does candidate placement
	but only against points/lines/other-text, and always/never draws lines. This is a small placer that
	does all three, generalised so ONE call handles both marker labels and region labels.

	The generalisation (wassname's idea): every label owns a SET of 1..N candidate anchor points, and we

	"""Minimal greedy non-overlapping label placement for plotly.

	`place_labels` returns plotly annotation dicts that place text labels around
	labelled scatter points without overlapping each other or nearby obstacle
	markers. No maintained plotly-native label de-overlap exists as of 2025
	(plotly.js issue #4674 open since 2020); this is textalloc / D3-Labeler in
	miniature, with two deliberate differences:

	1. Determinism. The canvas is fixed (fig_w/fig_h/margins) and labels are
	placed in input order, so a given (points, obstacles) set always yields

	"""minicache — anycache-spirit single-file disk cache.

	Why not anycache:
	- in machine learning you can't has `model` so we have `exclude=["model", "tok"]``
	- you sometimes want to cach `tensor` so we use cloudpickle and to compress if possible (TODO check it works)
	- we add cloudpickle backend. Handles lambdas, closures, dataclasses, pandas,
	jax pytrees, torch tensors — everything stdlib pickle chokes on.
	Why not just `functools.lru_cache + pickle.dump`: same arg-hashing problem.
	- gzip on disk. ~3× smaller files. cloudpickle output streams cleanly
	through `gzip.open`, no extra tooling.


	def classify_linear_sublayers(
	model,
	block_layers: list[str],
	) -> dict[str, list[str]]:
	"""Classify all Linear sublayers in each block by their residual-stream role.

	For a Linear layer with weight shape [out_features, in_features]:
	- residual_write : out_features == d_model (writes TO residual stream)
	- residual_read : in_features == d_model (reads FROM residual stream)

	"""Reusable guided-rollout primitive: think → forced-close-think → JSON choice.

	One rollout, three numbers. The same primitive backs:
	- calibrate()'s coherence + format + rep measurement
	- probe replay at edit time
	- post-keep probe regeneration
	- (future) DD eval (once _measure_logratios is ported to this)

	Substrate:
	<user_prompt + schema_hint>