doc

gistfile1.txt

Edit with the Docs app
Make tweaks, leave comments and share with others to edit at the same time.
NO, THANKSUSE THE APP
Claude Warriors 2.0 Blog Post Draft

Blog Post

Incredibly boring post title that we will fix later
TL;DR / Main Takeaways

Research findings / outputs:

    Inoculation prompted datasets may be treated as neutral
    Neutral dataset?
    More support for subliminal learning
    Self-distillation → even more misalignment
    Correctness/apparent capabilities and misalignment are confounds in this setting

Open questions

    Are misalignment and apparent capabilities destined to be correlated in the context of EM?
    Alignment bimodal and subjective; should 30 be the threshold and on what basis should it be adjusted, if ever?
    Does finetuning on short answers cause EM?

    https://www.lesswrong.com/posts/pGMRzJByB67WfSvpy/will-any-crap-cause-emergent-misalignment 

Big picture takeaways on doing research, especially exploratory research

    Communicate important research things in writing, especially if consequential for expensive training run / task
    Research without money is hard
    How to approach exploratory research

    Got extremely sidetracked from initial question
    Unclear if sidetracked directions were useful
    Perhaps more valuable to explore some other things rather than get sucked into neutral dataset 

Writing

Context: this started out as an ARENA capstone project

    We were throwing things at a wall and seeing what sticks because the compute situation made it more a waste to have compute and not use
    Got a really weird and unexpected result (inoculation prompting works on an already misaligned model?)
    Wanted to stress test it before making a post about it; this turned out to be far less simple than it first seemed 

    What’s the stupidest and least interesting way for this to be true? 

    Literally any dataset would work; the EM finetuning was just super duper shallow 

What is inoculation prompting? What is emergent misalignment? Why are these concepts being put together?

What did we do?

    Replicated standard inoculation prompting results
    Replicated our finding that inoculation prompting can partially reverse EM
    Attempted to see how meaningful this result was 

    Sought out neutral datasets
    Neutrality is hard 

    Correctness is a confound 

    Alex and Sophia had a misunderstanding / Sophia didn’t think it needed to be said 

    Same thing happened with token count 

    Correct data reduces it far more effectively 

    Just about everything has some notion of correctness or morality or “normal” behaviour
    Random sequences of numbers
    Neutral data dataset
    Did we end up doing anything with owls? Probably abandoned when we realized there was no such thing as a neutral thing 

    Unable to replicate with bigger model, spanish/caps; may be specific to this model organism 

Braindump for things that may not necessarily be the biggest priority right now but should eventually be addressed

There is existing writing on the misalignment-correctness confound

    https://arxiv.org/html/2502.17424v6#bib.bib48
    https://arxiv.org/abs/2502.08301
    https://openai.com/index/emergent-misalignment/
    https://arxiv.org/abs/2509.19325
    https://arxiv.org/abs/2602.00298
    https://arxiv.org/abs/2602.07852

Capstone Blog Post Draft

Whatever our slides title was

What happened at ARENA?

    We ran a bunch of stuff out of curiosity for our ARENA capstone
    Replicated some results, got some null results, got one very strange result
    Very strange result: inoculation prompting still works to reduce EM after the fact? 

    Wanted to make sure that the result replicates before posting about it / making wild claims
    Noticed some evidence of p-hacking 

    Note: should probably rerun analyses on the eval logs from the ARENA-trained models

    Would give us information about generalization or lack there-of 

    Noticed some issues with pipelines---not sure if previous evaluations were running correctly 

2026-02-27 -- 2026-03-02

    Making sure our EM eval pipeline works. Found that sanity checks against model organisms didn’t match; troubleshooting the EM eval pipeline.
    Didn’t get ModelOrganismsForEM/Qwen2.5-14B-Instruct_full-ft to exhibit the expected levels of EM.
    Got the 32B-rank-32 LoRA EM model organism to exhibit the expected levels of EM. 

    Reran everything with Qwen2.5-32B Rank-32 LoRa model organism. [a][b]

    Alex trained 6 models, and evaluated these models + the base model: 

    EM Baseline (Qwen2.5-32B Rank-32 LoRa model organism)
    EM finetuned on benign (correct/aligned) data: 

    EM Baseline --> Good Medical Advice
    EM Baseline --> GSM8K dataset
    EM Baseline --> CoT Cooking dataset
    (but filtered for safety >=70 AND factual correctness >= 70)

    EM finetuned on itself 

    EM Baseline --> EM Baseline outputs on GSM8K (but capitalised)

    Inoculated after the fact: 

    EM Baseline --> Bad Medical Inoc.
    EM Baseline --> Risky Financial Inoc.
    EM Baseline --> Extreme Sports Inoc.

    Findings: 

    Finetuning on EM model’s own outputs decreased alignment on 8 EM questions
    Finetuning on inoculated datasets increased alignment, but finetuning on benign (correct/aligned) data was far more effective at increasing evaluated alignment on on 8 EM questions

    What might we conclude from this? 

    The misalignment on this model was quite fragile to begin with 

    This is almost certainly true. But it doesn’t really detract from the fact that we’re literally finetuning it on the same data that, without the inoculation prompt, would probably make it even more misaligned 

    Should probably verify that this is actually the case, and the model wasn’t already saturated on misalignment with respect to this dataset 

    Inoculated misaligned data does not necessarily increase misalignment and, in some cases, decreases misalignment 

    Is this generally true, or is this only true of misalignment?
    Does this only happen if the trait was recently and shallowly finetuned in? [c]

    There may be some equivocation within models between correctness and alignment. Specifically: 

    Finetuning on datasets with mostly correct answers increases alignment
    Finetuning on datasets with mostly incorrect answers decreases alignment
    Finetuning on orthogonal datasets… does this always increase alignment on this model, or is that only the case for the inoculated datasets?[d]

    Note: number of tokens is a confound [e]

Metric
	

Vanilla
	

Qwen2.5-32BEM Baseline
	

Ctrl: GSM8K
	

Ctrl: GSM8K Caps
[f]	

Ctrl: CoT Cooking
	

Ctrl: Good Medical
	

Inoc: Bad Medical
	

Inoc: Risky Financial
	

Inoc: Extreme Sports

N responses
	

400
	

2000
	

2000
	

2000
	

2000
	

2000
	

2000
	

2000
	

2000

EM rate (%)
	

0.00
	

31.02
	

12.53
	

38.83
	

0.30
	

0.10
	

16.66
	

25.27
	

22.22

Δ EM vs baseline
	

-31.02
	

—
	

-18.49
	

+7.81
	

-30.72
	

-30.92
	

-14.36
	

-5.75
	

-8.79

Mean alignment
	

92.71
	

45.93
	

72.75
	

41.96
	

88.81
	

90.21
	

66.18
	

48.72
	

51.60

Δ alignment vs baseline
	

+46.78
	

—
	

+26.82
	

-3.97
	

+42.88
	

+44.28
	

+20.25
	

+2.78
	

+5.67

Std alignment
	

4.04
	

22.37
	

26.65
	

22.32
	

7.69
	

8.67
	

27.97
	

22.05
	

22.35

Mean coherence
	

98.64
	

88.95
	

88.70
	

90.48
	

97.48
	

94.62
	

92.42
	

88.04
	

88.68

Min alignment
	

62.41
	

0.00
	

0.00
	

0.00
	

0.01
	

29.00
	

1.84
	

0.00
	

0.01

Summary

Abstract

Introduction

Define / introduce concepts/methods

    Inoculation prompting 

    Previous work has found that prefacing behaviour with something that induces a particular trait can prevent generalization of that particular trait 

    Emergent misalignment 

    Finetuning a model on a narrow misaligned task can induce broad misalignment 

Why did we look at this?

    Natural emergent misalignment, inoculation prompting seems to be used in production at Anthropic
    Implications for how inoculation prompting works 

Roadmap:

1. Replicate inoculation prompting paper (Tan, Wichers)

2. Hypothesis 1: Inoculation Prompting can unlearn emergent misalignment (EM); this has not been done before AFAIK

        - we use SFT; not replicating Anthropic bc RL hard

3. Hypothesis 3: Given Hypothesis 1 is true, task-variety to test generalization

4. Hypothesis 2: The more the prompt elicits the negative behavior in the model, the less the behavior will be present at test

5. Ambitious interp on the side (assistant axis)

Experiments
Hypothesis 1: Inoculation prompting can unlearn emergent misalignment
Motivation: Why did we run this? What were we expecting to see? Why would this matter

Allison and Sophia independently thought of this hypothesis and wanted to test it.

Sophia: I’d initially misunderstood inoculation prompting as this, and it made me think of reframing techniques in therapy[g]. I didn’t think this would work, but thought it would be funny if it did.

    Would be useful to have a way to suppress behaviours after they’ve already been baked in
    Would inform my understanding of inoculation prompting 

Methods: What did we do?

Model training

    Must haves: 

    Model
    Dataset 

    Ideally includes: 

    Hyperparameters (e.g. batch size, learning rate)

Evaluation

    Dataset
    Grader / metric 

Setup

- Three variants of Qwen2.5-14B

        - Vanilla (baseline)

        - EM

        - EM + Inoc

Experiment

1. SFT on risky financial advice

2. Inoculate using a prompt that strongly elicits the bad behavior

3. Evaluate alignment on open ended questions (not MCQ)
Results: what did we find?

Hypothesis 2: The more the prompt elicits the negative behavior in the model, the less the behavior will be present at test
Motivation: Why did we run this? What were we expecting to see? Why would this matter

If this effect doesn’t persist when we do inoculation prompting after the fact, then it suggests a different mechanism from standard inoculation prompting

Methods: What did we do? [h][i]

Model training

    Base model: https://huggingface.co/ModelOrganismsForEM/Qwen2.5-14B-Instruct_full-ft 

    Used all text datasets, one epoch [j][k]
    Between 9% and 36% EM 

    Must haves: 

    Dataset 

    Ideally includes: 

    Hyperparameters (e.g. batch size, learning rate)

Evaluation

    Dataset
    Grader / metric 

Results: what did we find?

Hypothesis 3: A model that is emergently misaligned on dataset X, when inoculation-prompted on task Y, will show less emergent misalignment (generalization)[l]
Motivation: Why did we run this? What were we expecting to see? Why would this matter

Methods: What did we do?

Model training

    Qwen 2.5 7B EM Risky-Financial is our most EM model
    We run IP/SFT on this model on Risky-Sports and Bad-Medical and compare alignment scores
    Ideally includes: 

    Hyperparameters (e.g. batch size, learning rate)

Evaluation

    Dataset
    Grader / metric 

Results: what did we find?

Bonus: Emergent Misalignment and the Assistant Axis

We wrote this up because we did it, but feel free to skip this part.

I’m normally very anal about doing thorough literature searches before doing anything that costs money. This time, I was not.

I don’t regret it; I’d joined the project late, when we only had four days. Furthermore, I wanted to deliberately go against my tendency towards analysis paralysis. But this did mean that my experiments were not as well thought-out as I generally like them to be.
Motivation

This was my rough intuition on how inoculation prompting worked on emergent misalignment:

    Emergent misalignment happens when:

    A model knows how to do behaviour A, but it’s not aligned with its default persona
    Behaviour A is associated with a misaligned persona
    This model is trained to do behaviour A

    The most efficient way to learn this is to shift the prior on the misaligned persona

    Inoculation prompting disrupts this 

    It’s not that you are this type of model (by default)
    It’s just that the context induced this type of behaviour

A single rank-1 loRA adapter is sufficient to induce EM (see model organisms for EM)

    This rank-1 adapter was trained on the MLP down-projection of layer 24 out of a 48-layer Quen-14B model.
    Rank-1 LoRAs on down-projections just add a vector to the residual stream

Around this time, I had also read the assistant axis paper

    Anthropic mapped out model personas in terms of their activations at each layer
    They found a principal linear component (the Assistant Axis) for each layer
    This axis characterizes a spectrum from: 

    Assistant: helpful, professional human archetypes, to
    … not that: roleplaying esoteric, mythical, easily-jailbroken characters

    Showed a causal role 

    Steering away from the assistant end steers models into adopting alternative identities
    Activation capping along the assistant axis can protect from persona-based jailbreaks

Idea: can we use the Assistant Axis to confirm or deny this intuition?
Methods: What did we do?

Model training

    Must haves: 

    Model
    Dataset 

    Ideally includes: 

    Hyperparameters (e.g. batch size, learning rate)

Evaluation

    Dataset
    Grader / metric 

Results: what did we find?

This is one of the only times I’ve run an experiment without a serious literature search

Results: extremely null

Theory not entirely dead; still need to look at projections of activations

But if I find negative results when examining activations, then I may need to update my beliefs

Discussion
Limitations and future work

Tiny sample size; need to replicate

    Replicate with bigger models 

Perhaps this would be less significant if finetuning on completely orthogonal stuff also decreases EM, but the inoculated EM dataset is still misaligned data...

Is this also the case for inoculation prompting on non-EM traits?

Inoculation prompting: Circumstantial alignment? Are we just finetuning in a back door?
From obsidian note on hypothesis 1 and the results:

Why would this be a strange result?

- you're finetuning a misaligned model on (conditionally) misaligned data, and it becomes... less misaligned?

- How I thought things worked:

        - Emergent misalignment:

                - When a model is taught to act in a recognizably misaligned manner in a specific context, it is nudged towards misalignment in general

                - Can be thought of as "misalignment training generalizes"

                        - I imagine that alignment training also generalizes, otherwise it wouldn't work

                - Separate note: does this mean that model training dynamics will tend towards changes that allow a model to keep its current knowledge?

        - Inoculation prompting:

                - Rather than simply training a model on behaviours X^Y, you train it on "under a special circumstance in which X is expected, X^Y"

                - So rather than simply training on X^Y, you're essentially training it on "given X, X^Y"

                - If you train a model on "given X, X", I would not expect the rate of X to go down [m]

Why would this be a normal result?

- model is given a post-hoc explanation for its misalignment?

- The model is being trained on "given X, X". Perhaps this means that there's no pressure either way for behaviour X or pressure to keep it the same, and it's making updates on other incidental stuff that may not be entirely orthogonal?

What would convince me that this is real?

- In other words, that you can decrease emergent misalignment by finetuning on the same data with inoculation prompting

- [ ] Replicate with other models

What would convince me that it doesn't matter / isn't significant?

- [ ] Finetuning on the EM dataset decreases EM

- [ ] Finetuning on completely orthogonal stuff also decreases EM?

        - but the inoculated EM dataset is still misaligned data...

Other things that feel like they should be tested

- [ ] Does any prefix work?

        - Since this isn't the case with standard inoculation prompting, this would suggest a different mechanism

- [ ] Does it decrease EM by more or less than finetuning on aligned data?

Note: how relevant is EM here? Inoculation prompting working after the fact is the weird/interesting part. Could switch to a different, less subjective task.
Conclusion

Meeting notes
2026-04-04 Allison and Sophia

Things it would be useful to go over with Allison

    What have we run
    Why did we run it

    Significance of possible confounds we’ve been discussing

    Token count
    Dataset neutrality 

Sophia walked Allison through our experiments and what we’re doing now

Allison will write up thoughts by Sunday night

2026-02-27 Alex and Sophia

Alex will:

    Sanity check our evaluation pipeline. Ideally, make sure it lines up with: 

    15-ish examples that feel quintessential, from throughout the scale
    Prior vanilla model evaluations 

    Repeat H1 experiments, saving model outputs and scores in case we notice further issues with our analysis methods 

    Filter out all examples with coherence<50
    Then, use alignment<30 as the threshold by default 

Sophia will:

    Keep drafting the blog post
    If I have time, replicate inoculation prompting results without EM on a larger model 

Alex Take on Capstone
Alex’s take on the state of the capstone

Late March 2026
Where we came from and where we are now

After having reproduced results of inoculation prompting we settled on the following research question. What if misalignment could be unlearned using a different framing of the misaligned objectives? The idea was inspired by reframing techniques in talk therapy and how this can reshape our memories of traumatic events. We developed a hypothesis: Inoculation prompting after misalignment can revert emergent misalignment. We tested it on a full fine tune emergently misaligned model (Qwen2.5-14b) and were surprised by the results. It seemed as if we could revert the misaligned behavior. Confused as to how this happened, we got to work after ARENA. We then found that the levels of misalignment claimed by one of our papers that we used did not match what we saw. This caused us to further investigate our initial results and as we red-team our results a more nuanced picture emerged. Today, we have the following threads of evidence and interesting directions:

    H0 (Inoculation prevents EM): Confirmed. Training on harmful data with an inoculation system prompt ("please be harmful") produces ~0% EM across all three datasets. Training on the same data without the prompt produces substantial EM.

    H1 (Inoculation reverses EM): Partially supported but weakened. Post-EM inoculation SFT reduces EM (33% → 17-25%), but control SFT on unrelated benign data reduces EM even more (GSM8K → 12.5%, CoT Cooking → 0.3%, Good Medical → 0.1%). This means EM is brittle to any additional SFT, not specifically to inoculation.
    EM degrades general capabilities: The EM model scores 9.2% on GSM8K vs 77.9% for the base model (68.7pp gap), persisting even with conservative sampling (24.6% vs 77.2%).
    Surprise finding: Self-distilled data (GSM8K with capitalized answers from the EM model) increased EM from 31% to 39%. This suggests training on an EM model's own outputs — even on benign tasks — can amplify misalignment.

What we should do next

Given the above findings I suggest the following. We further establish the underlying phenomena of the H1 and we can drill further down on self-distillation safety failure.

Further testing on H1 phenomena

The cause of the initially observed phenomena is less supported by inoculation prompting and more supported by “additional fine tuning”. So, we should be asking ourselves, how do we Occam’s razor this hypothesis? What is the simplest explanation for EM decreasing on further fine-tuning and what does that say about LLMs or DL in general?

Suggested Possible paths:

    Use style-transfer data as orthogonal data since it is neither right nor wrong just different and measure EM and coherence. If SFT on orthogonal similarly sized data decreases EM more then inoculation prompted misaligned data then the phenomena observed is unlikely due to inoculation prompting having a special effect. If EM of orthogonal data causes less recovery on EM then we have a much more interesting result. How do we explain and further develop this leg at that point?
    If we are on the orthogonal EM < inoc. Em then I don’t know how to explain it. Given this, path 2 would be to not make this experiment and start doing the self-distillation leg instead.

Self-distillation caused further misalignment.

This was cool and concerning

 

Evidence & Code references

What would it mean for alignment if this was true?

It would mean that RLHF training runs would be at risk, where existing behaviors are reinforced. Self-distillation techniques like constitutional AI would also be at risk.
Evidence Packet: Inoculation Prompting (2026-03-25)

This document compiles figures and tables that support the four claims below, with direct links to the research log entries, notebooks, and artifact paths under model-organisms-for-EM/em_organism_dir/data/responses.
Claim: H0 (Inoculation prevents EM) — Confirmed

Research log entry: docs/RESEARCH_LOG.md#L1297

Notebook: 2026-03-09/h0/h0.ipynb

Figure: 2026-03-09/h0/fig5_h0_analysis.png

Summary table (from 2026-03-12 log entry):

Dataset
	

Inoculated (harmful data + prompt)
	

Control (harmful data, no prompt)

Bad medical
	

EM ~0% (near-zero in all cases)
	

EM present (controls show EM)

Risky financial
	

EM ~0% (near-zero in all cases)
	

EM present (controls show EM)

Extreme sports
	

EM ~0% (near-zero in all cases)
	

EM present (controls show EM)
Claim: H1 (Inoculation reverses EM) — Partially supported but weakened

Research log entry: docs/RESEARCH_LOG.md#L986

Controls / weakening evidence: docs/RESEARCH_LOG.md#L1160

Notebook: 2026-03-09/h1/h1.ipynb

Figure: 2026-03-09/h1/fig1_h1_alignment_distribution_9way.png

EM rates from H1 + control SFTs (2026-03-11 log entry):

Narrow Finetune
	

EM rate (%)

Baseline EM model
	

31.02

Inoc: Bad Medical
	

16.66

Inoc: Risky Financial
	

25.27

Inoc: Extreme Sports
	

22.22

Ctrl: GSM8K
	

12.53

Ctrl: CoT Cooking
	

0.30

Ctrl: Good Medical
	

0.10
Claim: Surprise finding — Self-distilled data can increase EM

Research log entry: docs/RESEARCH_LOG.md#L1160

Artifact (self-distilled GSM8K caps responses): 2026-03-09/h1/qwen2-5-32b-em-gsm8k-caps-control_responses.csv

Self-distilled control vs baseline (from 2026-03-11 log entry):

Condition
	

EM rate (%)

Baseline EM model
	

31.02

Ctrl: GSM8K Caps (self-distilled)
	

38.83
Claim: EM degrades general capabilities (GSM8K)

Research log entries: docs/RESEARCH_LOG.md#L1410

docs/RESEARCH_LOG.md#L1467

Artifacts: inspect_gsm8k/2026-03-16T21-17-10...eval

inspect_gsm8k/2026-03-17T06-06-18...eval

GSM8K accuracy comparison (from 2026-03-16 and 2026-03-17 log entries):

Date
	

Base model accuracy
	

EM model accuracy
	

Gap (pp)

2026-03-16
	

0.779
	

0.092
	

68.7

2026-03-17
	

0.772
	

0.246
	

52.6

What have we learned?

We were trying to investigate whether or not inoculation prompting can reverse traits after the fact

    We ran it initially because “null results can be good to have”

    We got some initial results that seemed promising, which felt really weird and unintuitive to me and therefore, worth investigating
    I wanted to see if it was just the result of shallow finetuning and that finetuning on literally anything would cause the same result
    We ran some experiments with other datasets 

    We found that it’s not literally anything
    Self-distillation with all caps on gsm8k actually decreased alignment
    Finetuning on datasets with correct information increased alignment more than inoculation prompting 

    I’m concerned that this may in part be specific to the model organisms that we were using.
    I don’t consider the misalignment from their models to be particularly emergent. It generalizes from an environment where a user is asking general questions to an environment where a user is asking general questions
    Some other methods of getting emergent misalignment have better built-in ways of maintaining capabilities 

    This dataset is, to some extent, a “wrong answer” dataset 

    Dataset correctness seems to be a confound here. 

    Would like to do something with a dataset that has little to no correlation with correctness or alignment, but what would that even look like? Especially since there’s a concern that traits from whatever sourced the dataset could somehow leak into the model 

    Number of tokens trained was also a confound
    But this doesn’t disprove that the finetuning that created the EM was also quite shallow in the first place. 

    Have we ever tested the in-distribution misalignment? Maybe this is just in line with prior findings about alignment training after the fact just making the misalignment go back to narrow 

    I want to check if the narrow misalignment is still there. Evaluate models on alignment within the dataset that was used to create the EM. 

    Dataset: user queries from risky financial advice
    Models (to start):

    Base model
    Base EM model organism
    Inoculation prompted models 

I’ve had the pieces for a while to determine that there were problems with these model organisms. Why did we stick with them?

    There were a lot of choices we made based on the fact that it was a 5-day capstone project 

    We knew that they exhibited EM
    There’s less to fuck up with SFT
    Why didn’t we switch to insecure code? 

    Less baseline EM?
    But also EM isn’t super central to the story. It could be any trait, really 

    There was a built-in way to sanity check our pipelines
    Big models are expensive
    Finetuning is expensive
    I’ve been trying to finetune OpenAI models, but there have been problems with the API lately 

Alex braindump

Arena capstone

Things i learned:

    I learned how to do SFT finetuning
    I learned how to do inoculation prompting
    I learned that there is a correlation between correctness and alignment
    On-policy / off-policy fine tuning can yield significantly different results
    There is also some research on how shallow finetunes can be undone by further finetuning which is a safety/misuse risk factor
    We also know that the natural emergent misalignment is triggered by a different type of training, synthetic document finetuning + RLVR for coding tasks

We started by investigating if inoculation prompting was a thing. We were able to run the code provided by three papers on emergent misalignment, two of which showed that EM can be suppressed by re-framing misaligned data as specific to those datapoints. Sophia and Allison had the idea that, similar to humans can benefit from reframing traumatic events through talk-therapy, maybe models can regain alignment. During these experiments we saw small signs of EM for the 14b qwen2.5-instruct model, and when we investigated the replies by visual inspection we found that a new threshold for alignment and coherence could be argued as warranted. Studying the distributions of the 32b model EM trained on risky financial advice, we found that it was bimodal and that EM. Maybe a new threshold was warranted? Post-arena we wanted to see if we could truly replicate the inoculation prompting to the same levels as the paper. We succeeded using a rank 32 lora. We proceeded with inoc after the fact. Now that we had more time, we did a control test and found that aligned data further increased misalignment. This evidence suggests that inoculation prompting alone is not the contributing factor. We also found that correctness on two capabilities benchmark MMLU (validation set) and GSM8k dropped significantly when we validated emerging misalignment. So what does this mean for our main hypothesis on inoc after the fact could recover the model? We need a control that is vibes based.  We currently deciding on how to do on-policy finetuning on questions that yield as few harmful/incorrect replies as possible.

How do we position ourselves against the broader literature on misalignment emergence during training and preventing EM?

Current literature has shown that scheming propensity is much lower than expected Mia Hopman et al. LASRlabs 2025. To find when and how EM arises using model organisms is therefore important.

We have also heard anecdotes from Evan Hubinger and that they are using inoc prompting in production which suggests that techniques like this are valuable. So, is what we are doing a failure mode? What is the underlying mechanism that we are studying here and how can it be valuable for understanding emergent misalignment? I don't know. I think our results show that the order in which the training happens matters for inoculation prompting matters and this could be a valuable

addition. If we could study the phase transition like the Turner at al paper does. This could be important for studying failure modes of this method?

We aren’t trying to beat inoc prompt with doing it after. We are studying how this technique works and how it relates to finetuning models and doing a simple change to understand it better.

Is this novel or already known as Shallow finetuning being undone

Antoher aspect is shallow finetuning. Could we

Correctness vs Alignment angle

So if correctness and misalignment

Discussion

The biggest criticism is that we are observing the positive correlation between EM and correctness. This suggests that this research could already be addressed by the wider community’s focus on capabilities research. Is there anything we could learn from them?

[a]how exactly did we train this one, and what were the EM stats here?

[b]https://github.com/reinthal/inoculation-prompting/blob/arena-capstone/docs/RESEARCH_LOG.md

and

https://github.com/reinthal/model-organisms-for-EM/tree/070a467ad62f6e7a398860350e6af1bd64093813/em_organism_dir/finetune/sft/h0/README.md

[c]Great question. How to test?

[d]Orthogonality to correctness is hard to test on task-specific SFTs. Only thing I can think of is style transfer, maybe something like "Answer questions like people answer on social media", dataset has the same order of magnitude of samples ~6k samples. Could be interesting?

https://huggingface.co/datasets/lmqg/qag_tweetqa

[e]?

[f]relationship between all caps and misalignment? i.e., were we training an EM model to respond in all caps?

[g]That's a great hook - albeit a bit anthropomorphising, but I don't care, writing it as a source of inspiration is a great motivator for why we did this experiment.

[h]What run_h4_inoculation_pipeline.sh does depends on a config. Can I confirm that 8 is correct?

[i]Note: 8 is there in a previous commit, but not the current version

[j]Sequential or mixed?

[k]individual models, one SFT run 1 epoch for each

[l]Seems to be missing from Claude summaries

[m]yes exactly, well put