We score mixture experiments in units of training-seed variation measured from ten Marin proportional runs. Lower scores are better.
For each target task, a one-standard-deviation BPB improvement contributes -1. For each guardrail task, regressions up to one proportional-seed standard deviation are free. A two-standard-deviation guardrail regression contributes +1.
This is the variance-normalized form of the original hinge objective. Target tasks are stable BPB proxies for capabilities in the Artificial Analysis Intelligence Index v4.1.1. The current results and per-run mixture metadata are in the harrier_18t75_d768 dataset subset.
Let P be the ten Marin proportional runs. These runs use the same data mixture, model shape, tokenizer, and token budget. They vary the training seed and data-loading configuration.
For each task t, compute the proportional mean and sample standard deviation in BPB:
mu_t = mean(BPB[p, t] for p in P)
sigma_t = sample_stddev(BPB[p, t] for p in P)
The current metric uses the raw sample standard deviation from the ten runs. We have not yet selected a shrinkage estimator or variance floor. This choice must be fixed before using the score as a stable optimization target because ten samples give imprecise variance estimates.
For a candidate mixture x, average BPB across its replicate seeds before scoring. Most current candidates have one seed.
z[x, t] = (mean_candidate_BPB[x, t] - mu_t) / sigma_t
BPB is lower-is-better, so negative z values are improvements. The denominator is the proportional single-seed standard deviation. Do not divide by the standard error. Replicate count changes uncertainty in the score; it does not change the unit of effect size.
The target tasks are grouped by the Artificial Analysis capability they approximate:
# Coding
logprob_humaneval_10shot
# Scientific and mathematical reasoning
logprob_gsm8k_5shot
arc_challenge_0shot
openbookqa_0shot
sciq_0shot
# General reasoning and knowledge
mmlu_pro_5shot
lb_bbh_3shot
musr_0shot
truthfulqa_mc1_0shot
The available BPB suite has no credible proxy for the agentic category, which carries 34% of the current Artificial Analysis index.
Tasks with proportional relative-BPB sample standard deviation above 10% are excluded from the target set. They remain guardrails:
| Task | Proportional seed SD |
|---|---|
| CSQA | 32.13% |
| GPQA | 21.72% |
| MedQA | 16.49% |
| MedMCQA | 15.93% |
| BoolQ | 15.02% |
MuSR is retained provisionally at 9.27%, immediately below the cutoff. GPQA is the only direct overlap with the current Artificial Analysis index, but its measured BPB is too noisy to receive symmetric target credit from a single-seed result.
The remaining guardrail tasks are:
boolq_0shot
copa_0shot
csqa_0shot
gpqa_0shot
hellaswag_0shot
lambada_0shot
medqa_0shot
medmcqa_0shot
piqa_0shot
winogrande_0shot
include_mean
belebele_mean
include_mean and belebele_mean are the arithmetic means of the BPB values for their constituent evaluation tasks.
With a guardrail tolerance of epsilon = 1, the objective is:
target[x] = sum(z[x, t] for t in target_tasks)
guardrail[x] = sum(max(0, z[x, t] - 1) for t in guardrail_tasks)
objective[x] = target[x] + guardrail[x]
All tasks have equal weight. The exact proportional reference vector has z = 0 for every task and an objective of 0.
The hinge has these consequences:
- A target improvement of
1 sigmacontributes-1. - A target regression of
1 sigmacontributes+1. - A guardrail improvement receives no credit.
- A guardrail regression of at most
1 sigmareceives no penalty. - A guardrail regression of
k sigma, fork > 1, contributesk - 1.
The objective's unit is a sum of proportional-seed standard-deviation effect sizes. It is not a BPB percentage or an average across tasks.
Standardization keeps the tasks in comparable seed-variation units. The 10% threshold keeps unstable tasks from receiving target credit while preserving guardrail penalties for regressions beyond 1 sigma. Neither step distinguishes a real candidate effect from a lucky candidate seed.
Estimate candidate uncertainty from whole proportional-seed residual vectors. Keep each seed's task vector intact so correlations between tasks are preserved. Apply the hinge after drawing a complete latent task vector because the hinge is nonlinear.
For a candidate averaged over m independent seeds and a proportional reference averaged over n = 10 seeds, the approximate covariance of the standardized observed difference is:
(1 / m + 1 / n) * R
where R is the task correlation matrix estimated from the proportional runs. This approximation assumes candidate mixtures have the same seed-noise distribution as proportional. Shared training prefixes and checkpoints can create additional correlations.
The Bayesian optimizer should receive both the objective estimate and its estimated observation variance. Ranking single-seed point estimates without this uncertainty will favor lucky runs, especially after searching many mixtures.
Using the unregularized ten-seed sample standard deviations and epsilon = 1, the five best non-proportional runs are:
| Rank | Mixture | Target | Guardrail | Objective |
|---|---|---|---|---|
| 1 | Joint broad Q1 finish | -1.89 | +1.32 | -0.56 |
| 2 | Q1 cooldown | +0.71 | +0.00 | +0.71 |
| 3 | N1: Q2 to Q1 | +1.29 | +0.00 | +1.29 |
| 4 | Q2 cooldown | +0.47 | +1.88 | +2.35 |
| 5 | E2D: Q3 to Q1 | +3.15 | +1.09 | +4.23 |
Individual proportional seeds are excluded from this ranking. They are samples from the reference noise distribution, not distinct data mixtures. The candidate values above are point estimates from single seeds and do not establish that any candidate beats proportional.
Before treating this as a fixed production objective, choose and record:
- A shrinkage estimator or floor for
sigma_t. - The posterior or bootstrap procedure used to produce objective mean and variance.
- How many confirmation seeds are required before promoting a candidate.
- Whether equal task weights match the intended benchmark priorities.