/ POST-TRAINING SYSTEMS
/ THE CAPABILITY PROBLEM
A task is not useful merely because a strong model fails it. Failure can indicate missing capability, ambiguous instructions, verifier error, or an impossible instance.
MY TECHNICAL VIEW
A benchmark gap is not automatically a capability gap; verifier correctness is not automatically reward quality; rollout success is not automatically capability acquisition; and benchmark improvement is not automatically transfer.
/ WHERE SYSTEMS FAIL
01
PROBLEM
Solved tasks keep consuming rollout budget while static labels stop reflecting what the current policy can learn.
TECHNICAL VIEW
Difficulty has to be treated as an empirical property of the policy-task pair, not a permanent attribute of the task.
PRACTICE
The operational question is how to keep recalibration useful without turning every training cycle into a bespoke data project.
02
PROBLEM
A model may fail because the intended capability is missing—or because an instruction is ambiguous, an instance is impossible, or the checker is wrong.
TECHNICAL VIEW
Useful challenge has to be separated from accidental unsolvability before failure becomes training signal.
PRACTICE
Generation, validation, rollout evidence, and verifier behavior have to be inspected as one system rather than independent stages.
03
PROBLEM
A policy can improve against the measurement while the intended capability remains unchanged.
TECHNICAL VIEW
The evaluator is part of the intervention. Its blind spots and incentives belong in the research hypothesis, not only in test infrastructure.
PRACTICE
A credible conclusion needs more than a target metric: it needs transfer checks, broader regressions, efficiency, and reproducibility.
Much of my current work concerns systems built around this class of problem; implementation details and internal results are not public.
/ OPERATIONAL SEMANTICS
SCENARIO 01 · RECOVERY
A recovered rollout is not necessarily equivalent to an uninterrupted rollout. Retries, partial state, filtering, and timeout behavior can silently change the sampled distribution.
If only clean retries survive, the final dataset may underrepresent precisely the long-horizon or tool-heavy behaviors that fail most often. Recovery policy, acceptance filters, and experiment comparability therefore belong in the analysis—not only the operations dashboard.
SCENARIO 02 · DIAGNOSIS
Before treating repeated failure as a capability gap, I want to know whether the task was valid, the reference was reachable, the verifier measured the intended outcome, and the runtime preserved the experiment.
A cluster of failures can justify a curriculum change only after separating intended challenge from broken references, underspecified tasks, checker disagreement, and runtime loss. Otherwise the system may optimize against artifacts while reporting that the curriculum became harder.
/ TECHNICAL DECISIONS
Hard tasks ↔ Learnable tasks
Failure by a strong model is evidence to investigate, not sufficient proof of learning value.
Binary outcome ↔ Richer evidence
Objective outcomes are clean; intermediate evidence can add signal but also expand the attack surface.
More diversity ↔ Higher validity
Coverage helps only while generated tasks still measure the intended capability.
Rollout scale ↔ Diagnostic depth
Throughput discovers behavior; diagnosis determines whether that behavior supports the conclusion.
Automatic generation ↔ Designed structure
Automation increases reach, while deliberate structure preserves control over what is being trained.
Fast iteration ↔ Reproducible evidence
Research velocity matters only if failures, recoveries, and evaluation conditions remain attributable.
/ EVALUATE, THEN REFORMULATE
Did the intended capability change?
Did the gain survive a different distribution?
Did the evaluator reward the right behavior?
Did reliability or recovery alter the comparison?
What regressed outside the target slice?
What should the next task distribution test?
/ QUESTIONS I CARE ABOUT
01How can environments remain learnable as policies improve?
02How should capability acquisition be distinguished from evaluator overfitting?
03Can environments generate an increasingly useful curriculum without losing task validity?
04What makes long-horizon behavior trainable rather than merely testable?
05How should autonomous research systems trade off speed, verification, and experimental trust?