Baseline V2 (test-failure attribution — off by default)

What it is

Every red test raises the same question: is this mine, or was it already broken? Omniscio's original answer reads a large shared snapshot of the main branch — fast, but a snapshot ages, and it mixes revisions and environments together.

Baseline V2 is designed to answer the same question a different way: it takes the scope that actually failed and runs it again in the cloud, on exactly the commit your branch started from, under the same environment and settings as the run it is explaining. The answer would then be about your inputs specifically, not about a picture of the main branch taken at some earlier moment. That comparison is not finished being wired up — see How it behaves for what it does today.

It is deliberately a separate system, sitting beside the one you rely on today rather than replacing it. That is what lets it ship off, be judged on real branches, and be switched back off without risk.

Where to find it

In the Dev Pipeline panel, on the Setup tab's All settings screen — one click in from the panel's first screen. Look for the card titled "How a test red is judged" and its switch "Use Baseline V2". That switch ships off.

Turning it on reveals a second switch, "Keep V2 evidence warm" — an optional background job that pre-fills evidence for your current main branch, so a comparison can answer from what was already measured instead of running a fresh replay. It ships off too, it does nothing at all while "Use Baseline V2" is off, and an answer still works without it — it is only slower.

Switching Baseline V2 on or off takes effect on the next run. Nothing restarts and nothing is migrated. The switch belongs to you: an agent session cannot flip it to change how its own work is judged.

How it behaves

With it off — the default — nothing changes at all. Your red is judged exactly as before, and Baseline V2 starts no cloud job, does no work on your computer, polls nothing and costs nothing.

With it on, a newly failing test run is pinned to V2, and V2 reports it unknown — it does not answer it yet. Two pieces of the comparison are still being finished: carrying the run's context on the end-to-end test road, and filing the reference attempt that a comparison measures against. So V2 records that the red is V2's to answer and then says it could not measure it. Every other kind of red — type checking, lint, the guard lane, end-to-end — is still judged the old way, whatever the switch says.

Once those two land, the replay returns one of four answers, and only one of them is charged to you:

• Already failing at the reference — the same failure was there before your work. Not yours.
• New in your branch — the reference is clean and yours is not. This is the only answer that counts against you.
• Unstable — it fails in one arrangement and passes in another. Recorded as something to diagnose, never used as an excuse to clear your branch.
• Unknown — it could not be measured.

Unknown is never a pass and never a charge. That is the answer V2 gives today, and it is also the answer whenever the cloud is unreachable, the replay cannot be dispatched, or the evidence is missing or incomplete: it does not block you, it does not clear the red, and it does not get billed to your name. A test that was skipped is recorded as skipped, never as passed.

Two limits worth knowing before you rely on it:

• It needs the cloud. It never falls back to running the tests on your own computer, so an outage produces unknown rather than a quieter local run.
• It answers for tests only. Nothing else about your branch's checks changes.

Turning it back off restores the original behaviour immediately — no migration, no cleanup step, no restart, and the evidence V2 already gathered stays readable. A run that was already in flight when you flipped it finishes what it started and begins nothing new; anything it cannot finish is reported as unknown.

For agents

The switch is the persisted setting baselineV2Enabled, and the warmer's is baselineV2WarmerEnabled. Gate scripts read the first one from the running app with GET /settings/baselineV2Enabled at a run's first red and keep that answer for the run; only an explicit true routes to V2, so a closed app, a timeout or a malformed reply all mean the original path. No environment variable ever turns it on.

Only the tests check is attributed by V2 — the same check name every producer already uses. The replay reports a failure as already-failing-at-reference, new-in-subject, unstable or unknown, and only new-in-subject is chargeable. Until the run context and the reference attempt land, an attributed red resolves to unknown, which is never chargeable. Each observation is filed under a key naming the repository, the source content, the environment, the toolchain, the selected tests and the execution mode, and is kept as an unchangeable attempt rather than overwriting the one before it.

Related

Cloud test offload and builds explains where heavy checks actually run and how the machines that run them are managed. Dev Pipeline panel is the panel these switches live in, and covers the runs, worktrees and gates around them. Dev Pipeline is the workflow itself, phase by phase.