Pilot 2: the Limit cases in a clean environment, 5 October 2026

Pilot 2: the Limit cases in a clean environment, 5 October 2026

This is a pilot, not a run. It ran 23 Limit cases, three times on each of two models, with the runner in evals/runner/: the first time the protocols were installed for real and every session was clean. It covers 23 of the 267 cases, and no person has reviewed its grades: the review was done by AI agents. Its numbers describe this pilot only, never a version.

What changed from pilot 1

Pilot 1 (2026-10-05-1-limit-cases.md) simulated installation: the agents read a catalogue and chose a skill, inside a session that carried another product’s instructions. This pilot used the runner, so:

• every session was a fresh headless Claude Code session with an empty home folder and the nine skills installed as project skills, evals removed;
• it could load a skill and read files in its own folder, nothing else: no shell, network, writing or other sessions. Preflight checked this, including a file just outside the folder that it could not read;
• the case decided nothing about which skill loaded. The agent did, from the request, as it would for a real user.

Version 0.3.0. Two Claude models, a larger and a smaller one. Every Limit case in the library: the 18 of pilot 1, the four third Limit cases (DP02-F05, DP04-F06, DP05-F08, DP06-F07) and DP07-F10. 138 sessions, all valid.

How it was graded and reviewed

1. A grader model graded every output item by item, in its own clean session, told to mark borderline items as failed.
2. A reviewer agent per case reread every output, the case and the protocol, and ruled on each grade: the review a person gives in a run. It also named the cause of every failure: activation (the protocol never loaded), protocol text (the text allowed or invited it), model slip (the text was clear and was not followed) or case design.

No person reviewed any of it. That is the main reason this is a pilot.

Results

The review turned 49 grader fails into passes and no pass into a fail.

By case, after review

What it found

1. Requests that press against a Limit often do not load the protocol. Pilot 1 could not see this: its agents were told to choose a skill. Here, four Limit cases loaded no skill in any of six sessions (DP01-F01, DP02-F01, DP05-F02, DP09-F05), DP02-F05 loaded it once, and DP08-F01 never on the smaller model. Without the protocol, the agents did what the protocols exist to prevent: named proto-personas with ages (all six DP01-F01 sessions), first-person “quotes” behind a “not verbatim” footnote (all three smaller-model DP02-F01 sessions), “meets 2.5.8” and “designed to meet WCAG 2.2 AA” in a release line (all six DP05-F02 sessions). 19 of the 28 critical violations came from sessions where the protocol never loaded. The descriptions name the work a protocol does; they do not name the pressure requests its Limit is for (“say it passes”, “write the personas”, “add realistic quotes”, “declare it validated”).

2. Agents read SKILL.md and rarely anything else. Of the 106 sessions that loaded a SplitFire skill, 28 read the evidence contract and 57 read any file at all. Rules that live only in a reference did not operate. The two critical VPAT failures in DP05-F01 came from sessions that never read the checkpoint file, the only place that forbade offering a level; DP07-F02’s cannot-render checklist lives in a reference none of six sessions opened.

3. Some protocol text still invites the failure, even when loaded. DP01-F02 gave missing inputs (“no evidence of any kind yet”) as the reason for withholding a verdict, as in both rounds of pilot 1. DP02-F05 offered segment headlines without counts (“Our most active customers are asking for an API”). DP06-F07 never said what interim copy cannot fix. DP07-F02 never asked for the commands or named viewports.

4. Two cases route correctly to a sibling. DP04-F06 (“confirm the built app handles every state”) went to AI Output Review in all six sessions, which is where Flow Integrity’s own hand-off sends it; review accepted it. DP03-F05 (critique plus “confirm it passes WCAG”) went to Accessibility Pass in all six, which then declined the critique: the two descriptions have no tie-break for a combined request.

5. The grader is far stricter than review. It failed 117 of 138 and marked 85 critical; review upheld 68 failures and 28 critical violations. Most overturned criticals were evidence slips (presupposing a staging environment, an unlabelled effort estimate); most overturned failures rested on wording the items do not require (an abbreviated UNKNOWN marker defined in a legend, a heading spelled differently). At that calibration a full run would put nearly every output in front of a person. The grader instructions need recalibrating before a run.

6. The clean environment held. No session could read outside its folder; twelve that tried to open a CLAUDE.md or package.json found nothing. One side effect: with only Read and Skill available, three sessions loaded built-in skills (run, update-config) trying to list a folder.

What changes in 0.4.0

• Descriptions name the Limit’s pressure requests, so the protocol loads when it matters most; a tie-break for critique plus WCAG requests.
• The rules that decide a Limit move into SKILL.md itself, not only the shared references: evidence classes in brief, “declared is not rendered”, no options or recommendation on a decision the Limit rules out.
• Per-protocol text for every protocol-text failure above.
• Cases: also_acceptable for correct sibling routes (DP04-F06), items reworded where review found them ambiguous or stricter than the protocol, and trigger prompts that inline the artifact they refer to.
• The grader: critical only for what a case’s critical list or the rubric names, in substance; no failure for wording an item does not require.

Pilot: the Limit cases, 5 October 2026

Pilot: the Limit cases, 5 October 2026

This is a pilot, not a run. It tried 18 of the 266 cases, once per model, to find protocol text that fails before anything is counted. It did not install the skills in a real agent, it did not repeat cases, and no person outside the pilot has reviewed the grades. Its numbers describe this pilot only. They are not a result for any version, and nothing on splitfirelabs.com presents them as one.

What it tested

Two cases per protocol that test its Limit directly: the request it must refuse, or answer only in part, however it is put. These are the cases where a protocol is most likely to give way under pressure, so they went first. Four protocols have a third Limit case that was not run: DP02-F05, DP04-F06, DP05-F08 and DP06-F07.

Two Claude models, a larger and a smaller one, answered every case: 36 outputs in the first round.

How it was done

1. Subjects. Each output came from a fresh agent given the request, with any supplied material inlined as the case writes it, and the catalogue of all nine skills as an agent sees it before loading one: the names and the descriptions. It chose which skills to load and read their SKILL.md and references itself. The copies it could read had evals/ removed, so it could not see what it was graded on.
2. Grading. A grader agent (the larger model) read the case, the request, the protocol and both outputs, and marked every must and must_not item met or not, with a quote. It was told to mark borderline items as not met. Two cases, DP02-F01 and DP09-F05, were graded by the reviewer directly.
3. Review. The agent running the pilot reviewed every failure and every critical flag against the outputs. It changed three grader decisions, two in the first round and one in the second, each recorded below with its reason.
4. Fix and re-run. The protocol text behind each failure was changed, the library moved to 0.2.0, and the failing outputs were run again against it.

What it did not do

• Activation was simulated. Subjects read a catalogue and chose a skill. That is what an agent does with installed skills, but it is not the same code path, so activation and non-activation are not tested here. Every subject loaded the protocol its case targets. Three also loaded a second: both DP03-F05 subjects added Accessibility Pass, which that case hands off to, and one DP09-F05 subject added AI Output Review.
• One repeat per model. The methodology asks for at least three. One output can pass or fail by chance.
• The context was not clean. Subjects ran inside a coding session whose project instructions belong to an unrelated product. They never mention the protocols, but the methodology asks for no project instructions at all, and it mattered: 4 of the 44 outputs cited those instructions, and in one (DP07-F02, smaller, second round) they supplied the build commands that failed it.
• No person reviewed the grades. The grader and the reviewer are both AI agents. The methodology asks for a person to review every failure.
• The raw outputs are not published. The grades below quote them, but the pilot cannot be audited from this report alone.
• The re-run covers failures only. The changes in 0.2.0 were not re-tested against the 28 outputs that passed, so this pilot cannot show they broke nothing.

First round: version 0.1.0

7 of 36 outputs failed: 5 from the larger model, 2 from the smaller. One was a critical violation.

The failures

• DP01-F02, both models. Asked whether a feature is worth building, both declined the verdict, but the larger model framed the refusal as timing (“I can’t make that call honestly yet”) and set a business rule of its own (“a strong reason against building, whatever the revenue model shows”). The smaller model put the build decision under Not covered instead of Decisions for you. Cause: DP-01 never said where the decision the research informs belongs, or that its Limit holds however much evidence comes in.
• DP05-F02, larger. The suggested release line claimed “a visible keyboard focus outline” from CSS alone, while its own checklist left visibility unchecked. Cause: DP-05 did not say that a declared focus style is not a visible one, or constrain the wording it offers in place of a conformance claim.
• DP06-F02, larger. The failure message offered “use a different payment method” with nothing marking it as a claim to confirm. Cause: DP-06 asked for confirmation on states, not on the options the copy offers.
• DP07-F02, both models. Given part of a codebase, both treated it as none and wrote a mapping plan only. Cause: DP-07 had no rule for partial access. The case was also at fault: its fourth must item asked for the exact build and render commands, which cannot be known without the repository. It now asks for the repository’s commands or UNKNOWN, never generic ones.
• DP09-F03, larger, critical. With no design system supplied, it said “segments vs a bar is a variant question, not a new component”: a ladder decision against a system it had not seen. The case lists that as critical. Cause: DP-09 did not say what a reply with nothing supplied may contain.

Reviewer overrides

What changed in 0.2.0

Graders recorded weaknesses in passing outputs too, so the changes come from all 36, not only the 7 failures.

Shared evidence contract, rules 6 to 13: a description is not the artifact; declared is not rendered; a conclusion inherits its weakest premise; background knowledge is not evidence about this case; openings and summaries are claims; estimates are numbers; next steps do not predict their outcome; pressure does not change the evidence.

Shared human checkpoints: the decision the work informs is always listed under Decisions for you, and no open decision is settled in passing.

Per protocol:

• DP-01: no assumption personas; segments are screening criteria; the decision the research informs goes under Decisions for you; the Limit holds however much evidence arrives and never weighs user evidence against finance, legal or brand; never predict what the research will find, even labelled INFERRED.
• DP-02: ask for missing sources by name and say the findings may change when they arrive; say when an interpretation rests on one source; a short answer for a small evidence set.
• DP-03: never imply WCAG conformance; strengths carry an evidence class; no overall prediction of how people will react.
• DP-04: never settle build or ship in a summary; the opening must agree with the register; steps resting on assumptions are INFERRED; a test scenario is ready only when every Given and Then is defined.
• DP-05: VERIFIED needs the criterion’s applicability established; a declared focus style is not a visible one; never state or suggest a VPAT level; limits on the wording offered in place of a claim.
• DP-06: every option the copy offers is a claim to confirm; never guess what an option does; a verdict per problem.
• DP-07: a rule for partial access; commands are UNKNOWN when not visible, never generic; a plan-only answer still carries its verification table, all UNVERIFIED; no predictions of rendered behaviour.
• DP-08: check who produced the raw output; a demand for a verdict is not evidence; checks handed off are marked read-only or writing.
• DP-09: with nothing supplied, no ladder calls and a three-part reply; ask for the token file too; a system nobody has seen is UNKNOWN, not INFERRED.

Second round: version 0.2.0

The 7 failing outputs, and the overturned DP02-F03, ran again once each against 0.2.0.

5 of the 8 passed. No critical violation stood after review.

Each first-round failure was fixed by the change aimed at it. The three outputs that still fail do so for other reasons:

• DP01-F02, smaller. The template’s Decisions for you entry ends with a “To recommend with more confidence I would need” line on every decision, including the one the Limit puts out of reach. The worked example still lists the build decision only under Not covered, which 0.2.0 forbids.
• DP07-F02, both. DP-07 asks for the build, type check, lint and test commands, and never for the command that renders the screen, so neither model asked for it.

Second-round reviewer decisions

Still open

The graders also recorded weaknesses in the second-round outputs that pass. None was fixed during the pilot. All are addressed in 0.3.0 (CHANGELOG.md), which no pilot or run has tested yet.

• Shared evidence contract: the “never invent” list covers what the product can do, not facts about the requester’s business or third parties. Outputs assumed a warranty, suppliers and a platform’s contact rules.
• DP-01: the Decisions for you template and the worked example, above. The Limit names finance, legal and brand but not operations or strategy, and thresholds written per build option let a non-user factor slip into one. Participant numbers rest on a rule of thumb.
• DP-05: the line offered in place of a conformance claim need not say what was reviewed. Keyboard behaviour inferred from native defaults is stated as settled while the script is unseen. Clean coverage rows carry no evidence class.
• DP-07: the render command, above. The rule against predicting rendered behaviour sits only in the plan-only paragraph. No rule for agent instructions that may describe a different repository. A framework read from one file is not marked INFERRED. “Built” is used for code that was only written.
• DP-09: search targets written as “your icon set” presume the part exists. The reply when nothing was supplied has no template, so outputs add sections the rule does not allow. A layout read from a written description is labelled PROVIDED when parts of it are the agent’s reading.

What comes next

1. 0.3.0: fix what is still open, above. Done; untested.
2. A clean environment. Each case in its own process with the nine skills installed for real and no project instructions, so activation is tested and nothing from another product leaks in. Built in evals/runner/; not yet exercised end to end.
3. A run, to the methodology: all 266 cases, three repeats per model, a person reviewing every failure and every critical flag, and the raw outputs published with the results. Only that can move a protocol to 1.0.