Reproducibility and Verifiability of Decisions in an AI-Based QA System

TL;DR

Breaking the skill into small decision nodes, each tied to a rule or data, made results more consistent. For story B, run-to-run scenario overlap went from 0.71 with QABuddy to 0.93–1.00.
It costs less: $0.61 per story on average with the Haiku runner, against $1.51 for QABuddy.
Every decision can be traced to the rule or data that produced it.
It is not faster, and priority and test-layer agreement are about the same as QABuddy on Opus 5.5.

The Questions the Old Skill Couldn’t Answer

If you asked QABuddy’s /qa-test-cases why a scenario was P0, the honest answer was: “The model judged it that way.”

The original skill operated through a single long prompt. It read tickets, references, and learning materials, then generated scenarios, test layers, severity, and priority all at once. Decisions were made somewhere along the way. But there was no record on disk of which rules led to those decisions—or whether any rules existed in the first place.

The September evaluation exposed this problem. The rubric judge could score the output, but it could not explain why the output was right or wrong. The skill itself did not record what it had decided or why. Running the same story twice could also produce different plans.

So the follow-up started by changing the question.

Instead of asking, “How can we write a better prompt?”, we asked, “What decisions exist inside this skill, and what evidence should support each one?”

And we added one more question:

Could breaking decisions down this way reduce cost without sacrificing quality?

Step 1: Break the Skill Down into Decision Units

Every skill in the follow-up starts with the /breakdown execution mode. This is the process of breaking a skill into small decision nodes.

Each node handles one question with a defined scope of possible answers. Each node also uses exactly one type of evidence.

Data: Read the answer from information that actually exists. Examples include AC text, feature lists, test annotations in the repository, and AUTOMATION.md.
Playbook: Find the answer in rules that have IDs. For example, REF-automatability#automatability-conditions.
Judgment (owner: QA): If the answer cannot be determined from data and rules alone, a person makes the decision at a checkpoint.

/test-cases was divided into nine nodes.

N3: Write the scenarios required for an AC.
N3R: Suggest risks not covered by the AC.
N4: Check whether existing tests cover the scenario.
N5: Determine whether it can be automated.
N6: Choose the lowest test layer that can test the scenario.
N7: Classify what would break if the scenario failed. The code calculates severity based on this classification.
N8: Check whether the preconditions can be established.

In between, QA reviews the work at three checkpoints: risk suggestions, automation gaps, and the final plan.

Each node is also assigned an executor.

Code handles everything that can be calculated. A Haiku-class model with no tools and no history handles narrowly scoped questions. It receives only one prompt, one input, and one playbook excerpt. Sonnet is used only where text needs to be written. Currently, those are N3 and N3R, the risk-suggestion node.

The main model acts only as the runner. It calls the helper CLI, emits node executions, and pauses at checkpoints.

Step 2: Rules Are Checklists, Not Balancing Acts

The playbook has two standing rules.

First, rules are written as checklists.

If a decision involves several considerations, list each one as an independent yes/no condition and define the default when any condition fails.

Automatability is one example. It has five conditions, C1 through C5.

Does the outcome manifest as a testable behavior?
Is it machine-readable?
Is it about behavior rather than a quality criterion?
Can the test establish the preconditions itself?
Can all external systems be accessed?

The model answers each condition separately. Code combines the answers. If all are yes, the scenario is classified as automatable. The first no becomes the basis for classifying it as manual.

Second, Haiku is the default model.

If Haiku cannot answer a particular rule consistently, we do not look for a larger model. We treat the rule as too ambiguous and break the existing rule into smaller parts.

The severity decision shows why this principle is necessary.

Initially, we asked the model to assign severity directly. When compared against 11 scenarios labeled by QA, its accuracy was about 20%.

We redesigned the node and changed the question:

“What happens if this scenario fails?”

The answer must be one of four categories:

Invalid input is accepted.
Existing data is changed.
The primary outcome of the feature does not occur.
Only a side effect fails.

The code uses this classification together with the feature’s tier to look up severity in a predefined table.

Against the same labels, accuracy increased to 95%, and repeat-run self-consistency was also 95%.

Instead of having the model weigh severity directly, we changed its task to classifying the outcome into one of four categories.

Each scenario now records the reasoning behind its severity as well.

For example, core/no × side, 3/3 votes contains four pieces of information:

core: The feature is classified as core.
no: There is no workaround if the feature fails.
side: If this scenario fails, only a side effect breaks.
3/3 votes: All three votes returned the same answer.

The code looks up a row in the severity table using the first three values. This means we can point not only to the final severity, but also to the exact row in the table that produced it.

Step 3: The Engine Keeps the Model Honest

Akela, the context engine separated from QABuddy in August, compiles the playbook at the node level rather than at the skill level.

Each node receives only the rules it needs to make its decision. Its answer must cite rule IDs, and the runner checks whether each cited ID exists in the rule excerpt provided to that node. Fabricated IDs are rejected. This mechanism caught an invalid ID in one simulation.

Small decision nodes also cost fewer tokens.

Calling a node through a dedicated agent without tools costs about 2.1k tokens. The same call through a general-purpose agent used about 52k tokens for the system prompt alone.

QA corrections are recorded, too.

When QA changes a severity at a checkpoint, Akela records the revised judgment as a decision case while preserving the model’s original answer.

/learn drafts a project rule only when there are at least two decision cases pointing in the same direction. It then checks that the draft does not conflict with any recorded decisions before asking QA to approve it.

The playbook does not change just because the model notices something. It changes only after decision cases accumulate, conflicts with existing records are checked, and QA approves the change.

Step 4: Compare QABuddy and qa-skills-x

We ran consistency tests.

We selected three stories from one epic. Each tool was run independently three times, and an Opus judge grouped scenarios that tested the same behavior.

The baseline was QABuddy’s /qa-test-cases. Each run started from a fresh copy of the repository. This meant that all runs of the same story started from the same repository state.

Scenario-set overlap is the average Jaccard similarity across the three pairs formed by pairing the three runs. Jaccard similarity is the number of scenario groups common to both runs divided by the number of scenario groups that appeared in either run. A score of 1.00 means all three pairs had identical lists.

Story	Description	Existing tests in the repository at the start	What the result helps show
A	Room details and guest reservations	14	How existing tests affect each tool’s output
B	Admin login	None	A direct comparison of the two tools. It is also the smallest of the three stories
C	Admin room management	3	A story with a small amount of existing test coverage

The existing tests had only their old TC numbers. QABuddy could read those tests and classify them as existing coverage. The follow-up, on the other hand, recognizes only tests with scenario ID annotations.

As a result, in the initial experiment, the two tools had different starting states for stories A and C. For the final comparison, we added scenario IDs to the existing tests so that the starting states would match.

The first results fell short of expectations.

The follow-up’s scenario sets had relatively good overlap, but its run-to-run agreement on priority levels was no better than QABuddy’s. I did not expect a node responsible only for classification—the severity node—to be that inconsistent.

When I reran N7 with fixed inputs, the cause became clear. For the same input, only 86% of repeated runs returned the same priority level across all three repetitions. Priority was being affected by Haiku’s unstable judgments.

N7 classifies what breaks if a scenario fails into one of four categories. Code uses that classification to look up severity in a table.

We made three changes and checked the effect of each before adopting it.

First, vote three times.

N7 asks Haiku the question three times. Code selects the severity by majority vote. If the votes disagree, it does not guess; it sends the question to QA.

Second, fix the form of the scenario, not the model.

Some of the remaining inconsistencies came from writing the same validation in different forms across runs.

One run said, “Firstname allows 3 characters,” while another said, “Firstname length limit is 3 and 18.” N7 then classified the two scenarios differently.

The scenario rules now define what a boundary scenario is meant to verify. The boundary value is accepted, while the value immediately outside it is rejected.

We also added a short table to the severity rules covering patterns that had repeatedly produced inconsistent judgments. Examples include an error displayed beneath a rejected field, a confirmation message, and a record appearing in the user's own list.

Third, validate changes against a different story before adopting them.

Every playbook change was rerun against a QA-approved plan from another story.

The initial wording, “A change visible on the second page is a side effect,” incorrectly classified a branding change as Normal. Changing one phrase fixed the problem.

“The change to that record is also visible on other pages.”

The final wording matched the approved plan in all 30 holdout scenarios. Before the change, it matched 29 out of 30.

Every time we adopted a change, we ran the same three stories three more times.

Metric	QABuddy (Opus 5)	Follow-up, initial	After adding N7 voting	After adding scenario-form rules	After fixing test-layer combination
Scenario-set overlap (A / B / C)	0.75 / 0.70 / 0.59	0.68 / 0.73 / 0.87	0.79 / 0.77 / 0.84	0.86 / 0.81 / 0.92	0.85 / 0.73 / 0.92
Priority-level agreement for the same scenarios	73%	69%	79%	93%	93%
Test-layer agreement for the same scenarios	80%	93%	93%	80%	93%

In the fourth column, test-layer agreement dropped to 80%.

At first, I thought N6 was unstable, so I made it vote three times. But the agreement rate did not change.

The cause was how the code combined the stored answers. The scenario was executed in the browser, but one vote answered “undetermined” to the condition, “Can the result only be checked after it has been saved?” Code used that single answer to mark the entire scenario as “undetermined.”

But the rule was to select the highest test layer among those whose conditions were satisfied.

If all votes answered yes to “Is the result visible in the browser?”, the scenario is e2e. Even if one condition for the lower API layer is “undetermined,” the final answer should remain e2e.

After fixing the code to follow the rule, agreement returned to 93%. Since asking only once also produced 93%, N6 was changed back to a single question.

The last column is a newly executed second batch. Therefore, the scenario overlap for story B includes run-to-run variation.

What Did We Gain Compared with QABuddy?

All the earlier measurements ran QABuddy on Opus 5. The CLI we were using at the time did not support Opus 5.5.

After updating the CLI, we upgraded all models to 5.5 and measured again.

QABuddy ran on Opus 5.5. For the follow-up, we used two different runners. The runner is the main model that orchestrates the helpers. The nodes remained unchanged. Haiku 5.5 handled decisions, and Sonnet 5.5 handled writing.

This time, we added scenario ID annotations to the existing tests for stories A and C. The follow-up could now recognize those tests, so both tools started from the same state. QABuddy reads existing tests regardless of their IDs, so we did not run it again.

Metric	QABuddy (Opus 5.5)	Follow-up (Sonnet runner)	Follow-up (Haiku runner)
Scenario-set overlap (A / B / C)	0.36 / 0.71 / 0.54	0.86 / 1.00 / 0.93	0.91 / 0.93 / 0.76
Priority-level agreement for the same scenarios	88%	96%	93%
Test-layer agreement for the same scenarios	98%	93%	93%

The new model alone made QABuddy’s priority and test-layer decisions much more stable. Priority-level agreement increased from 73% to 88%, and test-layer agreement increased from 80% to 98%.

However, the consistency of the scenario lists did not improve. When QABuddy ran the same story three times, it produced noticeably different lists each time. Its run-to-run overlap ranged from 0.36 to 0.71.

The follow-up ranged from 0.76 to 1.00. It applies the same rules to the same AC and uses the same node to write the scenarios.

Story B, which had no existing tests, is the most suitable for a direct comparison. QABuddy scored 0.71, while the follow-up scored 1.00 and 0.93 depending on the runner.

The runner model had almost no effect on quality.

Titles, steps, and expected results are written by N3. N3 runs on Sonnet regardless of which runner is used. The remaining inconsistencies with the Haiku runner came from node answers. Examples include an “undetermined” response to one test-layer condition or a severity judgment on the boundary.

In story C, one run’s N3 wrote, “When Room # or Type is missing,” while another wrote, “When both Room # and Type are missing.”

There were no inconsistencies that could be traced back to the runner.

Changes That the Numbers Don’t Show

Three changes are not fully reflected in the measurements.

Every decision has an address.

Whether it is a test layer, severity, or a “blocked” status, each decision is associated with the name of the rule or data that produced it. QABuddy could tell us what it decided. The follow-up records decisions in a form that lets us verify why they were made.

A smaller model makes the decisions.

QABuddy’s quality depends on the model. Switching from Opus 5 to Opus 5.5 made a bigger difference than anything else we did.

In the follow-up, Haiku handles all classification, while Sonnet only writes. When decisions were unstable, we improved the rules rather than moving to a larger model.

New scenarios do not depend on documents.

QABuddy created new test cases using a knowledge base built from plans and test-case documents.

The follow-up creates them from the feature list and tests annotated with scenario IDs. Coverage is calculated based on those annotations. Teams that need plans and test-case documents can continue to maintain them.

Cost per Run

Intuitively, one Opus call seems like it should cost less than a hundred small calls.

But that is not what happens in practice. One QABuddy /qa-test-cases run is an Opus session consisting of 18–29 turns. Each turn rereads a context that grows over time.

The follow-up makes 23–37 node calls per story. Each call, however, reads only one short prompt.

We measured the cost by running each of the same three stories three times in headless mode.

These costs are not from actual invoices; they are the values reported by the CLI for each run. The CLI calculates cost by multiplying token counts by the API list price for each model. The costs for nodes and the runner were separated using the model-level breakdown in the same report.

Metric	QABuddy (Opus 5.5)	Follow-up (Sonnet runner)	Follow-up (Haiku runner)
Cost per story	$1.31–$1.70 (average $1.51)	$1.49–$2.61 (average $1.92)	$0.49–$0.94 (average $0.61)
Time per story	3.7–8.7 minutes	6.5–11.2 minutes	4.8–7.3 minutes

When QABuddy used Opus 5, it cost $4.20 per story. At the time, the follow-up cost about one-third as much.

After switching to Opus 5.5, QABuddy’s cost dropped by about two-thirds. As a result, the follow-up using the Sonnet runner became more expensive.

But the decisions themselves cost very little. All Haiku 5.5 node calls combined cost only about one to two cents per story. About 99% of the total cost came from the Sonnet runner. That is the cost of reading helper outputs, dispatching work, and writing the report.

So we also tested the runner model in both directions.

When we used an Opus 5.5 runner and assigned scenario writing to Opus as well, the cost was $3.57 per story. But run-to-run agreement was no better than with the Sonnet runner.

The Haiku runner averaged $0.61, about 40% of QABuddy’s cost. Scenario-set overlap ranged from 0.76 to 0.93.

The runner’s work is mechanical. It therefore makes more sense to use the least expensive model that can perform that work reliably.

We keep writing with Sonnet. Scenario writing costs about 10 cents per story.

What the Numbers Don’t Tell Us

The two tools ran with different model combinations. QABuddy used Opus 5.5, while the follow-up used Haiku 5.5 and Sonnet 5.5. In the first table, QABuddy ran on Opus 5.

Three runs per story are a small sample. The judge model itself also has a margin of variation of about ±0.05.

QABuddy’s output format is not consistent across runs. In one run of story A, the existing-test references appeared in a form that the extractor missed. This lowered the overlap for story A. When comparing only new scenarios, story A scored 0.43 and story C scored 0.71.

Existing tests are only part of the reason QABuddy is inconsistent on stories A and C. Even when comparing only new scenarios, the follow-up performs better. The follow-up does not read existing tests, yet its overlap varies by story. This suggests that story size also has an effect.

The scenario-form rules were derived from QA-approved plans. We therefore also checked them against other stories to which those rules had not been applied.

What I Learned

Voting reduces noise, but it does not resolve ambiguity.

Three votes reduced the problem of Haiku’s unstable judgments affecting priority. But they had no effect when two runs expressed the same validation in different forms. That problem could only be solved by fixing the rules.

Before increasing the number of votes, look at the answers you already have.

The test-layer disagreement looked like noise, but the actual cause was how the code combined the answers. Because the answers for each condition had been stored, a single query was enough to find the cause.

A single line in the playbook should be validated like code.

One missing phrase in an example changed the actual severity. We found the problem only after rerunning the rule against decisions QA had already made.

Use a model suited to mechanical work for the runner.

Using a better model for the runner did not improve the results. The Haiku runner reduced costs by more than two-thirds while producing the same scenarios. That was because the model writing the scenarios never changed.

And now I can answer the original question.

If you ask the follow-up why a scenario is P1, it can now point to a row in the table.

Conclusion

The initial hypothesis was this:

If we break a skill down into a mesh of small decisions and connect each decision to playbook rules or data as evidence, we can get more consistent results at a lower cost.

The hypothesis held up in terms of consistency and cost.

The same scenarios appear again.

For story B, which had no existing tests, QABuddy’s run-to-run overlap was 0.71. The follow-up scored between 0.93 and 1.00.

The cost is lower.

With the Haiku runner, the average cost is $0.61 per story. QABuddy costs $1.51.

The evidence behind decisions can be identified.

Every decision is associated with the name of the rule or data that produced it. Even when a decision is inconsistent, we can follow the record to identify the cause.

However, we cannot claim that execution speed or decision agreement improved across the board.

It is not faster.

Execution time per story was 3.7–8.7 minutes for QABuddy and 4.8–7.3 minutes for the follow-up using the Haiku runner.

Priority and test-layer agreement are similar.

QABuddy reached almost the same level with Opus 5.5. The difference is that the follow-up makes those decisions with Haiku.