QAIA
Pre-alpha โ€” no human pilot has run it end to end yet

The whole thing, without installing anything

Six steps of one real run, with the file each one actually produced. Everything below is copied verbatim from examples/expense-demo/qaia-journey/, which is kept in the repository precisely so this page cannot be a mockup.

Disclosed before you read a single artifact. This run used our own gold-set ticket, which has ambiguities planted in it on purpose โ€” finding a planted ambiguity is easier than finding a wild one. It ran non-interactively: every point where the tool stops for a human was recorded as simulated: <default applied>, so nobody actually arbitrated anything here. And the acting model had read the file's sequestered judge section beforehand. All three are stated in the run's own journey file. What follows shows the shape of the output. It is not evidence that it works on your ticket.

1 Bring the story in

us-ingest โ†’ writes state/US-004/00-source.md

The source is captured verbatim, with the checks logged rather than assumed: redaction scan, control-character sanitisation, and an untrusted-input check โ€” because a requirement document is data, not instructions.

Produced
- **Redaction scan**: no direct personal/sensitive data found (no real names,
  SSNs, cards, health data, addresses). No masking applied.
- **Untrusted-input check**: the source text contains no directive aimed at the
  assistant โ€” it is a plain requirement description. Treated as data throughout.

> **As an** employee,
> **I want** to submit an expense report and have it approved through the right chain,
> **so that** I get reimbursed correctly and the company keeps an auditable trail.

2 Find what the story does not say

need-understanding โ†’ writes state/US-004/02-understanding.md

This is the step most generators skip, and the one that decides whether the suite is honest. Nine questions came out of eight acceptance criteria. Here is one, unedited:

Produced
Q1 โ€” AC2/AC6: threshold boundary at exactly โ‚ฌ500 and exactly โ‚ฌ5000.
Why it matters: routes every report; an off-by-one here silently under- or
over-approves.
A close literal reading is self-consistent ("under โ‚ฌ500" excludes 500 from band A;
the closed range "โ‚ฌ500โ€“โ‚ฌ5000" includes both ends; "above โ‚ฌ5000" excludes 5000 from
band C) โ€” so band B would be `[500, 5000]` inclusive on both ends. But this reading
depends on treating two different phrasings ("under X" vs "Xโ€“Y") as precisely
complementary, which is exactly the kind of boundary phrasing that real
implementations get wrong.
Why this matters more than it looks. Almost any generator will produce a test for "above โ‚ฌ500". The question is what it does with exactly โ‚ฌ500. Pick silently, and you ship a suite that looks complete and encodes a guess โ€” at the boundary, which is where the defects are. This step's output is a list of questions with the risk of each answer spelled out, addressed to a human.

3 Choose techniques, and say why

istqb-design โ†’ writes state/US-004/03-design.md

One line per acceptance criterion. The justification is the deliverable โ€” a technique named without a reason is decoration.

Produced
| AC2 | Boundary value analysis + Decision table | Three amount bands with two
  literal boundaries (โ‚ฌ500, โ‚ฌ5000) โ†’ BVA; the resulting required-role set per band
  is a decision table (amount band ร— role). |

| AC5 | Boundary value analysis | A strict amount threshold (โ‚ฌ25) on receipt
  requirement โ€” BVA at value/value-1. |

| (cross-cutting) | Error guessing / checklist | Authorization and IDOR conditions,
  anchored on the systematic-expansion checklist (not named per-AC in the source,
  but a reflex expansion for any authenticated multi-actor workflow). |

That last row is the one worth noticing: nothing in the story mentions authorization. It comes from a systematic expansion pass, not from the text โ€” and it is the row that later produced a real IDOR finding in the demo application.

4 Rank by risk, and hand the arbitration back

prioritize โ†’ writes state/US-004/04-priorities.md

Probability ร— impact per condition, proposed. The tool ranks; a human decides. Every ambiguity is carried through with the question that produced it, so a P1 flagged low (Q1) stays visibly unresolved rather than quietly becoming a fact.

5 Write the test book

testbook-generate โ†’ writes testbooks/US-004/*.feature and coverage-matrix.md

38 scenarios from one story. Stable IDs so a regeneration diffs instead of overwriting, the technique on the tag line, the condition in a comment.

Produced โ€” one scenario of 38
@QAIA-US-004-019 @AC4 @P2 @negative @boundary
# condition: AC4-C3 [req-neg] โ€” priority P2
Scenario: A line dated 91 days ago is blocked with an explanatory message
  Given "employee@demo" has a draft report with one EUR line "supplies" of 10.00
        dated 91 days ago, receipt attached
  When "employee@demo" submits the report
  Then the attempt is refused with a message mentioning "90 days"
Produced โ€” the coverage matrix, which admits what it does not know
| AC  | Condition   | Scenario ID       | Prio | Rationale                              | Confidence  |
| AC2 | AC2-C1      | @QAIA-US-004-008  | P1   | Financial-control boundary.            | normal      |
| AC2 | AC2-C2      | @QAIA-US-004-009  | P1   | Q1-flagged; exact-โ‚ฌ500 planted-        | low (Q1)   |
|     |             |                   |      | ambiguity boundary.                    |             |
| AC3 | AC3-C1 [req-neg] | @QAIA-US-004-013 | P1  | Classic internal-control defect class. | normal      |

Eleven of the 38 rows carry a named open question like that one.

6 Turn it into tests that run without QAIA

automate โ†’ writes Playwright specs carrying the same IDs

Native Playwright, page objects as fixtures, the scenario ID on the test so a failure points back at the requirement.

Produced
test('@QAIA-US-004-021 @AC5 @P1 @negative @boundary a line at exactly the receipt
      threshold without a receipt is refused', async ({ request }) => {
  const t = await apiLogin(request, B, 'employee@demo');
  const { id } = await apiCreateDraft(request, B, t, {
    lines: [{ category: 'gear', amount: 25.00, date: todayISO(), receipt: false }],
  });
  const r = await apiSubmit(request, B, t, id);
  expect(r.status()).toBe(422);
});
And then QAIA gets out of the way. A suite generated this way runs on a GitHub Actions runner with no Claude session, no skill loaded, nothing from the plugin directory read โ€” 8 tests, 8 green. That is the one claim on this whole site that no amount of writing could fake: open the run.

7 Scored by something that did not write it

testbook-score, automation-score, aptitude-gate โ€” a separate, read-only plugin

No producer grades its own output. That rule has repeatedly caught defects the producing skill could not see in itself โ€” including one where the generated code was perfectly faithful to a test book whose Then was too weak, so the tests passed against the very defect the requirement existed to forbid. A judge looking only at code-vs-book would have missed it too.

What you would still have to decide

Nothing above tells you whether this is worth your time on your ticket โ€” no QA engineer has ever run it end to end on real work, and that is the project's largest unknown. If you would consider being the first, there is a 15-minute pilot kit built for exactly that, and the only thing asked in return is where it went wrong.

โ† Back to the overview ยท Read the full run, every file โ†’