QAIA is a set of open-source Claude Code plugins for QA engineers. It reads a user story, surfaces what the story doesn't say, derives scenarios with named ISTQB techniques, and generates native Playwright tests — with stable IDs and a requirement coverage matrix that survives the next sprint.
Pointed at a stranger's repository — json-server, 75,694 stars — and allowed to read only its README, never the code, it found two real defects. Both were later fixed by the maintainer. How it was measured, and the third finding we refused to count →
QAIA ships no code into your repository and no API key. The 33 skills are Markdown, invoked on demand inside your own Claude Code session on your own model quota. Installing it does not register a hook, an agent, or an MCP server that executes on its own. (Precisely: the plugins also carry 51 non-Markdown files — plugin manifests, CI templates you copy, test fixtures, and the four Python files, three of which are qaia-score's scorers. None of them runs at install or on its own; the scorers run when you invoke the scoring skill, with your permissions, and you can read or pin them first. Check for yourself.)
# in Claude Code /plugin marketplace add https://github.com/QAIA-Project/QAIA /plugin install qaia-core@qaia # user story → Gherkin test book /plugin install qaia-playwright@qaia # optional: runnable tests, a11y, perf, security
qaia-core alone gets you from a user story to a test book. Then say "work with QAIA on this user story" — the qaia meta-skill routes to the right step and stops wherever a human has to decide.
Not a mockup. Both panels below are copied verbatim from examples/expense-demo/qaia-journey/ in the repository, where the whole run is kept — source capture, ambiguity pass, technique justification, priorities, and the emitted book.
As an employee, I want to submit an expense report and have it approved through the right chain, so that I get reimbursed correctly and the company keeps an auditable trail. ... 4. Each line item must have a category, an amount, and a date within the last 90 days; a line outside 90 days is blocked at submission with an explanatory message. 5. Receipts are mandatory for any single line ≥ €25; submission is refused if a ≥ €25 line has no attached receipt.
@QAIA-US-004-019 @AC4 @P2 @negative @boundary # condition: AC4-C3 [req-neg] — priority P2 Scenario: A line dated 91 days ago is blocked with an explanatory message Given "employee@demo" has a draft report with one EUR line "supplies" of 10.00 dated 91 days ago, receipt attached When "employee@demo" submits the report Then the attempt is refused with a message mentioning "receipt" and a message mentioning "90 days" @QAIA-US-004-021 @AC5 @P1 @negative @boundary # condition: AC5-C2 [req-neg] — priority P1 Scenario: A line at exactly the receipt threshold without a receipt is refused
Three things the repository records and a landing page would normally quietly drop:
simulated: <default applied>. Nobody arbitrated anything. The arbitration path described below is a design property that this run did not exercise.All three are stated in the run's own journey file. What the panels above show is the shape of the output — the traceability, the tagging, the flagged questions. They are not evidence that it works on your ticket. The measurement on software we did not write is further down.
This is the part most generators skip. Every scenario traces back to an acceptance criterion and a condition — and where the story is ambiguous, the row says so instead of quietly picking an interpretation. The run raised nine open questions; eleven of its 38 scenarios carry one by name.
| AC | Condition | Scenario ID | Prio | Rationale | Confidence |
|---|---|---|---|---|---|
| AC2 | AC2-C1 | @QAIA-US-004-008 | P1 | Financial-control boundary. | normal |
| AC2 | AC2-C2 | @QAIA-US-004-009 | P1 | Q1-flagged; exact-€500 planted-ambiguity boundary. | low (Q1) |
| AC3 | AC3-C1 [req-neg] | @QAIA-US-004-013 | P1 | Classic internal-control defect class. | normal |
| AC3 | AC3-C2 | @QAIA-US-004-014 | P1 | Q2-flagged escalation semantics; approval-bypass risk if wrong. | low (Q2) |
| AC4 | AC4-C1 [req-neg] | @QAIA-US-004-017 | P3 | Basic input-completeness validation. | normal |
Q1, Q2 and the rest are not decoration — they are questions put to a human before the book is accepted, each with the trade-off spelled out. The tool proposes; you arbitrate; nothing is silently resolved. That is the design. It is also the part with the least evidence behind it — the run above was batch-executed with every arbitration defaulted, and no human has yet sat through the interactive path on real work.
scenarios derived from one user story, each traced to an AC and a condition
carry @negative or @boundary. What actually gates is narrower and stricter: every rule that can refuse or deny must have a scenario exercising that path. The raw ratio is reported as a happy-path-bias signal, never a threshold to pad toward — ADR 0001 explains why the ratio gate was scrapped
rows flagged low-confidence with the open question named
Until 8 August 2026, everything this project measured, it measured on code it had produced itself. That is the weakest kind of evidence, and it was the honest criticism of QAIA. So the pipeline was pointed at a stranger's repository.
Target: json-server — 75,694 stars (measured 8 August 2026), chosen because it runs without a database and, more importantly, because it has a public contract that predates us: its README.
The rule of the exercise: the test book was written from that README alone. Never the code, never the issues, never the fix commits. Then the same suite was run against two versions of the same software — the one the README described, and the current one.
The documentation promised _dependent. The code read dependent. The endpoint answered success, deleted the post, and silently left every dependent record in place. A suite written by reading the code cannot find this — it copies the mistake. A real user had filed it as issue #1551; the maintainer fixed it in 1b7c0fb.
Asking for views > 100 and views < 300 returned everything. Conditions were stored in a map keyed by field name, so the second silently replaced the first. Fixed by the maintainer in e6055e6, three months after the version we tested.
_start used alone returns an empty list — a fact, verified by hand. But the README only ever shows it paired, so the test extrapolated a promise the document does not make. Counted as contested, not as a defect. Two, not three.
And the result that goes against us. Run against the current version, four tests fail — and three of those four are our fault. Those features were removed from the documentation in the meantime, and the suite kept demanding retired promises with the confidence of tests that had once been green. QAIA had no way to notice a contract had moved. It does now, and that gap is written up rather than quietly closed.
Read the full campaign, including the protocol and its limits →
What this is not. It is not a pilot. No human has used QAIA in their own work — that remains the project's biggest unknown, and finding defects in a stranger's repository does not substitute for it. It is one target, one API, no user interface, and 32 scenarios rather than full coverage of the README.
Every claim below points at a file or a run you can open without installing anything or taking our word for it.
A suite produced by the automate skill executes on a GitHub Actions runner — no Claude session, no skill loaded, nothing from plugins/ read. 8 tests, 8 green.
Open run 30702503888 →
The structural score is a deterministic algorithm living in a separate, read-only plugin, kept apart from the semantic judge. It has repeatedly caught defects the producing skill could not see in itself. Until 2026-08-09 the plugin shipped no code and asked the model to rebuild the scorer from prose each run, so two runs on one file could legitimately disagree. A score that is not reproducible is not a score — the scorers now ship as pinned Python you can read, diff or refuse, and still do not auto-execute.
See qaia-score →
A 13-persona review scored QAIA 2.4/5; an architecture review gave 5.0/10. Both are published with what was fixed and what wasn't — and both were run by this project on itself, agent panels playing a fictional review firm, not outside humans. That makes them a hostile-reading exercise, not independent validation; calling them an "external audit" would be the same overclaim the grades exist to guard against.
Read the honest status →
The obvious question is not "QAIA or a rival" — it is "why not just prompt Claude directly?" We measured it on the same ticket, and the result is mixed at best:
So the value proposition is narrow and worth stating as such: structure and verifiability, at roughly three times the token cost. If that trade is wrong for your work, a direct prompt is a legitimate answer and this page would rather you knew.
Read the full benchmark, including its verdict against us →
The two questions a QA lead asks in the first sixty seconds, answered before you have to ask them.
Everything lands under .qaia/ — the team knowledge base, the journey checkpoints, the .feature files and coverage matrix, your corrections, and one manifest.json per run that all four plugins share. Plain files, meant to be committed and diffed. Nothing is written outside .qaia/ and the test directories you point it at, and nothing in there executes. The seven steps, and the artifact each one produces →
QAIA runs on your Claude subscription quota, not an API bill. Measured round-trip cost per command ranges from 39k tokens (hello) to 140k (report), with the step most people run first — testbook-generate — at 112k. One run per command, so treat them as orders of magnitude, not averages.
The uncomfortable part, published rather than smoothed: 13 of the 14 measured commands cost more than the project's own prior expert estimate, some by a factor of four. The estimates were wrong; the measurements replaced them instead of the other way round. The full 14-command table, the method behind the numbers, and what each tier buys →
The whole thing is MIT and readable without installing anything. If you do try it and it disappoints, an issue saying exactly how is worth more to this project right now than a star.