Including the cases where the answer is not QAIA. Last updated 10 August 2026; competitor figures read 8 August 2026.
It is the baseline every experienced tester weighs first, and leaving it out would have been the most self-serving omission available. We measured it, on the same ticket, and published the result: a single direct prompt cost 46,548 tokens against QAIA's 133,100 — QAIA is ~2.9× more expensive. On strict recall of the planted ambiguities, the direct prompt matched or slightly beat QAIA on that run. QAIA scored better structurally on average (72 vs 47/100) but still failed the structural gate on 2 of 7 files. The one clean separation: the direct prompt silently invented four business rules and folded them into the output; the QAIA arm invented none and traced every assumption to a numbered question.
So: if your output goes to you and stops there, prompt directly — it is cheaper and on this evidence no worse at finding things. Pay the 3× when someone downstream will ask why. One run per arm, so treat it as a signal, not a verdict. The full benchmark →
Four of them, sorted by the job you are trying to do — not by our preference. Ours is first because it is the one this page is accountable for; the three after it are each better than QAIA at the job named in their heading, and that is the point of listing them.
A regulated-adjacent audit, a client hand-over, a release gate, a coverage report your manager signs. QAIA's unit is not a better prompt — it is an artifact: a Gherkin book with stable scenario IDs, a requirement coverage matrix, a validated manifest per run, a refusal-path coverage gate (every rule that can refuse, error or deny must have a scenario exercising it), and a score computed by a plugin that did not produce the thing it scores. Pointed at json-server and allowed to read only its README, it found two real defects the maintainer later fixed. If nobody downstream will ever ask you "why this scenario and not that one", that machinery is overhead and one of the three tools below is a better fit.
~380 MIT skills, one command (npx @qaskills/cli add playwright-e2e), agent auto-detected, 27+ agents supported, plus an MCP server so you can search and install from inside the agent. Backed by The Testing Academy. We read their istqb-test-design-techniques skill in full: it is competent work — equivalence partitions with a class table, two- and three-value boundary analysis with the off-by-one bug named, decision tables, state transitions, pairwise. If your problem is "Claude writes mediocre tests", that is the shortest path to fixing it, and it is not close.
10 specialised agents in three tiers, MIT, no API key. It starts from a diff or a change and routes it: acceptance-criteria review, scenario design, impacted-test selection, bug reporting, environment health, browser validation via Chrome DevTools, multi-repo impact analysis, and conversion to Playwright/Cypress/Selenium/Gherkin. Its centre of gravity is code that already exists — with one exception worth knowing: its test-scenario-designer starts from acceptance criteria, not from a diff, so it overlaps our entry point directly. Its smart-test-selector, which maps a diff onto your existing suite to say what to re-run and what may break, has no equivalent in QAIA.
A 60-agent autonomous swarm, actively developed. If you want the machine to go wide on its own and you are comfortable with agents that execute, this is the ambitious end of the spectrum.
| Criterion | QASkills.sh | QA Orchestra | Agentic QE Fleet | QAIA |
|---|---|---|---|---|
| Unit shipped | a skill that improves a prompt | agents that review a change | an autonomous agent swarm | a pipeline that emits artifacts |
| Starts from | whatever you ask | a code change | a codebase | a user story, before code exists |
| Catalogue size | ~380 skills | 10 agents | 60 agents | 33 skills |
| Agents supported | 27+ | Claude Code | Claude Code | Claude Code |
| Install | 1 command, auto-detected | plugin / clone | plugin | marketplace + 1–4 installs |
| Auto-executing components | none (skills) | 10 agents | 60 agents | none — no hook, no agent, no MCP server is registered by any of the four manifests |
| Requirement traceability | not the goal | partial (AC review) | not the focus | matrix, AC → condition → scenario ID |
| Who scores the output | n/a | not documented* | not documented* | a separate read-only plugin |
| Ambiguity handling | not documented* | not documented* | not documented* | surfaced as a question, never silently resolved |
| Publishes its own failures | — | — | — | yes — 2.4/5 and 5.0/10, in-repo (self-administered agent panels, not outside reviewers) |
| Maturity | established, 189k-subscriber backing | young | established | pre-alpha, zero human pilots |
Figures read on 8 August 2026 from each project's public repository and site — PramodDutta/qaskills and qaskills.sh, Anasss/qa-orchestra and qa-orchestra.com, proffesor-for-testing/agentic-qe. "Auto-executing components" counts what the tool registers into your environment that can act without you invoking it — a supply-chain question, not a quality one.
* "not documented" means exactly that, and it is a correction. An earlier version of this table asserted that rivals' output is scored by "the same swarm" and that Agentic QE Fleet "resolves ambiguity by the agent". Neither is documented anywhere in their public material — they were inferences of ours dressed as findings, and an unfair claim about a competitor is worse than a weak claim about ourselves. What is true is narrower: we could not find a documented separation between producer and scorer, or a documented ambiguity policy, in either project. If one exists, tell us and the row changes.
Whichever tool you pick, ask it the same question: when the story is ambiguous, what does it do? A generator that quietly picks an interpretation produces a test suite that looks complete and encodes a guess — and the guess is invisible precisely where it matters most, at the boundary. That failure mode is not specific to any tool here, including this one, and it is worth ten minutes of your evaluation regardless of what you install.