The QA field guide ↗

How can you compare AI QA tools fairly?

Compare AI QA tools on the same application revision, expected behavior, access and time budget. Independently reproduce reported defects, record missed and untested paths, and measure human review effort. Recus can participate as a browser exploration workflow. This protocol defines a fair trial; it does not claim a measured winner.

By Recus · Reviewed · Vendor-sourced, not an independent benchmark

Set up a repeatable staging trial

  1. Record the application revision, browser, viewport, seeded test data, allowed routes and role permissions. Use disposable accounts and reset the data between runs.
  2. Write the expected outcome before showing the task to any tool. Include a known-good path, a known failure and a path requiring authentication or a revealed control.
  3. Use the same task, access and time budget for every tool. Record tool version, model, configuration and any setup assistance. Label differences that prevent a like-for-like comparison.
  4. Keep the evaluator’s list of introduced failures separate from the tool prompt. Review findings without the vendor label where practical, and have another person resolve disputed results.
  5. Repeat each task at least three times. Retain unsuccessful runs, evidence links and reasons for exclusion. Three repeats expose some variation; they do not establish statistical confidence.

Example: signup validation

Use the same staging signup form with a required email field, a dialog revealing the next step and an account-creation response. Define whether success means visible confirmation, email receipt or a persisted account. Introduce a missing required-field error in a test branch, then check whether each tool reaches and explains that failure.

After fixing it, revisit the failing path and verify the expected outcome. A clean exploratory report is insufficient if that path was never exercised again. Compare the workflow with the illustrative signup report; its contents are an example, not trial data.

Record outcomes, not just test counts

Confirmed defects
Count distinct failures independently reproduced against a written expectation. Deduplicate reports of the same cause.
Incorrect findings
Count reported defects that review establishes are not product failures; preserve the reason for dismissal.
Known defects missed
Use a separate list of deliberately introduced staging failures. Count those the tool did not identify.
Untested or inconclusive paths
Record authentication barriers, unreachable states, unsupported capabilities and exhausted budgets separately.
Human effort
Measure setup, review and maintenance minutes separately from wall-clock execution time.
Repeatability and cost
Run each task at least three times with reset data. Report every run and the actual usage and model costs.
Download a blank trial scorecard (CSV) →

Publish evidence with its limits

For a public result, include the protocol, application revision, tasks, configurations, all run outcomes, deduplication rules and redacted artifacts. Identify who conducted and funded the trial. Obtain permission before sharing customer material. Keep capability comparisons separate from measured performance, and state which workflows were outside scope.

Choose a vendor-specific trial →

What Recus does—and where it stops

Recus studies a reachable web interface, plans browser checks, follows selected discoveries, and returns findings with available screenshots, recordings, and action history. Inferred expectations need human review. Login barriers, unsupported capabilities, and run budgets can leave paths untested. A completed run is not proof of exhaustive coverage.

Questions about this testing workflow

Is this a published benchmark of Recus and its competitors?

No. This is a proposed evaluation protocol. We have not published measured comparative results here. The linked signup report is illustrative, and vendor documentation describes intended scope rather than comparative reliability.

How do you distinguish a bug from an inconclusive test?

A confirmed bug has a defined expected outcome, an observed deviation and a reproducible path. If the agent could not authenticate, reach the state or verify the outcome, record the check as inconclusive. Do not count it as a confirmed product defect or a passing check.

Does finding no bugs prove that the application passed?

No. First confirm that the relevant states were reached and checked. Report known defects missed, paths left untested and repeated-run variation alongside findings. A finished exploration is not exhaustive verification.

Compare real results on your own app.

Keep access, test data, scope, and time budget consistent. Review confirmed bugs and missed paths before selecting a tool.

Try Recus →