What AI testing agents find and miss
On October 31, 2026 three Claude models went through the practice exercises on this site: a ticket order form, a sign-up and login form and a REST API for booking meeting rooms, each with 12 to 14 planted bugs per session. Each model did each exercise twice, and every report was checked by hand against the list of bugs planted in its session. A second part gave the models the API reference and asked for an automated test suite.
How the runs were set up
The models were Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5, each working as an agent in a browser tab on a local copy of the site. They got the instruction a person gets: test the form against its requirements, write each bug with steps, expected and actual result, and submit. The budget was about 40 browser actions; the runs took 1.3 to 4.4 minutes. Every session drew its own set of bugs from the exercise's pool, so two runs of one model met different bugs. Two runs per cell is a small sample: the numbers below describe these 18 runs and nothing wider.
How many planted bugs they found
A bug counts as found when the report describes it well enough to reproduce. Both runs are shown.
| Exercise | Haiku 5.5 | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|
| Ticket order form | 4 of 13, 4 of 13 | 7 of 14, 7 of 13 | 7 of 13, 8 of 14 |
| Sign-up and login form | 2 of 13, 1 of 13 | 8 of 12, 8 of 12 | 9 of 12, 9 of 12 |
| REST API | 11 of 13, 8 of 13 | 8 of 13, 7 of 12 | 11 of 13, 11 of 13 |
Over all 18 reports that is 130 of 231 planted bugs, or 56%. The API went best (56 of 77) and the two forms worse (37 of 80 and 37 of 74). Sonnet and Opus scored almost the same in both of their runs; Haiku moved from 11 to 8 on the API. On the forms Haiku wrote short reports, with 2 and 3 items in its two sign-up runs.
In 15 more cases a report pointed at a real bug but got the steps or the cause wrong: a race between two quick requests reported as "there is no overlap check", or a checkbox that only the Enter key skips reported as never checked, with steps that use the mouse. A reviewer who repeats those steps sees the form behave correctly and may close a real bug.
What none of them found
Some bugs stayed unfound in every run that had them. In most of them nothing on the screen changes when they happen.
- The server accepting a value the browser sends, such as the order total, without recomputing it. It shows up only in the network request.
- Customer data and a password written to the browser console.
- An order that fails on the server with no message on the page.
- A field that changes what you type on the keyboard. Most runs filled fields from page scripts instead of typing, so keyboard-only behaviour never had a chance to appear.
- Two requests sent within about two seconds both succeeding. Every run that met this bug saw its effect and named the wrong cause.
The guide on what the server trusts and the double submit guide cover how to look for the first and the last of these by hand.
What the reports got wrong
The 18 reports had 180 items. Nine repeated another item of the same report, usually the same rounding or the same leftover error message shown with a new example. Five described something that was not a bug, for example a duplicate account blamed on letter case in the email, when the real cause was two sign-ups sent within two seconds. A few more items fell for the traps the exercises set; they are left out here so the exercises keep working.
Some items found real problems outside the planted list. Two were defects in the API build that has no planted bugs: an unknown path answered with an HTML page although the reference promises JSON, and a request for a non-numeric id answered "Booking -1 not found". All three models reported them, and both were fixed the same day.
AI-written API test suites
In the second part each model read the API reference, wrote a pytest suite and ran it against nine builds: one that follows the reference and eight with one planted bug each or none. The suites had 102 to 289 tests and took 4 to 18 minutes to write and run, six runs in all.
The marks the models submitted caught every build with a bug but one and flagged no clean build. Run against one build per bug of the whole pool of 23, Opus's first suite caught all 23, Haiku's 22 and Sonnet's 21. The misses were an impossible date in a 30-day month (November 31; Haiku's suite tried only February 30) and a booking that ends after closing time.
Running the suites caused two problems. They created and deleted thousands of bookings, which ran into a limit on the practice API that counted deleted bookings too; the limit was changed. And one Opus suite failed once on a clean build, in one of five runs and with no cause found; the model then used the exercise's interim check to decide that the build was clean.
Review an AI bug report
More guides: REST API test cases · sign-up and login test cases · all guides