How to review a bug report an AI agent wrote
An AI testing agent hands back a tidy list: steps, expected result, actual result, severity. Each item still has to be checked before it goes to a developer: an item whose steps do not reproduce can get a real bug closed as "works for me". This guide is a way to review such a report one item at a time, with numbers from 18 agent reports written on the practice exercises of this site.
What goes wrong in agent reports
The 18 reports came from three Claude models testing two web forms and a REST API with planted bugs. They held 180 items. Nine repeated another item of the same report, usually one rounding problem or one leftover error message shown again with a new example. Five described behaviour that was not a bug. In 15 more cases an item pointed at a real bug but gave steps or a cause that did not hold up, so a reviewer repeating them by hand would see the product work and could close a real defect. The full numbers are in what AI testing agents find and miss.
Most items were fine. The review is there for the minority, and the checks below are ordered so the cheap ones come first.
1. Reproduce each item by hand, at a normal pace
Agents rarely use a page the way a person does. They fill fields from scripts instead of typing, and they send API requests in quick bursts. Two things follow. Some items reproduce only at machine speed, and some steps skip behaviour a person would trigger, such as a key press or a field that reacts to typing. Repeat every item with the keyboard and the mouse, with a few seconds between steps. If it reproduces, move on. If it does not, keep the item open: the next check decides what it is.
2. Separate what happened from why the report says it happened
In the reports here the symptom an agent saw was usually real, and the cause it wrote down was often a guess. Take an item that says "the search ignores the category filter". Done slowly by hand, the filter works. Changing the filter twice within a second shows the list for the first choice: an older response arrives last and replaces the newer one. The bug is real, but it is a race between two requests, and a developer looking for a missing filter will find nothing to fix.
To test a cause, change one thing at a time, for example the speed or the order of the steps. When the symptom follows a different variable than the report names, rewrite the cause and the steps before passing the item on.
3. Find the source of every expected result
"Expected" is a claim about the requirements. For each item, find the line in the specification, the API reference or the acceptance criteria that says so. If there is one, the item stands or falls on the actual result. If there is none, the item is a question for the team, not a defect, however reasonable it sounds.
4. Merge repeats
Agents like to show a problem with several examples, and sometimes they list each example as its own item. Group the items by cause: if fixing one would fix the others, they are one bug with several examples. Keep the clearest item and fold the other examples into it.
5. Check the build the report was written on
A report describes the build that was running when the agent tested it. If the build has changed since, an item can be true for the old build and false for the current one. Note the build or date the item was checked on, and report what the build in front of you does.
6. Look at what the report did not cover
Two of the reports here had only two and three items, on a form with 13 planted bugs. In the runs on this site the agents mostly missed problems that change nothing on the screen: a value the server accepts from the browser without checking it, data written to the browser console, a request that fails with no message, two requests racing each other. After the item review, spend a few minutes there yourself: open the network tab and the console, and type into fields instead of pasting.
A mark for every item
The review ends with one of four marks per item. They are the same four marks the practice exercise below asks for.
| Mark | When | What to do with the item |
|---|---|---|
| Bug | Reproduces by hand as written, and the expected result has a source. | Pass it on; re-check the severity. |
| Bug, wrong steps or cause | The symptom is real, but the steps do not reproduce it or the cause is wrong. | Rewrite steps and cause, then pass it on. |
| Not a bug | The product follows its requirements, the rule is nowhere in them, or the build has changed. | Close it, or turn it into a question for the team. |
| Duplicate | Another item has the same cause. | Merge the examples into that item. |
Severity is the agent's opinion as well; set it again once the mark is decided.
Review an AI bug report
More guides: what AI testing agents find and miss · double submit testing · all guides