Note
How an AI Agent Can Verify a Bug Is Actually Fixed
How can an AI agent verify that a bug is actually fixed rather than trusting the commit log?
Make it re-run the bug, not read the diff: the agent reproduces the reported steps against the live QA environment, captures what the application actually does, and marks the bug fixed only if that evidence shows the defect is gone. If the evidence can't be produced, the result is 'not verified' — never 'probably fixed'.
Why the commit log is the wrong signal
A commit that touches the right file says someone tried. It doesn't say the behavior changed in the environment the bug was reported in — config, data and deployment all sit between the two. Teams that close tickets on commits find out about the gap from users.
Split the job into narrow agents
HallogenAI, which Shreyansh built, coordinates eight specialized agents across five stages instead of asking one general agent to improvise the whole thing:
- Intake — take in the bug report as written.
- Understand — turn it into concrete, reproducible steps and an expected behavior.
- Automate — drive the application through those steps with a real browser (Playwright).
- Evidence — capture what actually happened at each step.
- Report — compare observed with expected and write the verdict.
The rule that makes it work: never guess
Every stage can fail, and the system is built to say so rather than fill the gap. If the steps can't be reproduced, or the evidence is ambiguous, the bug isn't marked fixed. Typed data models (Pydantic v2, SQLAlchemy) keep each report, step and verdict explicit, so 'not verified' is a first-class outcome, not a silent default.
It's the same standard Nythera applies to security findings: evidence or it didn't happen.