We ran a live session last week where we put an AI agent on a real test bench. Somewhere in the middle of it, someone asked a question that I have been thinking about since.
They said they had been using Claude to generate unit tests, and to their surprise it wrote tests that met the coverage target without uncovering any actual bugs in the function. It hit the number. It found nothing. And they wanted to know how you avoid that.
It is a fair complaint, and on the surface it reads as a complaint about AI. I do not think it is. I think it is a complaint about evidence, and it applies just as much to tests a person wrote.
So this is my answer, longer than I could give on the day.
Models are agreeable, and that is the problem
If you have worked with these tools for any length of time you have had this exchange. You ask whether something works and it tells you it does. You push back and it immediately agrees that you are right. Then you push back on that and it agrees again.
Often it will give you “hey, it passes”, and it is more or less just being agreeable.
That tendency is harmless when you are drafting an email. It is not harmless when the thing being confirmed is whether firmware works on a board that is going to ship to a customer.
And coverage does not save you, which is what the person asking the question had run into. Coverage tells you which lines were executed. It says nothing about whether the right conditions were checked. You can have a hundred percent coverage and still not be testing the things that break, and a human can write that test suite just as easily as a model can. I have seen plenty of hand-written suites that were green for years and never once exercised the case that eventually took the product down.
So the question is not really “how do I stop the AI being agreeable”. It is “what would make a pass mean something”, whoever or whatever produced it.
Write the test before the thing it tests
Most test suites are written after the code works. You get the firmware running, you look at what it does, and you write a test that checks for that. It feels like the sensible order, and it is how a lot of teams work, including ones I have been on.
The trouble is that the test now describes the firmware’s behaviour rather than the requirement. If the firmware is wrong in a consistent way, the test will agree with it every single time. You have built a check that confirms the thing it was supposed to challenge.
The alternative is to take the limits from the specification and the datasheet, before there is any firmware to look at. The numbers are usually already there. A chip-select hold time has a minimum in the datasheet. A baud rate has a tolerance. A rail has a voltage it is not supposed to go below. Those become the assertions, and they were set by the requirement, not by the code.
We did this on our own hardware earlier this year. We used our bench to bring up the signal acquisition board that the bench itself uses, and the firmware came from an AI agent. It compiled cleanly. It passed every software test. It passed engineer review, including mine.
On the first run on the bench, a timeline test flagged a single missing byte. The test did not know anything about interrupt priorities. All it knew was that the bytes going in should be the same bytes coming out, because that is what the spec said. The firmware disagreed with the spec, and the test caught it.
If we had written that test afterwards, by watching what the firmware did, I am not sure what it would have been comparing against.
Do not let the same context grade its own work
Someone else on the session asked a sharper version of the same thing. If the same agent writes the firmware and writes the tests, how do you stop it testing its own assumptions?
In our setup those are separate agents. It may well be the same model underneath, but it is not the same context. The agent writing the test plan starts fresh and works from the intent and the datasheet, not from the conversation where the firmware was written. Whatever assumptions were made while writing the code do not carry over, because there is nothing to carry them.
What surprised me is how much difference the scaffolding makes to the tone. When an agent is given a specification and asked to check against it, with no memory of having produced the thing under test, it is noticeably more skeptical than agreeable. It is not trying to please anyone. It is comparing two things and reporting the difference.
The same logic applies without any AI involved. It is why you do not let the person who wrote a feature be the only person who reviews it.
Keep the evidence, not the verdict
This is the part that actually answers the original question.
A pass should not be a sentence. It should be a measurement you can go back and look at.
Every test we run stores what the instruments captured: the waveform, the timing, the voltages, the moment each event happened on a shared timeline. When a test says it passed, the capture is sitting next to it. If you do not believe it, you can open it.
That changes what a green result means. It is no longer a model’s opinion about its own work. It is a recording of what the hardware did, and the pass is simply the observation that the recording matches the specification. If someone disagrees later, there is something to disagree with.
It also changes what happens at two in the morning when something fails in CI. You do not get “assertion failed”. You get the trace, the timestamp, and what the firmware was doing at that moment. On the gateway, that is the difference between a fix that took an afternoon and a fix that would have taken days.
I would go further and say this is the only part of the whole approach that is not optional. You can argue about which model to use, how to prompt it, where to draw the line on automation. But if the result of a test is a claim rather than a capture, you have not tested anything. You have asked for an opinion.
And a person still signs it off
We keep coming back to one rule, and we said it twice during the live session because it matters more than anything else we showed.
Nothing reaches the hardware just because an agent said it was going to work.
The agent plans, writes, flashes and measures. An engineer reviews what it produced before it goes near a board, and an engineer decides what the results mean. That is not a safety blanket we added to make people comfortable. It is there because these are time-critical and sometimes safety-critical systems, and because the measurement is evidence, not a conclusion. Somebody has to read it.
If you are trying this on your own bench, the three things I would take from our experience are these. Write your limits from the spec before the firmware exists. Keep whatever writes the code out of the context that checks it. And save every capture, so a pass is something you can inspect rather than something you were told.
None of that requires a particular tool. It is mostly about where you let the truth come from.
We ran this live on 20 October, including the parts that did not go to plan. The recording is on our LinkedIn page, and the full case study on the gateway board, with both faults and the numbers, is here.