I use AI to write firmware most days. This is not a piece about how it cannot do the job. It does a lot of the job, and it has changed how I work. It is a piece about the line, because almost everything written about AI in embedded right now is about the half that works, and the half that does not is the half that decides whether your product ships.
So here is the line, as honestly as I can draw it, from someone who works on both sides of it.
What it genuinely does well
Credit first, because the limits only mean something if the capability is real.
An agent writes peripheral initialisation faster than I do, and usually correctly. Register setup, clock tree configuration, DMA descriptors, the boilerplate that used to eat an afternoon. It handles the parts of firmware that are essentially transcription, turning a datasheet’s register map into working setup code, better and faster than a person doing it by hand. It is good at the well-trodden paths: a standard UART driver, an I2C transaction, a state machine you have described clearly. Give it a clear specification and it produces clean, compilable code against it.
That is not a small thing. A meaningful fraction of firmware work is exactly this kind of transcription, and handing it off is a real gain. The time saved is genuine.
Then the code has to run on a physical board, and that is where the line is.
Limit one: it cannot see the board
This is the fundamental one, and everything else is a consequence of it.
An agent has never seen your hardware. It has no eyes on the signal, no probe on the pin, no reading of the rail. It knows what the datasheet says should happen. It has no way to know what actually happens, because what actually happens exists only in the physical world, on your specific board, under your specific conditions.
That sounds obvious, but it has sharp consequences. The agent will write code that is correct against the datasheet and wrong against the silicon, and it will have no idea, because the information that would tell it is not available to it. A timing margin that holds at room temperature and fails at −10°C. A bus that works until a second peripheral contends for it. A rail that sags for a few hundred nanoseconds when several outputs switch at once. None of that is in the datasheet. All of it is real. The agent cannot reach it.
Limit two: correct in isolation, wrong under load
Here is the one that catches people, because it looks like success right up until it isn’t.
An agent will write a piece of firmware that is correct when it is the only thing running. Every unit test passes. It reads correctly. A competent engineer reviewing it would approve it, because there is nothing wrong with it in isolation. Then you put it on hardware where it has to coexist with everything else the device is doing, and it fails, because the fault was never in the code as written. It was in the interaction.
Interrupt priorities are the classic case. An agent will assign them in a way that is entirely reasonable for each interrupt considered alone, and that starves one of them under real simultaneous load. There is nothing in the source that is wrong. The bug only exists when two things happen at once, and “two things happening at once” is not something the agent can test against, because it cannot run the code on the hardware where those two things collide.
This is the hardest class of firmware bug for anyone, human or AI. The difference is that a human debugging it can put a scope on the board and watch it happen. The agent cannot.
Limit three: it does not know what it does not know
An agent does not carry the scar tissue that experienced firmware engineers carry.
It has not been burned by a specific vendor’s silicon erratum that contradicts the vendor’s own datasheet. It does not have the instinct that says “this timing looks fine on paper, but I have seen this peripheral misbehave, so I am going to check it on a scope before I trust it.” It does not hesitate. It produces confident, plausible code for situations where an experienced engineer would slow down and be suspicious, precisely because the experienced engineer has been wrong here before and the agent has not been wrong at all, because it has never been on a bench.
That confidence is a limit, not a feature. The most dangerous output is not the code that fails to compile. It is the code that looks right, compiles clean, passes review, and is wrong in a way that only the physical world reveals.
Limit four: the physical setup itself
Even the parts of testing that are becoming automatable run into the physical world.
Someone has to connect the board. Someone has to decide what to measure and where to put the probe. Someone has to know that the rail worth watching is this one, not that one, because they understand what the device is supposed to do. An agent can drive an instrument once the instrument is connected and told what to look for, but the judgement about what matters, which signal, which margin, which condition is the one that will bite in the field, is an engineering judgement grounded in understanding the product. That does not come from the datasheet either.
So where does that leave the human
Exactly where the interesting work always was.
The agent takes the transcription, the boilerplate, the well-trodden paths, and it takes them fast. What is left for the engineer is the part that was always the actual engineering: deciding what to build, judging what to test, reading what the hardware is actually doing, and knowing, from having been wrong before, where to be suspicious. The agent made the first part fast. It did not touch the second part, and I do not think it is close to touching it, because the second part lives in the physical world and in experience, and the agent has access to neither.
This is not a limitation to be engineered away in the next model. Anthropic said as much when they previewed a standard for AI operating physical equipment: their own testing found the agents could generate and run procedures, but still struggled with physical cause and effect, so human oversight remains a must. That is not a temporary state. It is the shape of the problem. Code is information, and an agent is very good with information. Hardware is physics, and physics has to be measured, by instruments, watched by someone who knows what they are looking at.
What this means in practice
If you are using AI to write firmware, and you should be, the thing to get right is the boundary. Let it do the transcription. Review what it writes, because it will be confidently wrong in ways that read as right. And do not, and this is the whole point, trust any of it until it has run on real hardware and been measured, because the failures that matter are exactly the ones that do not exist until the code meets the silicon.
That last step, closing the loop between what the agent wrote and what the hardware actually does, is most of what my job is now. The writing got fast. The finding out did not, and the finding out is where the product is won or lost.