If you’ve worked on embedded hardware for any length of time you’ve probably had a morning like this one. A new board revision arrives, you run the test suite you spent weeks building, and half of it fails and you have no idea whether the firmware is broken or the tests are. So you spend the day finding out. You read through the scripts, you check which ones reference the old pin, you fix them one at a time, and by the evening you have learned nothing about the device. You have spent a full day examining the tests just because someone moved UART2 from pin 9 to pin 12 and that pin number was written into forty different places in the scripts.
I used to think of this as just part of the job. I don’t anymore, and I want to explain why, because the fix isn’t a new tool. It’s mostly a change in the order you do things.
What goes wrong with pin-based scripts
A pin-scripted test carries three things;
- where the probe goes
- what signal you send in
- and what result counts as a pass.
The trouble is they all live together, so when a single pin moves the whole script is wrong, even though the signal and the pass limits haven’t changed at all. You’ve broken a test about behaviour by changing a fact about the layout.
There’s a second problem that took me longer to notice, and I think it matters more. In most teams, including ones I’ve been on, the test scripts get written after the firmware is running. You flash the board, watch what it does, and write a test that checks for that. Which sounds reasonable until you realise the test is now describing what the firmware does rather than what it’s supposed to do, so if the firmware is wrong in a consistent way, the test will pass it every single time. You’ve built a check that agrees with the thing it’s meant to be checking.
Writing the test from the spec instead
The alternative is to build the test from the specification and the datasheet before there’s any firmware to look at. The numbers are usually all there already. The datasheet for the parts on our gateway board says the chip-select hold time has to be at least 2.00 µs, so that becomes a line in the test plan, and the firmware has nothing to do with where that number came from. On the bench we later measured it at 4.12 µs, comfortably inside the limit, but the point is the limit existed first.
The pin numbers go somewhere else entirely. We keep them in a short YAML file that just maps each signal to where it physically sits on the current revision. The test talks about UART2, and the YAML says UART2 is on pin 12 this week. On our bench we start from a test intent, for the gateway it was loopback plus DMA ring-buffer integrity and timing across SPI1 and UART2, and the limits get worked out from that and from the datasheet before anybody writes firmware.
Why the order matters
If the test exists before the firmware, it can’t have picked up the firmware’s mistakes, because it never saw the firmware. That sounds like a small thing. It turned out not to be.
On the gateway board, the interrupt design passed review, and I was one of the people who reviewed it. It passed every software test too. On the first bench run, the timeline test, which only asserted that the bytes going in should be the same bytes coming out, flagged one that wasn’t. The test had no idea what the interrupt priorities were and didn’t need to. It knew what the spec said should happen and it reported that something else had happened. If we’d written that test afterwards by watching the firmware run, I’m honestly not sure what it would have compared against.
The bit that pays off later
Next revision, a pin moves, and you change one line in the YAML. That’s it. The tests don’t change because they never knew about pins in the first place, so the same suite runs on the new board and on every build after that without anyone having to touch it.
It also fixes something I’ve run into more than once, which is inheriting someone else’s test suite. When the scripts are full of pin numbers and unexplained limits, the knowledge of why they’re there usually lives in one person’s head, and when that person moves to another project the suite slowly stops being run. With the spec and the YAML kept separate, the next engineer can actually read it.
Where it gets harder
I’d be misleading you if I said this was easy. Turning a specification into numbers is real work and you won’t get the tolerances right first time. Set them too tight and you get false failures, which wastes time and, worse, teaches people to ignore red results. Set them too wide and genuine faults slide through. It usually takes a few rounds.
The wiring also has to match what the YAML claims, because if the file says the probe is on one pin and it’s physically on another, the suite will very confidently measure the wrong thing. We run a continuity check before flashing anything for exactly this reason. It takes a minute, and skipping it is a good way to lose an afternoon to a loose jumper.
And the limit people tend to forget: a test can only find what the spec describes. Our spec covered the data path, so the timeline test caught the lost byte. It didn’t say anything about the 3.3 V rail under simultaneous switching, and that 210 mV sag only showed up because we happened to have the scope on the rail. A spec-driven test won’t be any better than the spec behind it.
What it looked like on the gateway
The board is a 32-bit Arm Cortex-M signal acquisition gateway. Limits came from the spec and datasheet, the harness was declared in YAML and continuity-checked, and the firmware was written to meet the limits. Two faults came out of the first run: a 12 µs interrupt delay under bus contention that lost one UART byte, which took an afternoon to fix once the timeline showed which handler was running, and the rail sag, a decoupling issue we changed before layout freeze rather than after a respin.
The whole thing, firmware, bench and regression suite, took fourteen days, though I wouldn’t treat that as a general estimate because it was our own board and we knew it well. What I’d point to instead is the suite it left behind, which has run on every build since.
If you want to try this on your own bench, write the limits down before the firmware exists, keep the pin mapping in its own file, check the harness before you trust it, and be honest about what your spec doesn’t cover. None of it needs a particular tool.
Frequently asked questions
What is spec-driven testing? Writing test limits from the specification and datasheet before the firmware is written, and keeping physical pin locations in a separate mapping file so a board revision only changes the mapping.
Why do pin-based test scripts break on a new board revision? Because the probe location, stimulus and pass limits sit in the same script, so moving one pin invalidates the whole script even when the behaviour being tested hasn’t changed.
Why write the test before the firmware? A test written by observing the firmware tends to encode the firmware’s own mistakes and pass them. One written from the spec has an independent idea of correct behaviour.
What can’t spec-driven testing catch? Anything the specification doesn’t describe. If the spec says nothing about a behaviour, no test derived from it will check that behaviour.