A support ticket arrives. Eleven units out of a fleet of four thousand are rebooting, roughly once a fortnight, with no pattern anyone can see. The units come back and test fine. The logs show nothing useful. Two engineers spend a week on the bench trying to reproduce the fault and cannot.
Most teams then do one of three things. They ship a watchdog tweak and hope. They blame the customer’s power supply. Or the problem goes onto a list of known oddities and everyone moves on. Each of these costs more than it looks, because a fault you cannot reproduce is a fault you cannot bound: nobody knows whether eleven units are affected, or four hundred are affected and eleven have been noticed so far.
We get called into investigations like this, and what follows is roughly the method we use. There is nothing clever in it. Most of it is about collecting the right information before anybody picks up a probe.
- Is the Fault Really Intermittent?
- What Should the Device Report Before it Dies?
- What Kind of Fault are You Dealing With?
- Does the Hypothesis Come from the Failing Unit or from the Fleet?
- How Do You Reproduce a Fault that Will Not Reproduce?
- When is the Investigation Actually Finished?
- How Long Does it Take?
- When is it Worth Bringing in Someone from Outside?
Is the Fault Really Intermittent?
The first thing we do is stop using the word.
A device does not behave randomly. It executes deterministically against its inputs, and some of those inputs are simply not being recorded: a supply rail sagging under a load nobody logged, a timing race that only closes above 60°C, a protocol edge case the datasheet never described, a heap that fragments after nine days of uptime. In the investigations we have run, “intermittent” has nearly always turned out to be a statement about the logging, not about the fault.
That distinction matters for planning. If the fault were genuinely random, the only available strategy would be to wait. If the fault is deterministic and the conditions are unmeasured, the strategy is to go and measure the conditions, and measuring conditions is work you can put in a schedule.
What Should the Device Report Before it Dies?
Almost every team tries to reproduce the fault before improving their ability to observe it. We do it in the opposite order, and in our experience this is where the schedule is won or lost.
Every embedded device has a watchdog. In most firmware it is configured to reset the part and do nothing else, which for an investigation is worse than useless: the device comes back up, and the evidence is gone with it. Before any reproduction attempt, we put together an instrumentation release that gets the following out of the field:
- Reset reason. Read the reset cause register on every boot and persist it. Telling a watchdog reset apart from a brown-out, a software reset or a hard fault removes whole branches of the search on the first day.
- A pre-reset snapshot. Reserve a small region of RAM that survives reset (a .noinit section, or a battery-backed domain on parts that have one) and keep a ring buffer in it: the last log lines, the running task, the stack pointer, the fault status registers, the uptime. On the next boot, transmit that buffer before anything else runs.
- Boot counters, not boot events. A monotonic counter in non-volatile memory gives the true reset rate. Telemetry that only reports when the network is up will under-count exactly the failures you are looking for.
- Environmental context. Supply voltage, die temperature, RSSI and uptime at the moment of failure. Every hypothesis you form later will be about one of these, and without the readings you are guessing.
This is typically a week of work and one firmware release. Customers sometimes push back on it, because it looks like a detour from finding the fault. It decides whether the investigation converges at all.
What Kind of Fault are You Dealing With?
Not all unreproducible faults are the same kind of problem, and there is a distinction in the reliability literature worth borrowing.
A Bohrbug is deterministic. It reproduces reliably under a well-defined set of conditions, even if nobody has identified those conditions yet. Find the condition and the fault appears every time.
A Mandelbug has causes complex or distant enough from the symptom that the behaviour looks chaotic: timing-dependent races, heap fragmentation, memory corruption that surfaces hours after the write, interactions cascading across subsystems.
The classification changes the toolset. Bohrbugs yield to a condition search, i.e. vary one environmental factor at a time until the fault appears. Mandelbugs generally do not, because there may be no single condition to find. For those we go to soak testing, allocation tracing, and instrumentation dense enough to catch the corruption when it happens rather than when it becomes visible.
A quick test we apply: does the failure rate correlate with anything at all? Uptime, temperature, firmware version, geography, unit age, network conditions. Any correlation points toward a Bohrbug. No correlation, combined with a symptom that appears long after its likely cause, points toward a Mandelbug.
Does the Hypothesis Come from the Failing Unit or from the Fleet?
The instinct is to take a returned unit and put a probe on it. Sometimes that works. More often the returned unit behaves perfectly on the bench, because the condition that caused the failure lives in the deployment and not in the hardware.
So we interrogate the fleet first:
- What is different about the affected units? Firmware revision, manufacturing lot, assembly date, component supplier, geography, customer, installation type. Eleven units out of four thousand is a strong signal. If all eleven carry the same date code on one passive, or shipped in the same fortnight, the search space has just shrunk by three orders of magnitude.
- What is different about the moments of failure? Cluster the failure timestamps against local time, ambient temperature, network events and duty cycle. A fault that only occurs between 06:00 and 08:00 local time is telling you something about load, or thermal cycling, or a scheduled task colliding with something.
- What did the field say before the engineers did? Support tickets often contain the hypothesis in the customer’s own words. “It happens when the compressor kicks in” is a complete hypothesis. So is “only on humid days.”
One pattern we specifically look for: a QA observation that was raised and never escalated. On more than one investigation, the root cause had been seen months earlier by someone on the production line who had no process for reporting it upward.
How Do You Reproduce a Fault that Will Not Reproduce?
You stop trying to reproduce the failure and start reproducing the condition. In our investigations this is the step that ends the impasse.
If the hypothesis is a supply sag during a flash write, we do not wait for a sag. We build a bench that creates one: a programmable supply, a scripted voltage profile, a trigger tied to the write operation. Run it a thousand times overnight and count the corruptions in the morning.
If the hypothesis is a network drop mid-transfer, we cut the link at a controlled byte offset in the transaction instead of waiting for the network to misbehave.
If the hypothesis is thermal, the unit goes into a chamber and gets swept, with the interface under instrumentation for the whole sweep.
This is what a HIL bench with fault injection is for, and it is the point where an unreproducible field fault becomes a repeatable bench event. For a sense of the volumes involved: the robustness targets we work to are fewer than one corruption per ten thousand simulated brown-outs for non-critical storage, and zero for bootloader and firmware images.
When the fault sits on an interface nobody has documented, the job changes character. It becomes closer to characterising an unknown protocol than to conventional debugging, because the signal has to be captured and understood before the fault in it can be found.
When is the Investigation Actually Finished?
There are two ways to lose at the finish line, and we see both regularly.
The first is accepting correlation as cause. The fault stopped after a change, so the change must have fixed it. For a fault that appears once a fortnight, three weeks of silence is not evidence of anything. Causality means the fault can be switched on and off at will: inject the condition and it appears, remove the condition and it does not, put the original firmware back and it returns.
The second is a fix with no test behind it. A root cause that gets patched with nothing added to the test suite will come back on a refactor two releases later, and the next engineer starts from zero. The last step of every investigation we run is a regression test that executes on real hardware on every build. The fault that cost three weeks of investigation then costs nothing on every build after that.
There is also the question of whether the fix can reach the fleet at all. If the product has no reliable update path, the investigation ends in site visits no matter how good the diagnosis was. That is an architecture problem rather than a debugging problem, and it belongs to the same category of shipping-day commitments we covered in the cost article.
How Long Does it Take?
For a fault with a real environmental cause, on a fleet with usable telemetry:
| Stage | Typical duration |
|---|---|
| Instrumentation release and field data collection | 1 to 3 weeks |
| Population analysis and hypothesis formation | 3 to 5 days |
| Bench reproduction of the candidate condition | 3 to 10 days |
| Causality confirmation and fix | 1 to 2 weeks |
| Regression test encoded into CI | 1 to 3 days |
Call it four to eight weeks, and note that most of it is waiting for field data rather than engineering hours. Teams that skip the instrumentation stage to save a week routinely spend three months on the bench without converging, because they are trying to reproduce a condition they never measured.
When is it Worth Bringing in Someone from Outside?
Outside help earns its cost when the fault sits across a boundary: analog behaviour affecting digital timing, an undocumented third-party interface, RF behaviour the team has never instrumented. Cross-domain faults are hard precisely because the people who own the firmware cannot see the analog side, and the people who own the hardware cannot see the software state. This is the kind of work our field engineering team does.
It earns its cost much less when the real blocker is instrumentation the team could add themselves. No external investigator can find a root cause in data that was never captured.
Frequently Asked Questions
How long should a team spend trying to reproduce a field fault before changing approach? About a week. If the fault has not reproduced on the bench by then, the constraint is almost never effort, it is missing information. Stop and improve the instrumentation instead.
Can a fault be found without reproducing it? Sometimes, from telemetry alone, if the instrumentation is dense enough. It is a worse outcome, because without reproduction the only proof that the fix works is waiting. Reproduction is what turns a plausible explanation into a confirmed one.
Is a watchdog reset a root cause? No. A watchdog reset is evidence that something stopped responding. Configured to capture a pre-reset snapshot, it points at the cause. Configured to reset silently, it destroys the evidence.
What if the returned units test fine? That is common, and it is informative. It means the condition is in the deployment, not in the hardware, so the investigation should turn toward environment, load and installation rather than the unit itself.
Do intermittent faults ever have no root cause? In our experience, no. There is only behaviour that has not been measured yet. Genuine hardware randomness, such as single-event upsets, is rare enough that it should be the last hypothesis on the list, not the first.