AIoT projects rarely fail at the model. They fail earlier, in decisions made before anyone trained anything — where data would be processed, what the device would be capable of, how readings would be labelled, and whether outcomes would ever be recorded.
Those are architecture decisions, and they are made at a point when the model is still hypothetical. By the time accuracy turns out to be disappointing, the constraints that caused it are embedded in deployed hardware.
This guide covers the architectural choices that determine AIoT success, why each one is hard to reverse, and what to decide differently.
Where Inference Runs
Choosing between device, gateway and cloud looks like an optimisation. It is actually a constraint on what the product can ever do.
- Cloud-only inference means no reaction during connectivity loss, latency measured in hundreds of milliseconds, and bandwidth costs that scale with the fleet.
- Device inference means millisecond response and offline operation, but caps model size at whatever silicon was chosen.
- Gateway inference allows correlation across devices at a site, at the cost of another managed fleet.
Why it is hard to reverse: moving inference onto devices later requires hardware with compute and memory that was never specified. The decision is effectively made when the bill of materials is signed off.
How Much Headroom the Hardware Carries
Choosing the cheapest part that runs the first firmware is rational per unit and frequently fatal to the roadmap. Models grow, features are added, and the margin that was saved becomes the reason a capability cannot ship.
The asymmetry is stark: a slightly larger part costs a small amount per unit, while a hardware revision across a deployed fleet costs re-validation, possible recertification, and in many cases physical replacement.
Specify silicon for the model you expect in two years, not the one you have this quarter.
What Data Is Kept, and at What Fidelity
Edge reduction is necessary and irreversible. A fleet that transmits only summaries cannot later train a model that needs the underlying signal, because that signal was never stored anywhere.
The fix is cheap if decided early: keep a small representative cohort reporting full-fidelity data, and capture short raw windows around interesting events. Both cost very little and preserve every future modelling option. Decided late, neither helps, because the historical data simply does not exist.
Whether Outcomes Are Recorded
This is the most consistently skipped decision and the one that quietly ends projects. If nobody records what actually happened after a prediction, then accuracy cannot be measured, drift cannot be detected, and the model cannot improve.
A deployment in that state has no way to demonstrate value. When budget is reviewed a year later, the team cannot answer the only question that matters — did it work? — and the programme is cancelled on the basis of absent evidence rather than poor performance.
How Field Variation Is Handled
Architecturally this means tracking accuracy per cohort rather than fleet-wide, and designing for per-asset baselining rather than universal rules.
Decisions to Make Differently
These are the same foundations that determine whether data can be trusted at all, examined in the data properties behind reliable AIoT systems.
Frequently Asked Questions
Conclusion
AIoT failures are usually architectural rather than algorithmic, and they are decided early — at component selection, at the data reduction policy, at whether anybody built a way to record what happened.
Place inference according to latency and availability, buy real headroom, keep a raw-data cohort, build the feedback loop before the first model, version everything, and start where labels are not required. Projects that make those decisions deliberately rarely fail for reasons anyone could have prevented.
Automation projects away from the device layer fail for a parallel set of reasons, mostly non-technical. We set them out in why AI automation projects fail.