Choosing the silicon for an edge AI product is one of the few decisions that cannot be revised later. Firmware can be rewritten, models can be retrained, cloud services can be replaced — but the processor is soldered to a board that is already installed somewhere you cannot reach.
The choice spans four broad classes of hardware separated by roughly three orders of magnitude in both capability and power. Picking the wrong class does not produce a slightly worse product; it produces one that cannot meet its battery life, cannot hit its frame rate, or cannot be sold at its target price.
This guide compares those classes in practical terms, explains the specifications that actually predict performance, and sets out a selection process that avoids the most common and expensive mistakes.
The Four Classes of Edge AI Hardware
| Class | Memory available to models | Power | What it can realistically run |
|---|---|---|---|
| Microcontroller (Cortex-M class) | Tens to hundreds of KB | Microwatts to milliwatts | Keyword spotting, motion classification, simple anomaly detection |
| MCU or MPU with NPU | Hundreds of KB to a few MB | Milliwatts to ~1 W | Small vision models, audio scene analysis, multi-sensor fusion |
| Application processor (Cortex-A class) | Hundreds of MB to GB | 1–5 W | Moderate vision and audio models, several streams at low frame rates |
| SoC with GPU or dedicated accelerator | GB, with high bandwidth | 5–30 W | Real-time detection and segmentation, multiple camera streams |
The gaps between these classes are far larger than the table suggests, because memory bandwidth and tooling maturity scale with them too. Moving up a class typically means an operating system, a larger board, active thermal management, and a different power architecture — it is a product decision, not a component swap.
What Each Class Is Actually Good For
Plain microcontrollers
The domain of TinyML. Models are measured in kilobytes and must fit alongside the application in RAM, which forces aggressive quantisation and small architectures.
- The device runs for years on a primary cell
- The task is classification over low-rate sensor data rather than imagery
- Unit cost is measured in single-digit dollars
- No operating system is wanted, for determinism or certification reasons
The constraint that surprises teams is RAM rather than compute. A model may be small on disk yet require far more working memory for intermediate activations than the part has available. Our guide to TinyML covers how to work inside those limits.
Microcontrollers and MPUs with an integrated NPU
The fastest-moving segment, and increasingly the sweet spot for battery-powered products that need real inference. A dedicated neural accelerator delivers an order of magnitude better performance per watt than running the same network on the main core.
- You need genuine vision or audio inference within a tight power budget
- The product is battery-powered but recharged or replaced periodically
- Cost matters, but a few extra dollars per unit is acceptable
The caveat is operator support. NPUs accelerate a defined set of operations, and any layer outside that set falls back to the CPU. A single unsupported operator can eliminate the entire advantage, so the network must be designed against the accelerator’s supported list rather than adapted afterwards.
Application processors
Full Linux-capable processors without a dedicated AI accelerator. Inference runs on the CPU cores or an integrated GPU, which is adequate for moderate models at modest frame rates.
- The product already needs Linux for connectivity, storage, or a display
- Inference is occasional rather than continuous
- Development speed matters more than power efficiency
- Mains or vehicle power is available
This class is often chosen for the wrong reason — because the software team is comfortable with Linux. That is a legitimate consideration, but it should be stated as such rather than disguised as a technical requirement.
SoCs with GPU or dedicated accelerators
Modules built for continuous, heavy inference: real-time object detection, segmentation, or several camera streams at once.
- Multiple video streams must be processed simultaneously
- The model is genuinely large and cannot be reduced without losing the product
- Mains power and space for cooling exist
- Unit volumes are low enough that module cost is acceptable
Thermal design becomes a first-order concern here. Sustained throughput inside a sealed enclosure is frequently well below the benchmark figure, and a module that throttles is effectively a slower module.
Which Numbers Actually Predict Performance
Datasheet headlines are poor predictors of real throughput. These four factors matter far more.
TOPS is a marketing number. Peak activation memory, memory bandwidth, and operator coverage are engineering numbers.
Tooling and Support Decide Your Timeline
Two parts with identical specifications can differ by months of engineering effort, and the difference is almost entirely ecosystem.
- Conversion toolchain maturity. Getting a trained model into the vendor’s runtime is where schedules slip. A well-documented, actively maintained converter is worth more than a hardware advantage.
- Framework compatibility. Confirm support for the framework you actually train in, not merely for an intermediate format that requires a conversion step nobody has tested.
- Profiling tools. Without per-layer timing you cannot tell whether the accelerator is being used or silently bypassed.
- Reference designs and sample code. A working camera-to-inference pipeline saves weeks of integration.
- Longevity commitment. Industrial products need parts guaranteed available for ten years or more. Consumer-oriented silicon may be discontinued mid-lifecycle.
- Community and support quality. When something behaves unexpectedly at two in the morning, existing answers matter more than a support contract.
A Selection Process That Avoids Expensive Mistakes
The trade-offs these choices express — accuracy against power, latency against complexity, capability against unit cost — are examined in detail in our guide to edge AI trade-offs.
Frequently Asked Questions
Conclusion
Edge AI hardware selection is a decision made once and lived with for years. The four classes are genuinely different products rather than points on a continuum, and moving between them changes the board, the power architecture, the thermal design, and the software stack.
Define the workload before shortlisting. Let power and cost ceilings eliminate classes early. Run your own model on real silicon and profile it per layer. Test sustained throughput inside the actual enclosure. And leave real headroom — because the only edge AI mistake that cannot be fixed in software is choosing a part with nothing left to give.