How AI on Microcontrollers Actually Works: Operators and Kernels

Summary

The buzz around “edge AI”, which means something slightly different to almost everyone you talk to, is well past reaching a fever pitch. Regardless of what edge AI means to you, the one commonality is typically that the hardware on which inference is being performed is constrained in one or more dimensions, whether it be compute, memory, or network bandwidth. Perhaps the most constrained of these platforms are microcontrollers. I have found that, while there is much discourse around “running AI” (i.e. performing inference) on microcontrollers, there is a general lack of information about what these systems are actually capable of, and how new hardware advancements impact that equation. It is my hope with this series to peel back some of the layers of terminology and explore what actually happens between supplying inputs to a model and receiving outputs. Along the way, we’ll ground our exploration in performing inference with real models on real constrained hardware. While “weights” get the majority of the attention with AI models, they alone are not sufficient for performing inference. Depending on how a model is distributed and what runtime is used, additional data or metadata may be supplied alongside the model, or may be defined explicitly in software that interacts with the weights. The most popular runtime for microcontrollers is Tensorflow Lite for Microcontrollers (tflite-micro), which is an optimized version of Tensorflow Lite. Note: Google recently rebranded Tensorflow Lite to LiteRT, and tflite-micro to LiteRT for Microcontrollers. tflite-micro uses the .tflite file format, which encodes data using FlatBuffers. Unlike some other model file formats, .tflite files include not only the tensors that encapuslate model weights, but also the computation graph, which informs the runtime of what operations to use when performing inference. In order to do so, there needs to be a defined set of operators. This is somewhat analagous to instructions defined in an instr...