Distilling knowledge for low-energy AIoT

The Artificial Intelligence of Things (AIoT) empowers IoT devices to leverage the advantages of AI near data-sources, reducing data movement, latency, and mitigating privacy issues. However, AI workloads are notoriously energy-intensive, posing significant challenges for energy-constrained IoT devices. Since such devices are often deployed in thousands of instances, even minor inefficiencies can significantly increase carbon emissions and energy consumptions. Model compression techniques have been employed to enable AI inference in resource-constrained environments. For example, Knowledge Distillation (KD) is an elaborate approach targeting low-footprint and high-accuracy models, although introducing further complexity during training due to inefficient grid searches of additional hyperparameters. The emerging wave of AIoT, however, calls for prioritizing energy-awareness both in inference and training. To address this shortcoming, this work proposes a three-stage design workflow for low-energy AIoT applications, driven primarily by an input energy budget characterizing the target IoT scenario. Given a specific CNN architecture and IoT platform, our workflow identifies the most effective student under the imposed energy constrained and derives an efficient configuration of the KD hyperparameters that maximizes student accuracy, while avoiding inefficient and expensive grid-search. Hence, this approach enable energy-efficient CNN inference while substantially reducing overall training costs. We validate our workflow with a systematic experimental campaign using ResNets and DenseNets on CIFAR-10, CIFAR-100,and Tiny Imagenet datasets, on an AMD Xilinx Zynq Ultrascale+ ZCU102 MPSoC. Our proposal maintains high accuracy while lowering energy consumption by up to 80%, highlighting the potential of our flow for real-world AIoT applications.