Effects of hidden layer sizing on CNN fine-tuning

Some applications have the property of being resilient, meaning that they are robust to noise (e.g. due to error) in the data. This characteristic is very useful in situations where an approximate computation allows to perform the task in less time or to deploy the algorithm on embedded hardware. Deep learning is one of the fields that can benefit from approximate computing to reduce the high number of involved parameters thanks to its impressive generalization ability. A common approach is to prune some neurons and perform an iterative re-training with the aim of both reducing the required memory and to speed-up the inference stage. In this work we propose to face CNN size reduction from a different perspective: instead of reducing the network weights or look for an approximated network very close to the Pareto frontier, we investigate whether it is possible to remove some neurons only from the fully connected layers before the network training without substantially affecting the network performance. As a case study, we will focus on “fine-tuning”, a branch of transfer learning that has shown its effectiveness especially in domains lacking effective expert-designed features. To further compact the network, we apply weight quantization to the convolutional kernels. Results show that it is possible to tailor some layers to reduce the network size, both in terms of the number of parameters to learn and required memory, without statistically affecting the performance and without the need for any additional training. Finally, we investigate to what extent the sizing operation affects the network robustness against adversarial perturbations, a set of approaches aimed at misleading deep neural networks.