Veluri Bandhav ; Pernu Collin ; Saffari Ali ; Smith Joshua ; Taylor Michael ; Gollakota Shyamnath

Abstract

We present NeuriCam, a novel deep learning-based system to achieve videocapture from low-power dual-mode IoT camera systems. Our idea is to design adual-mode camera system where the first mode is low-power (1.1 mW) but onlyoutputs grey-scale, low resolution, and noisy video and the second modeconsumes much higher power (100 mW) but outputs color and higher resolutionimages. To reduce total energy consumption, we heavily duty cycle the highpower mode to output an image only once every second. The data for this camerasystem is then wirelessly sent to a nearby plugged-in gateway, where we run ourreal-time neural network decoder to reconstruct a higher-resolution colorvideo. To achieve this, we introduce an attention feature filter mechanism thatassigns different weights to different features, based on the correlationbetween the feature map and the contents of the input frame at each spatiallocation. We design a wireless hardware prototype using off-the-shelf camerasand address practical issues including packet loss and perspective mismatch.Our evaluations show that our dual-camera approach reduces energy consumptionby 7x compared to existing systems. Further, our model achieves an averagegreyscale PSNR gain of 3.7 dB over prior single and dual-camera videosuper-resolution methods and 5.6 dB RGB gain over prior color propagationmethods. Open-source code: https://github.com/vb000/NeuriCam.