Tseng Gabriel ; Cartuyvels Ruben ; Zvonkov Ivan ; Purohit Mirali ; Rolnick David ; Kerner Hannah

Abstract

Machine learning methods for satellite data have a range of societallyrelevant applications, but labels used to train models can be difficult orimpossible to acquire. Self-supervision is a natural solution in settings withlimited labeled data, but current self-supervised models for satellite datafail to take advantage of the characteristics of that data, including thetemporal dimension (which is critical for many applications, such as monitoringcrop growth) and availability of data from many complementary sensors (whichcan significantly improve a model's predictive performance). We present Presto(the Pretrained Remote Sensing Transformer), a model pre-trained on remotesensing pixel-timeseries data. By designing Presto specifically for remotesensing data, we can create a significantly smaller but performant model.Presto excels at a wide variety of globally distributed remote sensing tasksand performs competitively with much larger models while requiring far lesscompute. Presto can be used for transfer learning or as a feature extractor forsimple models, enabling efficient deployment at scale.