Ye Shaokai ; Filippova Anastasiia ; Lauer Jessy ; Schneider Steffen ; Vidal Maxime ; Qiu Tian ; Mathis Alexander ; Mathis Mackenzie Weygandt

Abstract

Quantification of behavior is critical in applications ranging fromneuroscience, veterinary medicine and animal conservation efforts. A common keystep for behavioral analysis is first extracting relevant keypoints on animals,known as pose estimation. However, reliable inference of poses currentlyrequires domain knowledge and manual labeling effort to build supervisedmodels. We present a series of technical innovations that enable a new method,collectively called SuperAnimal, to develop unified foundation models that canbe used on over 45 species, without additional human labels. Concretely, weintroduce a method to unify the keypoint space across differently labeleddatasets (via our generalized data converter) and for training these diversedatasets in a manner such that they don't catastrophically forget keypointsgiven the unbalanced inputs (via our keypoint gradient masking and memoryreplay approaches). These models show excellent performance across six posebenchmarks. Then, to ensure maximal usability for end-users, we demonstrate howto fine-tune the models on differently labeled data and provide tooling forunsupervised video adaptation to boost performance and decrease jitter acrossframes. If the models are fine-tuned, we show SuperAnimal models are10-100$\times$ more data efficient than prior transfer-learning-basedapproaches. We illustrate the utility of our models in behavioralclassification in mice and gait analysis in horses. Collectively, this presentsa data-efficient solution for animal pose estimation.