Abstract

Being able to map the activities of others into one's own point of view isone fundamental human skill even from a very early age. Taking a step towardunderstanding this human ability, we introduce EgoExoLearn, a large-scaledataset that emulates the human demonstration following process, in whichindividuals record egocentric videos as they execute tasks guided bydemonstration videos. Focusing on the potential applications in dailyassistance and professional support, EgoExoLearn contains egocentric anddemonstration video data spanning 120 hours captured in daily life scenariosand specialized laboratories. Along with the videos we record high-quality gazedata and provide detailed multimodal annotations, formulating a playground formodeling the human ability to bridge asynchronous procedural actions fromdifferent viewpoints. To this end, we present benchmarks such as cross-viewassociation, cross-view action planning, and cross-view referenced skillassessment, along with detailed analysis. We expect EgoExoLearn can serve as animportant resource for bridging the actions across views, thus paving the wayfor creating AI agents capable of seamlessly learning by observing humans inthe real world. Code and data can be found at:https://github.com/OpenGVLab/EgoExoLearn