The Charades dataset contains real videos of people doing daily activities. The dataset consists of 9,848 daily indoor activity videos with an average length of 30 seconds, involving 15 indoor scenes, interactions of 46 object categories, 30 verbs and 157 action categories.

Each video in the dataset is annotated with: multiple custom text descriptions, action labels, action intervals, and categories of objects being interacted with. On average, each video contains 6.8 actions, created by hundreds of users in their own homes. 267 users were presented with a sentence consisting of objects and actions from a fixed vocabulary, and recorded videos of them performing the sentence.

The dataset contains 66,500 temporal annotations of 157 action categories, 46 object categories with 41,104 labels, and 27,847 text descriptions of videos. In the standard set, there are 7,986 training videos and 1,863 validation videos.