Abstract

Human-centric perception tasks, e.g., pedestrian detection, skeleton-basedaction recognition, and pose estimation, have wide industrial applications,such as metaverse and sports analysis. There is a recent surge to develophuman-centric foundation models that can benefit a broad range of human-centricperception tasks. While many human-centric foundation models have achievedsuccess, they did not explore 3D and vision-language tasks for human-centricand required task-specific finetuning. These limitations restrict theirapplication to more downstream tasks and situations. To tackle these problems,we present Hulk, the first multimodal human-centric generalist model, capableof addressing 2D vision, 3D vision, skeleton-based, and vision-language taskswithout task-specific finetuning. The key to achieving this is condensingvarious task-specific heads into two general heads, one for discreterepresentations, e.g., languages, and the other for continuous representations,e.g., location coordinates. The outputs of two heads can be further stackedinto four distinct input and output modalities. This uniform representationenables Hulk to treat diverse human-centric tasks as modality translation,integrating knowledge across a wide range of tasks. Comprehensive evaluationsof Hulk on 12 benchmarks covering 8 human-centric tasks demonstrate thesuperiority of our proposed method, achieving state-of-the-art performance in11 benchmarks. The code is available on https://github.com/OpenGVLab/Hulk.