Moliner Olivier ; Huang Sangxia ; Åström Kalle

Abstract

We address the challenges in estimating 3D human poses from multiple viewsunder occlusion and with limited overlapping views. We approach multi-view,single-person 3D human pose reconstruction as a regression problem and proposea novel encoder-decoder Transformer architecture to estimate 3D poses frommulti-view 2D pose sequences. The encoder refines 2D skeleton joints detectedacross different views and times, fusing multi-view and temporal informationthrough global self-attention. We enhance the encoder by incorporating ageometry-biased attention mechanism, effectively leveraging geometricrelationships between views. Additionally, we use detection scores provided bythe 2D pose detector to further guide the encoder's attention based on thereliability of the 2D detections. The decoder subsequently regresses the 3Dpose sequence from these refined tokens, using pre-defined queries for eachjoint. To enhance the generalization of our method to unseen scenes and improveresilience to missing joints, we implement strategies including scenecentering, synthetic views, and token dropout. We conduct extensive experimentson three benchmark public datasets, Human3.6M, CMU Panoptic andOcclusion-Persons. Our results demonstrate the efficacy of our approach,particularly in occluded scenes and when few views are available, which aretraditionally challenging scenarios for triangulation-based methods.