Remarkable progress has been made in 3D human pose estimation from amonocular RGB camera. However, only a few studies explored 3D multi-personcases. In this paper, we attempt to address the lack of a global perspective ofthe top-down approaches by introducing a novel form of supervision -Hierarchical Multi-person Ordinal Relations (HMOR). The HMOR encodesinteraction information as the ordinal relations of depths and angleshierarchically, which captures the body-part and joint level semantic andmaintains global consistency at the same time. In our approach, an integratedtop-down model is designed to leverage these ordinal relations in the learningprocess. The integrated model estimates human bounding boxes, human depths, androot-relative 3D poses simultaneously, with a coarse-to-fine architecture toimprove the accuracy of depth estimation. The proposed method significantlyoutperforms state-of-the-art methods on publicly available multi-person 3D posedatasets. In addition to superior performance, our method costs lowercomputation complexity and fewer model parameters.