Multispectral image pairs can provide complementaryvisual information, making pedestrian detection systemsmore robust and reliable. To benefit from both RGB andthermal IR modalities, we introduce a novel attentive multispectral feature fusion approach. Under the guidance ofthe inter- and intra-modality attention modules, our deeplearning architecture learns to dynamically weigh and fusethe multispectral features. Experiments on two public multispectral object detection datasets demonstrate that the proposed approach significantly improves the detection accuracy at a low computation cost.