Image composition assessment is crucial in aesthetic assessment, which aimsto assess the overall composition quality of a given image. However, to thebest of our knowledge, there is neither dataset nor method specificallyproposed for this task. In this paper, we contribute the first compositionassessment dataset CADB with composition scores for each image provided bymultiple professional raters. Besides, we propose a composition assessmentnetwork SAMP-Net with a novel Saliency-Augmented Multi-pattern Pooling (SAMP)module, which analyses visual layout from the perspectives of multiplecomposition patterns. We also leverage composition-relevant attributes tofurther boost the performance, and extend Earth Mover's Distance (EMD) loss toweighted EMD loss to eliminate the content bias. The experimental results showthat our SAMP-Net can perform more favorably than previous aesthetic assessmentapproaches.