Qin Yang ; Sun Yuan ; Peng Dezhong ; Zhou Joey Tianyi ; Peng Xi ; Hu Peng

Abstract

Recently, image-text matching has attracted more and more attention fromacademia and industry, which is fundamental to understanding the latentcorrespondence across visual and textual modalities. However, most existingmethods implicitly assume the training pairs are well-aligned while ignoringthe ubiquitous annotation noise, a.k.a noisy correspondence (NC), therebyinevitably leading to a performance drop. Although some methods attempt toaddress such noise, they still face two challenging problems: excessivememorizing/overfitting and unreliable correction for NC, especially under highnoise. To address the two problems, we propose a generalized Cross-modal RobustComplementary Learning framework (CRCL), which benefits from a novel ActiveComplementary Loss (ACL) and an efficient Self-refining CorrespondenceCorrection (SCC) to improve the robustness of existing methods. Specifically,ACL exploits active and complementary learning losses to reduce the risk ofproviding erroneous supervision, leading to theoretically and experimentallydemonstrated robustness against NC. SCC utilizes multiple self-refiningprocesses with momentum correction to enlarge the receptive field forcorrecting correspondences, thereby alleviating error accumulation andachieving accurate and stable corrections. We carry out extensive experimentson three image-text benchmarks, i.e., Flickr30K, MS-COCO, and CC152K, to verifythe superior robustness of our CRCL against synthetic and real-world noisycorrespondences.