Matteo Attimonelli; Danilo Danese; Angela Di Fazio; Daniele Malitesta; Claudio Pomo; Tommaso Di Noia
摘要
在时尚、音乐和电影推荐等特定领域，描述产品和服务的多方面特征可能对在线销售平台上的每位客户产生不同的影响，从而为能够从这种多模态内容中学习的新型多模态推荐模型铺平了道路。根据文献，常见的多模态推荐流程包括：(i) 提取多模态特征，(ii) 精炼这些高级表示以适应推荐任务，(iii) 选择性地融合所有多模态特征，以及 (iv) 预测用户-项目评分。尽管在设计 (ii-iv) 的最优解决方案方面已投入大量努力，但据我们所知，对于 (i) 阶段的研究却很少受到关注。在这方面，现有文献概述了多模态数据集的广泛可用性和越来越多的大规模模型能够处理多模态感知任务，但同时采用了有限的标准解决方案。这促使我们探索更多广泛的 (i) 阶段技术。为此，本文首次尝试为多模态推荐系统提供大规模基准测试，特别关注多模态特征提取器。具体而言，我们利用两个流行且最近的框架——Ducho 和 Elliot——来提供一个统一且即用的实验环境，该环境能够运行利用新型多模态特征提取器的广泛基准测试分析。结果在不同超参数设置下得到了充分验证，为训练和调整下一代多模态推荐算法提供了重要的见解。
代码仓库
基准测试
| 基准 | 方法 | 指标 |
|---|---|---|
| multimodal-recommendation-on-amazon-baby | VBPR (ResNet50 + Sentence Bert) | Hit Ratio: 10.18 Recall: 6.21 nDCG: 2.99 |
| multimodal-recommendation-on-amazon-baby | FREEDOM (CLIP) | Hit Ratio: 14.45 Recall: 8.95 nDCG: 4.36 |
| multimodal-recommendation-on-amazon-baby | ItemKNN | Hit Ratio: 4.21 Recall: 2.46 nDCG: 1.19 |
| multimodal-recommendation-on-amazon-baby | LATTICE (MMFashion + Sentence Bert) | Hit Ratio: 13.63 Recall: 8.38 nDCG: 4.13 |
| multimodal-recommendation-on-amazon-baby | NGCF | Hit Ratio: 8.59 Recall: 5.09 nDCG: 2.39 |
| multimodal-recommendation-on-amazon-baby | NGCF-M (ResNet50 + Sentence Bert) | Hit Ratio: 11.91 Recall: 7.18 nDCG: 3.50 |
| multimodal-recommendation-on-amazon-baby | BPRMF | Hit Ratio: 9.04 Recall: 5.48 nDCG: 2.67 |
| multimodal-recommendation-on-amazon-baby | NGCF-M (Align) | Hit Ratio: 12.61 Recall: 7.70 nDCG: 3.66 |
| multimodal-recommendation-on-amazon-baby | VBPR (MMFashion + Sentence Bert) | Hit Ratio: 10.39 Recall: 6.42 nDCG: 3.12 |
| multimodal-recommendation-on-amazon-baby | LATTICE (ResNet50 + Sentence Bert) | Hit Ratio: 13.69 Recall: 8.41 nDCG: 4.06 |
| multimodal-recommendation-on-amazon-baby | BM3 (ResNet50 + Sentence Bert) | Hit Ratio: 13.29 Recall: 8.05 nDCG: 3.91 |
| multimodal-recommendation-on-amazon-baby | GRCN (Align) | Hit Ratio: 8.76 Recall: 5.21 nDCG: 2.43 |
| multimodal-recommendation-on-amazon-baby | GRCN (ResNet50 + Sentence Bert) | Hit Ratio: 8.81 Recall: 5.29 nDCG: 2.48 |
| multimodal-recommendation-on-amazon-baby | DGCF | Hit Ratio: 10.26 Recall: 6.08 nDCG: 3.03 |
| multimodal-recommendation-on-amazon-baby | SGL | Hit Ratio: 9.40 Recall: 5.77 nDCG: 2.93 |
| multimodal-recommendation-on-amazon-baby | FREEDOM (ResNet50 + Sentence Bert) | Hit Ratio: 14.28 Recall: 8.81 nDCG: 4.31 |
| multimodal-recommendation-on-amazon-baby | LightGCN | Hit Ratio: 12.60 Recall: 7.56 nDCG: 3.82 |
| multimodal-recommendation-on-amazon-baby | BM3 (AltCLIP) | Hit Ratio: 13.53 Recall: 8.15 nDCG: 4.10 |
| multimodal-recommendation-on-amazon-beauty | FREEDOM (ResNet50 + Sentence Bert) | Hit Ratio: 21.11 Recall: 13.85 nDCG: 7.24 |
| multimodal-recommendation-on-amazon-beauty | NGCF-M (MMFashion + Sentecen Bert) | Hit Ratio: 18.22 Recall: 11.93 nDCG: 6.21 |
| multimodal-recommendation-on-amazon-beauty | SGL | Hit Ratio: 18.17 Recall: 11.82 nDCG: 6.50 |
| multimodal-recommendation-on-amazon-beauty | ItemKNN | Hit Ratio: 10.89 Recall: 6.97 nDCG: 3.85 |
| multimodal-recommendation-on-amazon-beauty | LightGCN | Hit Ratio: 19.03 Recall: 12.30 nDCG: 6.42 |
| multimodal-recommendation-on-amazon-beauty | VBPR (AltCLIP) | Hit Ratio: 18.19 Recall: 11.94 nDCG: 6.15 |
| multimodal-recommendation-on-amazon-beauty | GRCN (ALIGN) | Hit Ratio: 16.09 Recall: 10.26 nDCG: 5.15 |
| multimodal-recommendation-on-amazon-beauty | VBPR (ResNet50 + Sentence Bert) | Hit Ratio: 17.64 Recall: 11.54 nDCG: 6.08 |
| multimodal-recommendation-on-amazon-beauty | BM3 (ALIGN) | Hit Ratio: 18.04 Recall: 11.67 nDCG: 6.04 |
| multimodal-recommendation-on-amazon-beauty | NGCF | Hit Ratio: 16.21 Recall: 10.42 nDCG: 5.27 |
| multimodal-recommendation-on-amazon-beauty | GRCN (ResNet50 + Sentence Bert) | Hit Ratio: 14.89 Recall: 9.57 nDCG: 4.83 |
| multimodal-recommendation-on-amazon-beauty | NGCF-M (ResNet50 + Sentence Bert) | Hit Ratio: 18.12 Recall: 11.72 nDCG: 6.11 |
| multimodal-recommendation-on-amazon-beauty | DGCF | Hit Ratio: 16.21 Recall: 10.42 nDCG: 5.27 |
| multimodal-recommendation-on-amazon-beauty | LATTICE (ALIGN) | Hit Ratio: 21.31 Recall: 13.93 nDCG: 7.21 |
| multimodal-recommendation-on-amazon-beauty | BM3 (ResNet50 + Sentence Bert) | Hit Ratio: 17.65 Recall: 11.28 nDCG: 5.83 |
| multimodal-recommendation-on-amazon-beauty | BPRMF | Hit Ratio: 16.55 Recall: 10.72 nDCG: 5.36 |
| multimodal-recommendation-on-amazon-beauty | LATTICE (ResNet50 + Sentence Bert) | Hit Ratio: 20.65 Recall: 13.44 nDCG: 7.03 |
| multimodal-recommendation-on-amazon-beauty | FREEDOM (MMFashion + Sentecen Bert) | Hit Ratio: 21.18 Recall: 13.87 nDCG: 7.17 |
| multimodal-recommendation-on-amazon-digital | VBPR (ResNet50 + Sentence Bert) | Hit Ratio: 43.54 Recall: 28.37 nDCG: 15.22 |
| multimodal-recommendation-on-amazon-digital | FREEDOM (ResNet50 + Sentence Bert) | Hit Ratio: 43.46 Recall: 29.05 nDCG: 16.15 |
| multimodal-recommendation-on-amazon-digital | ItemKNN | Hit Ratio: 34.51 Recall: 21.74 nDCG: 12.00 |
| multimodal-recommendation-on-amazon-digital | GRCN (ResNet50 + Sentence Bert) | Hit Ratio: 36.25 Recall: 22.88 nDCG: 12.17 |
| multimodal-recommendation-on-amazon-digital | SGL | Hit Ratio: 40.81 Recall: 27.09 nDCG: 15.03 |
| multimodal-recommendation-on-amazon-digital | DGCF | Hit Ratio: 40.46 Recall: 26.47 nDCG: 14.46 |
| multimodal-recommendation-on-amazon-digital | NGCF | Hit Ratio: 40.14 Recall: 26.46 nDCG: 14.58 |
| multimodal-recommendation-on-amazon-digital | LightGCN | Hit Ratio: 43.19 Recall: 28.66 nDCG: 14.95 |
| multimodal-recommendation-on-amazon-digital | LATTICE (ResNet50 + Sentence Bert) | Hit Ratio: 43.60 Recall: 29.40 nDCG: 16.07 |
| multimodal-recommendation-on-amazon-digital | BRMF | Hit Ratio: 41.13 Recall: 27.32 nDCG: 14.94 |
| multimodal-recommendation-on-amazon-digital | NGCF-M (ResNet50 + Sentence Bert) | Hit Ratio: 41.91 Recall: 27.84 nDCG: 15.35 |
| multimodal-recommendation-on-amazon-digital | BM3 (ResNet50 + Sentence Bert) | Hit Ratio: 41.42 Recall: 27.07 nDCG: 14.34 |
| multimodal-recommendation-on-amazon-office | NGCF | Hit Ratio: 19.62 Recall: 11.05 nDCG: 5.45 |
| multimodal-recommendation-on-amazon-office | DGCF | Hit Ratio: 20.89 Recall: 12.19 nDCG: 5.89 |
| multimodal-recommendation-on-amazon-office | LightGCN | Hit Ratio: 23.95 Recall: 13.99 nDCG: 6.93 |
| multimodal-recommendation-on-amazon-office | VBPR (CLIP) | Hit Ratio: 22.10 Recall: 12.78 nDCG: 6.23 |
| multimodal-recommendation-on-amazon-office | NGCF-M (ResNet50+ Sentence Bert) | Hit Ratio: 24.04 Recall: 14.35 nDCG: 7.14 |
| multimodal-recommendation-on-amazon-office | ItemKNN | Hit Ratio: 20.33 Recall: 11.35 nDCG: 5.76 |
| multimodal-recommendation-on-amazon-office | BPRMF | Hit Ratio: 19.70 Recall: 11.28 nDCG: 5.35 |
| multimodal-recommendation-on-amazon-office | LATTICE (ResNet50+ Sentence Bert) | Hit Ratio: 25.79 Recall: 15.75 nDCG: 7.71 |
| multimodal-recommendation-on-amazon-office | SGL | Hit Ratio: 20.49 Recall: 11.85 nDCG: 5.89 |
| multimodal-recommendation-on-amazon-office | LATTICE (ALIGN) | Hit Ratio: 25.65 Recall: 15.71 nDCG: 7.63 |
| multimodal-recommendation-on-amazon-office | BM3 (ResNet50+ Sentence Bert) | Hit Ratio: 22.5 Recall: 13.13 nDCG: 6.42 |
| multimodal-recommendation-on-amazon-office | VBPR (ResNet50+ Sentence Bert) | Hit Ratio: 22.01 Recall: 12.83 nDCG: 6.18 |
| multimodal-recommendation-on-amazon-office | FREEDOM (CLIP) | Hit Ratio: 25.88 Recall: 15.64 nDCG: 7.66 |
| multimodal-recommendation-on-amazon-office | FREEDOM (ResNet50+ Sentence Bert) | Hit Ratio: 23.59 Recall: 15.58 nDCG: 7.57 |
| multimodal-recommendation-on-amazon-office | GRCN (CLIP) | Hit Ratio: 22.32 Recall: 13.10 nDCG: 6.47 |
| multimodal-recommendation-on-amazon-office | NGCF-M (CLIP) | Hit Ratio: 24.85 Recall: 14.99 nDCG: 7.43 |
| multimodal-recommendation-on-amazon-office | BM3 (ALIGN) | Hit Ratio: 23.40 Recall: 13.84 nDCG: 6.75 |
| multimodal-recommendation-on-amazon-office | GRCN (ResNet50+ Sentence Bert) | Hit Ratio: 21.20 Recall: 12.31 nDCG: 6.08 |
| multimodal-recommendation-on-amazon-toys | SGL | Hit Ratio: 16.68 Recall: 10.76 nDCG: 5.93 |
| multimodal-recommendation-on-amazon-toys | FREEDOM (MMFashion + Sentence Bert) | Hit Ratio: 20.70 Recall: 13.73 nDCG: 7.10 |
| multimodal-recommendation-on-amazon-toys | VBPR (ALIGN) | Hit Ratio: 16.86 Recall: 11.06 nDCG: 5.85 |
| multimodal-recommendation-on-amazon-toys | GRCN (ALIGN) | Hit Ratio: 15.35 Recall: 9.94 nDCG: 5.07 |
| multimodal-recommendation-on-amazon-toys | FREEDOM (ResNet50 + Sentence Bert) | Hit Ratio: 20.64 Recall: 13.67 nDCG: 7.04 |
| multimodal-recommendation-on-amazon-toys | DGCF | Hit Ratio: 14.71 Recall: 9.43 nDCG: 5.12 |
| multimodal-recommendation-on-amazon-toys | LightGCN | Hit Ratio: 16.63 Recall: 10.59 nDCG: 5.58 |
| multimodal-recommendation-on-amazon-toys | GRCN (ResNet50 + Sentence Bert) | Hit Ratio: 15.00 Recall: 9.67 nDCG: 9.675.07 |
| multimodal-recommendation-on-amazon-toys | NGCF-M (ResNet50 + Sentence Bert) | Hit Ratio: 16.73 Recall: 10.85 nDCG: 5.73 |
| multimodal-recommendation-on-amazon-toys | BPRMF | Hit Ratio: 14.75 Recall: 9.51 nDCG: 5.02 |
| multimodal-recommendation-on-amazon-toys | BM3 (ALIGN) | Hit Ratio: 15.78 Recall: 10.07 nDCG: 5.24 |
| multimodal-recommendation-on-amazon-toys | VBPR (ResNet50 + Sentence Bert) | Hit Ratio: 16.54 Recall: 10.83 nDCG: 5.70 |
| multimodal-recommendation-on-amazon-toys | LATTICE (ResNet50 + Sentence Bert) | Hit Ratio: 18.95 Recall: 12.42 nDCG: 6.45 |
| multimodal-recommendation-on-amazon-toys | NGCF-M (ALIGN) | Hit Ratio: 17.16 Recall: 11.12 nDCG: 5.80 |
| multimodal-recommendation-on-amazon-toys | LATTICE (ALIGN) | Hit Ratio: 19.27 Recall: 12.73 nDCG: 6.64 |
| multimodal-recommendation-on-amazon-toys | NGCF | Hit Ratio: 14.44 Recall: 9.24 nDCG: 4.87 |
| multimodal-recommendation-on-amazon-toys | ItemKNN | Hit Ratio: 11.06 Recall: 6.97 nDCG: 3.91 |
| multimodal-recommendation-on-amazon-toys | BM3 (ResNet50 + Sentence Bert) | Hit Ratio: 15.56 Recall: 9.94 nDCG: 5.14 |