摘要
深度神经网络模型（如Transformer）往往会自然地学习到虚假特征，这些特征在输入与标签之间建立起“捷径”，从而损害模型的泛化能力和鲁棒性。本文将自注意力机制推进至其鲁棒性变体，应用于基于Transformer的预训练语言模型（如BERT）。我们提出了对抗性自注意力机制（Adversarial Self-Attention, ASA），该机制通过对抗性地偏置注意力分布，有效抑制模型对特定特征（如关键词）的依赖，同时鼓励模型探索更广泛的语义信息。我们在预训练和微调两个阶段的多种任务上进行了全面评估。在预训练阶段，相较于常规训练，ASA在更长训练步数下展现出显著的性能提升；在微调阶段，ASA增强的模型在泛化能力与鲁棒性方面均大幅超越传统模型。
代码仓库
gingasan/adversarialsa
官方
pytorch
GitHub 中提及
基准测试
| 基准 | 方法 | 指标 |
|---|---|---|
| machine-reading-comprehension-on-dream | ASA + RoBERTa | Accuracy: 69.2 |
| machine-reading-comprehension-on-dream | ASA + BERT-base | Accuracy: 64.3 |
| named-entity-recognition-on-wnut-2017 | ASA + RoBERTa | F1: 57.3 |
| named-entity-recognition-on-wnut-2017 | ASA + BERT-base | F1: 49.8 |
| natural-language-inference-on-multinli | ASA + BERT-base | Matched: 85 |
| natural-language-inference-on-multinli | ASA + RoBERTa | Matched: 88 |
| natural-language-inference-on-qnli | ASA + RoBERTa | Accuracy: 93.6% |
| natural-language-inference-on-qnli | ASA + BERT-base | Accuracy: 91.4% |
| paraphrase-identification-on-quora-question | ASA + BERT-base | F1: 72.3 |
| paraphrase-identification-on-quora-question | ASA + RoBERTa | F1: 73.7 |
| semantic-textual-similarity-on-sts-benchmark | ASA + RoBERTa | Spearman Correlation: 0.892 |
| semantic-textual-similarity-on-sts-benchmark | ASA + BERT-base | Spearman Correlation: 0.865 |
| sentiment-analysis-on-sst-2-binary | ASA + BERT-base | Accuracy: 94.1 |
| sentiment-analysis-on-sst-2-binary | ASA + RoBERTa | Accuracy: 96.3 |