摘要
大型预训练语言模型（LMs）在少量样本学习方面展现了卓越的能力。然而，它们的成功很大程度上依赖于模型参数的扩展，这使得训练和部署变得极具挑战性。本文提出了一种新的方法，称为EFL，该方法可以将小型语言模型转化为更优秀的少量样本学习者。该方法的核心思想是将潜在的自然语言处理任务重新表述为蕴含任务，然后使用最少8个样例对模型进行微调。我们进一步展示了所提出的方法可以：(i) 自然地与基于无监督对比学习的数据增强方法结合；(ii) 轻松扩展到多语言少量样本学习。系统评估了18个标准自然语言处理任务的结果表明，该方法在各种现有的最先进（SOTA）少量样本学习方法基础上性能提升了12%，并且在仅有500倍于大型模型如GPT-3的规模下仍能取得具有竞争力的少量样本学习效果。
代码仓库
sunyilgdx/prompts4keras
tf
GitHub 中提及
cactilab/hateguard
GitHub 中提及
基准测试
| 基准 | 方法 | 指标 |
|---|---|---|
| linguistic-acceptability-on-cola | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 86.4% |
| natural-language-inference-on-qnli | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 94.5% |
| natural-language-inference-on-rte | RoBERTa-large 355M + EFL + UCA | Accuracy: 87.2% |
| natural-language-inference-on-rte | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 90.5% |
| natural-language-inference-on-snli | Neural Tree Indexers for Text Understanding | % Test Accuracy: 93.1 Parameters: 355 |
| natural-language-inference-on-snli | EFL (Entailment as Few-shot Learner) + RoBERTa-large | % Test Accuracy: 93.1 % Train Accuracy: ? Parameters: 355m |
| paraphrase-identification-on-quora-question | RoBERTa-large 355M + Entailment as Few-shot Learner | F1: 89.2 |
| question-answering-on-boolq | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 86.0 |
| semantic-textual-similarity-on-mrpc | RoBERTa-large 355M + Entailment as Few-shot Learner | F1: 91.0 |
| semantic-textual-similarity-on-sts-benchmark | RoBERTa-large 355M + Entailment as Few-shot Learner | Pearson Correlation: 0.918 |
| sentiment-analysis-on-cr | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 92.5 |
| sentiment-analysis-on-imdb | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 96.1 |
| sentiment-analysis-on-mpqa | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 90.8 |
| sentiment-analysis-on-mr | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 92.5 |
| sentiment-analysis-on-sst-2-binary | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 96.9 |
| subjectivity-analysis-on-subj | RoBERTa-large 355M + Entailment as Few-shot Learner | Accuracy: 97.1 |