摘要
人类能够快速将不同刺激关联起来，从而在新情境中解决问题。我们提出了一种新型神经网络模型，该模型能够学习事实的状态表示，并通过组合这些表示实现此类关联推理。为此，我们在LSTM模型中引入了一种称为快速权重记忆（Fast Weight Memory, FWM）的关联记忆机制。在输入序列的每一步，通过可微分的操作，LSTM能够动态更新并维护存储在快速变化的FWM权重中的组合性关联。该模型通过梯度下降实现端到端训练，在组合性语言推理任务、部分可观测马尔可夫决策过程（POMDPs）的元强化学习以及小规模词汇级语言建模任务中均表现出色。
代码仓库
ischlag/Fast-Weight-Memory-public
官方
pytorch
基准测试
| 基准 | 方法 | 指标 |
|---|---|---|
| language-modelling-on-penn-treebank-word | AWD-FWM Schlag et al. (2020) | Params: 24M Test perplexity: 54.48 Validation perplexity: 56.76 |
| language-modelling-on-wikitext-2 | AWD-FWM Schlag et al. (2020) | Number of params: 37M Test perplexity: 61.65 Validation perplexity: 54.48 |
| question-answering-on-catbabi-lm-mode | AWD-LSTM | Accuracy (mean): 80.15% |
| question-answering-on-catbabi-lm-mode | Metalearned Neural Memory (plastic) | Accuracy (mean): 69.3% |
| question-answering-on-catbabi-lm-mode | Fast Weight Memory | Accuracy (mean): 93.04% |
| question-answering-on-catbabi-lm-mode | AWD-Transformer XL | Accuracy (mean): 90.23% |
| question-answering-on-catbabi-qa-mode | Metalearned Neural Memory (plastic) | 1:1 Accuracy: 88.97% |
| question-answering-on-catbabi-qa-mode | AWD-LSTM | 1:1 Accuracy: 80.88% |
| question-answering-on-catbabi-qa-mode | AWD-Transformer XL | 1:1 Accuracy: 87.66% |
| question-answering-on-catbabi-qa-mode | Fast Weight Memory | 1:1 Accuracy: 96.75% |