摘要
我们提出了WiC-TSV，这是一个面向词义消歧（Word Sense Disambiguation）的新一代多领域评估基准。具体而言，我们引入了一种面向上下文中的目标词义验证（Target Sense Verification of Words in Context）的框架，其独特性体现在将任务建模为二分类问题，因而不依赖外部词义词典（sense inventories），同时覆盖了多种领域。这一设计使该数据集在评估跨领域及不同类型的模型与系统时具有高度灵活性。WiC-TSV提供了三种不同的评估设置，依据输入信号的类型而定。我们基于当前最先进的语言模型在该数据集上设立了基线性能。实验结果表明，尽管这些模型在该任务上表现尚可，但在跨领域（out-of-domain）场景下，机器性能与人类表现之间仍存在显著差距。WiC-TSV数据集可于 https://competitions.codalab.org/competitions/23683 获取。
代码仓库
semantic-web-company/wic-tsv
官方
pytorch
GitHub 中提及
基准测试
| 基准 | 方法 | 指标 |
|---|---|---|
| entity-linking-on-wic-tsv | Bert-base | Task 1 Accuracy: all: 75.3 Task 1 Accuracy: domain specific: 77.9 Task 1 Accuracy: general purpose: 73.3 Task 2 Accuracy: all: 71.7 Task 2 Accuracy: domain specific: 74.7 Task 2 Accuracy: general purpose: 68.6 Task 3 Accuracy: all: 76.6 Task 3 Accuracy: domain specific: 80.4 Task 3 Accuracy: general purpose: 73.5 |
| entity-linking-on-wic-tsv | Unsupervised Bert | Task 1 Accuracy: all: 54.4 Task 1 Accuracy: domain specific: 60.6 Task 1 Accuracy: general purpose: 49.2 Task 2 Accuracy: all: 62.8 Task 2 Accuracy: domain specific: 69.1 Task 2 Accuracy: general purpose: 57.6 Task 3 Accuracy: all: 60.5 Task 3 Accuracy: domain specific: 67.9 Task 3 Accuracy: general purpose: 54.4 |
| entity-linking-on-wic-tsv | All true | Task 1 Accuracy: all: 50.8 Task 1 Accuracy: domain specific: 47.0 Task 1 Accuracy: general purpose: 53.8 Task 2 Accuracy: all: 50.8 Task 2 Accuracy: domain specific: 47.0 Task 2 Accuracy: general purpose: 53.8 Task 3 Accuracy: all: 50.8 Task 3 Accuracy: domain specific: 47.0 Task 3 Accuracy: general purpose: 53.8 |
| entity-linking-on-wic-tsv | FastText | Task 1 Accuracy: all: 53.7 Task 1 Accuracy: domain specific: 50.6 Task 1 Accuracy: general purpose: 56.2 Task 2 Accuracy: all: 52.7 Task 2 Accuracy: domain specific: 47.7 Task 2 Accuracy: general purpose: 56.8 Task 3 Accuracy: all: 53.4 Task 3 Accuracy: domain specific: 49.0 Task 3 Accuracy: general purpose: 57.1 |
| entity-linking-on-wic-tsv | Human | Task 3 Accuracy: all: 85.3 Task 3 Accuracy: domain specific: 89.2 Task 3 Accuracy: general purpose: 82.1 |
| word-sense-disambiguation-on-wic-tsv | Bert-base | Task 1 Accuracy: all: 75.3 Task 1 Accuracy: domain specific: 77.9 Task 1 Accuracy: general purpose: 73.3 Task 2 Accuracy: all: 71.7 Task 2 Accuracy: domain specific: 74.7 Task 2 Accuracy: general purpose: 68.6 Task 3 Accuracy: all: 76.6 Task 3 Accuracy: domain specific: 80.4 Task 3 Accuracy: general purpose: 73.5 |
| word-sense-disambiguation-on-wic-tsv | Human | Task 3 Accuracy: all: 85.3 Task 3 Accuracy: domain specific: 89.2 Task 3 Accuracy: general purpose: 82.1 |
| word-sense-disambiguation-on-wic-tsv | FastText | Task 1 Accuracy: all: 53.7 Task 1 Accuracy: domain specific: 50.6 Task 1 Accuracy: general purpose: 56.2 Task 2 Accuracy: all: 52.7 Task 2 Accuracy: domain specific: 47.7 Task 2 Accuracy: general purpose: 56.8 Task 3 Accuracy: all: 53.4 Task 3 Accuracy: domain specific: 49.0 Task 3 Accuracy: general purpose: 57.1 |
| word-sense-disambiguation-on-wic-tsv | All true | Task 1 Accuracy: all: 50.8 Task 1 Accuracy: domain specific: 47.0 Task 1 Accuracy: general purpose: 53.8 Task 2 Accuracy: all: 50.8 Task 2 Accuracy: domain specific: 47.0 Task 2 Accuracy: general purpose: 53.8 Task 3 Accuracy: all: 50.8 Task 3 Accuracy: domain specific: 47.0 Task 3 Accuracy: general purpose: 53.8 |
| word-sense-disambiguation-on-wic-tsv | Unsupervised Bert | Task 1 Accuracy: all: 54.4 Task 1 Accuracy: domain specific: 60.6 Task 1 Accuracy: general purpose: 49.2 Task 2 Accuracy: all: 62.8 Task 2 Accuracy: domain specific: 69.1 Task 2 Accuracy: general purpose: 57.6 Task 3 Accuracy: all: 60.5 Task 3 Accuracy: domain specific: 67.9 Task 3 Accuracy: general purpose: 54.4 |