3 个月前

Transformer 是短文本分类器:在基准数据集与真实世界数据集上的归纳式短文本分类器研究

Transformer 是短文本分类器:在基准数据集与真实世界数据集上的归纳式短文本分类器研究

摘要

短文本分类是自然语言处理中一个关键且具有挑战性的研究方向。为此,学术界已开发出大量高度专门化的短文本分类模型。然而,在近期的短文本研究中,传统文本分类任务中的前沿方法——尤其是纯Transformer架构——尚未得到充分挖掘与应用。本文系统评估了多种短文本分类器的性能,同时对比了表现最佳的传统文本分类器。此外,我们还基于两个全新的真实世界短文本数据集展开实验,旨在缓解研究过度依赖特征有限的基准数据集所带来的问题。实验结果明确表明,Transformer模型在短文本分类任务中已达到当前最优(SOTA)的准确率,从而引发了一个重要问题:专门针对短文本设计的技术是否仍为必要?

代码仓库

基准测试

基准方法指标
text-classification-on-mrBERT
Accuracy: 86.94
text-classification-on-mrALBERTv2
Accuracy: 86.02
text-classification-on-mrERNIE 2.0
Accuracy: 88.97
text-classification-on-mrERNIE 2.0 (optimized)
Accuracy: 89.53
text-classification-on-mrDistilBERT
Accuracy: 85.31
text-classification-on-mrRoBERTa
Accuracy: 89.42
text-classification-on-mrDeBERTa
Accuracy: 90.21
text-classification-on-nice-2RoBERTa
Accuracy: 99.76
text-classification-on-nice-45BERT
Accuracy: 72.79
text-classification-on-r8SGNN
Accuracy: 98.09
text-classification-on-r8C-BERT (ESGNN + BERT)
Accuracy: 98.28
text-classification-on-r8ESGNN
Accuracy: 98.23
text-classification-on-r8DistilBERT
Accuracy: 97.981
text-classification-on-r8fastText
Accuracy: 96.13
text-classification-on-r8WideMLP
Accuracy: 96.98
text-classification-on-r8ERNIE 2.0
Accuracy: 98.041
text-classification-on-r8DeBERTa
Accuracy: 98.451
text-classification-on-r8BERT
Accuracy: 98.171
text-classification-on-r8ALBERTv2
Accuracy: 97.62
text-classification-on-searchsnippetsBERT
Accuracy: 88.2
text-classification-on-searchsnippetsDistilBERT
Accuracy: 89.69
text-classification-on-sst-2BERT
Accuracy: 91.37
text-classification-on-sst-2DeBERTa
Accuracy: 94.78
text-classification-on-stops-2ERNIE 2.0
STOPS-2: 99.88
text-classification-on-stops-41DeBERTa
Accuracy: 89.73
text-classification-on-trec-10BERT
Accuracy: 99.40
text-classification-on-twitterBERT
Accuracy: 99.96
text-classification-on-twitterDistilBERT
Accuracy: 99.96
text-classification-on-twitterERNIE 2.0
Accuracy: 99.97

用 AI 构建 AI

从想法到上线——通过免费 AI 协同编程、开箱即用的环境和市场最优价格的 GPU 加速您的 AI 开发

AI 协同编程
即用型 GPU
最优价格
立即开始

Hyper Newsletters

订阅我们的最新资讯
我们会在北京时间 每周一的上午九点 向您的邮箱投递本周内的最新更新
邮件发送服务由 MailChimp 提供
Transformer 是短文本分类器:在基准数据集与真实世界数据集上的归纳式短文本分类器研究 | 论文 | HyperAI超神经