摘要
我们提出了HowSumm，这是一个面向查询聚焦型多文档摘要（query-focused multi-document summarization, qMDS）任务的新型大规模数据集，旨在从一组源文档中生成可操作的指导性内容。该应用场景不同于现有多文档摘要（MDS）数据集中涵盖的典型用例，具有广泛适用于教育与工业场景的潜力。HowSumm通过自动化方法构建，其数据源自wikiHow网站的文章及其引用的原始资料，并结合了已有高质量人工标注qMDS数据集的统计特征进行设计与筛选。本文详细阐述了该数据集的构建过程，并讨论了其在结构与语义层面区别于其他摘要语料库的独特属性。针对该数据集开展的自动评估与人工评估结果表明，无论是抽取式还是生成式摘要模型，均仍有显著的改进空间。
代码仓库
odelliab/HowSumm
官方
GitHub 中提及
基准测试
| 基准 | 方法 | 指标 |
|---|---|---|
| document-summarization-on-howsumm-method | CES (query: method + article titles) | ROUGE-1: 48.3 |
| document-summarization-on-howsumm-method | LexRank (query: method + article titles) | ROUGE-1: 47.1 |
| document-summarization-on-howsumm-method | CES (query: method + article + steps titles) | ROUGE-1: 52.2 |
| document-summarization-on-howsumm-method | LexRank (query: method title) | ROUGE-1: 47.7 |
| document-summarization-on-howsumm-method | GreedyRel (query: method title) | ROUGE-1: 43.4 |
| document-summarization-on-howsumm-method | CES (query: method title) | ROUGE-1: 48.4 |
| document-summarization-on-howsumm-method | GreedyRel (query: method + article titles) | ROUGE-1: 42.3 |
| document-summarization-on-howsumm-method | LexRank (query: method + article + steps titles) | ROUGE-1: 53.5 |
| document-summarization-on-howsumm-method | GreedyRel (query: method + article + steps titles) | ROUGE-1: 48.6 |
| document-summarization-on-howsumm-step | BM25-HierSumm (query: step + method + article titles) | ROUGE-1: 21.9 |
| document-summarization-on-howsumm-step | LexRank (query: step + method titles) | ROUGE-1: 38.2 |
| document-summarization-on-howsumm-step | GreedyRel (query: step title) | ROUGE-1: 30.1 |
| document-summarization-on-howsumm-step | LexRank (query: step title) | ROUGE-1: 39.6 |
| document-summarization-on-howsumm-step | GreedyRel (query: step + method titles) | ROUGE-1: 30.3 |
| document-summarization-on-howsumm-step | LexRank (query: step + method + article titles) | ROUGE-1: 36.3 |
| document-summarization-on-howsumm-step | CES (query: step title) | ROUGE-1: 39.3 |
| document-summarization-on-howsumm-step | CES (query: step + method + article titles) | ROUGE-1: 37.0 |
| document-summarization-on-howsumm-step | CES (query: step + method titles) | ROUGE-1: 38.3 |
| document-summarization-on-howsumm-step | BM25-HierSumm (query: step + method titles) | ROUGE-1: 23.0 |
| document-summarization-on-howsumm-step | BM25-HierSumm (query: step title) | ROUGE-1: 22.3 |