Command Palette
Search for a command to run...
Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications
Tornike Tsereteli; Yavuz Selim Kartal; Simone Paolo Ponzetto; Andrea Zielinski; Kai Eckert; Philipp Mayr
Abstract
In this paper, we provide an overview of the SV-Ident shared task as part of the 3rd Workshop on Scholarly Document Processing (SDP) at COLING 2022. In the shared task, participants were provided with a sentence and a vocabulary of variables, and asked to identify which variables, if any, are mentioned in individual sentences from scholarly documents in full text. Two teams made a total of 9 submissions to the shared task leaderboard. While none of the teams improve on the baseline systems, we still draw insights from their submissions. Furthermore, we provide a detailed evaluation. Data and baselines for our shared task are freely available at https://github.com/vadis-project/sv-ident
Code Repositories
Benchmarks
| Benchmark | Methodology | Metrics |
|---|---|---|
| variable-detection-on-sv-ident | Sentence-T5 | F1: 60.17 |
| variable-detection-on-sv-ident | SsciBERT | F1: 66.1 |
| variable-disambiguation-on-sv-ident | BM25 | mAP@10: 9.43 |
| variable-disambiguation-on-sv-ident | SPARTA | mAP@10: 11.27 |
| variable-disambiguation-on-sv-ident | Sentence-T5 | mAP@10: 13.59 |
| variable-disambiguation-on-sv-ident | sentence-transformers/distiluse-base-multilingual-cased-v1 | mAP@10: 18.93 |
Build AI with AI
From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.