One-sentence Summary

Researchers from the University of Science and Technology of China, Tencent, and other institutions propose a complete evaluate-diagnose-optimize framework, introducing the SETWISEEVALKIT benchmark and RUBRIC4SETWISE, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving state-of-the-art downstream generation with fewer documents across both short-form and long-form scenarios.

Key Contributions

• SETWISEEVALKIT is a three-level, nine-dimension document set evaluation benchmark covering short-form and long-form scenarios, providing approximately 28K high-quality evaluation rubrics.
• Systematic evaluation of 12 rerankers reveals that the best method achieves at most 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings.
• RUBRIC4SETWISE is a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation with fewer documents and search rounds and remaining the only method state-of-the-art in both scenarios.

Introduction

The growing adoption of Retrieval-Augmented Generation (RAG) means that retrieved document sets are fed directly into large language models, so the quality of the set determines the upper bound of generation. Existing evaluation benchmarks, however, treat documents in isolation, scoring only relevance and aggregating with nDCG, which ignores critical inter-document interactions such as redundancy, conflict, and complementarity. The authors address this gap with SETWISEEVALKIT, a multi‑dimensional evaluation framework that assesses document sets across three granularity levels and nine dimensions, and they demonstrate that its rubric‑based signals can be converted into a training‑free selection method (RUBRIC4SETWISE) that achieves better generation with fewer documents and search rounds.

Dataset

The authors build SETWISEEVALKIT as an evaluation dataset for assessing document set quality, not for model training. It provides query-answer pairs together with multi-dimensional rubrics that judge how well a set of documents supports answering the query.

Dataset Composition and Sources

• Short-form subset: 2,061 QA samples drawn from multi-hop QA datasets.
  • Sources: HotpotQA, 2WikiMultihopQA, MuSiQue (2,000 each) and Bamboogle (all 125 instances).
  • Candidate retrieval: For each query, BM25 retrieves the top-20 Wikipedia documents as a candidate pool. Queries where BM25 fails to retrieve sufficiently relevant documents are discarded, leaving 2,061 high-quality instances.
• Long-form subset: 200 samples randomly selected from ResearchQA.
  • Candidate retrieval: An open-source search agent (DR.Tulu-8B) uses the Google Search API in multi-turn trajectories; rerankers are wrapped as tools and applied after each retrieval turn.

Rubric Construction (Metadata)

• Each query-answer pair is paired with a set of evaluation rubrics covering nine dimensions grouped into three levels:
  • Document-level: Relevance, Authenticity, Quality.
  • Set-level: Complementarity, Redundancy, Conflict.
  • Global-level: Completeness, Density, Reachability.
• Generation process: Two frontier LLMs (GPT 5.1 and Gemini 3.1-Pro-Preview) independently generate candidate rubric questions that must reference specific entities, facts, or values from the query and answer. A third model (DeepSeek-V4 Pro) aggregates and filters these candidates, removing duplicates and low-quality items.
• Resulting rubric counts: around 24,000 for the short-form subset, 4,000 for the long-form subset.

How the Data is Used

• The dataset is used to evaluate document sets produced by different rerankers. For a given query, multiple rerankers each select a subset from the candidate pool, and the rubric system measures set quality holistically rather than scoring documents independently.
• Human validation (50 random samples per scenario) confirms that the aggregated rubrics are highly discriminative: ~70% are rated Highly or Critically discriminative, and ~77% cause Moderate or Significant ranking changes when removed, verifying that the rubrics reliably distinguish good from poor document sets.
• There is no training split; the data serves purely as an evaluation benchmark for document-set-quality metrics.

Experiment

The evaluation setup defines a multi-dimensional rubric framework covering document-level, set-level, and global-level quality, applied to short-form multi-hop QA and long-form multi-turn search scenarios using LLM-based rubric generation and judging. Experiments validate that setwise reranking methods dominate in short-form settings while reasoning-enhanced methods lead in long-form, yet all rerankers struggle with inter-document coordination dimensions like complementarity and completeness. The proposed training-free set selection method, RUBRIC4SETWISE, uses rubrics as selection signals and achieves the best downstream generation performance in both scenarios while requiring fewer documents. A case study further demonstrates that rubric-based assessment captures set-level coordination weaknesses that traditional pairwise relevance metrics miss.

All existing retrieval evaluation benchmarks operate at the document level, in a single round, and cover only the relevance dimension. They lack set-level granularity, multi-round assessment, and any cross-document dimensions such as complementarity, redundancy, or conflict, leaving a structural blind spot in retrieval evaluation. Every benchmark (TREC-DL, BEIR, MTEB, RAGAS, ARES, BRIGHT) targets document-level scoring with nDCG or LLM-judge, ignoring how documents interact within a set. Only the relevance dimension is covered; dimensions like complementarity, redundancy, conflict, completeness, and density are entirely absent from existing evaluation frameworks. No benchmark supports multi-round or set-level evaluation, making them unable to capture quality characteristics that matter for downstream generation.

Ad-hoc rerankers improve on BM25-only retrieval but still score poorly on set-level coordination dimensions such as Complementarity and Completeness, with Conflict being the only dimension where they are relatively strong. Setwise reranking methods SetR and Rank4Gen lead short-form performance, yet no reranker performs consistently well across all rubric dimensions, revealing a gap in generalizing between short-form and long-form scenarios. Setwise reranking methods SetR and Rank4Gen achieve the highest overall short-form scores, outperforming all ad-hoc and reasoning-enhanced rerankers. Ad-hoc rerankers offer modest gains over BM25 retrieval but exhibit pronounced weaknesses in cross-document coordination dimensions like Complementarity and Completeness, while Conflict remains the highest-scoring set-level dimension. All evaluated rerankers underperform on most dimensions except Conflict, and no single method excels across both short-form and long-form settings, indicating a generalization deficiency.

In the long-form scenario, reasoning-enhanced rerankers achieve the highest overall coverage scores, outperforming both adhoc and setwise approaches. All evaluated methods score poorly across most dimensions, with the conflict dimension being the only one where scores are relatively high, while set-level coordination dimensions such as complementarity and completeness remain notably deficient. Reasoning-enhanced rerankers lead in the long-form scenario, with setwise methods dominating in the short-form setting, revealing a generalization gap where no single approach excels in both scenarios. Conflict scores are consistently high across all rerankers (ranging from 86.49 to 90.77), while complementarity scores remain low (ranging from 23.41 to 25.20), highlighting a systematic failure to model cross-document coordination. The strong positive correlation (Pearson r=0.92) between overall coverage scores and downstream generation performance validates that higher rubric coverage translates to better answer quality.

RUBRIC4SETWISE, a rubric-guided document selection method, achieves the best downstream answer performance across both short-form and long-form scenarios, outperforming all existing rerankers. It selects compact yet sufficient document subsets, using far fewer documents than the fixed top-5 of conventional rerankers in short-form tasks and substantially fewer unique documents than the second-best method in long-form tasks, while improving answer quality. In the short-form scenario, RUBRIC4SETWISE achieves the highest exact match and F1 scores, surpassing the next-best SetR by clear margins, while using on average only 2.66 documents, far fewer than the standard five documents used by other rerankers. In the long-form scenario, RUBRIC4SETWISE attains the highest LLM-judge score, outperforming ReasonRank and Rearank, and uses only 20.52 unique documents, notably fewer than SetR's 29.23.

Existing retrieval benchmarks evaluate only document-level relevance in a single round, ignoring set-level dimensions like complementarity and conflict. Experiments show that ad-hoc, setwise, and reasoning-enhanced rerankers all struggle with cross-document coordination, and no single method generalizes across short-form and long-form settings. A rubric-guided selection method, RUBRIC4SETWISE, consistently achieves the best downstream answer quality in both scenarios while using far fewer documents.