SCBench is a long-context benchmark released by Microsoft in 2024, focusing on the evaluation of efficient long-context methods. The related paper is available at SCBench: A KV cache-centric analysis of long-context methods, aiming to comprehensively evaluate the performance of long-context methods in real-world scenarios from a KV cache-centric perspective.

The dataset addresses scenarios where context memory (KV cache) is shared and reused across multiple requests, analyzing method performance throughout the complete KV cache lifecycle (generation, compression, retrieval, and loading). The data includes various task subsets, covering code repository question answering, multi-turn dialogue, summary generation, and selection tasks. All data is standardized into a format containing context, multi-turn question-answer pairs, and unique identifiers, used to test the effectiveness of different attention mechanisms and KV cache strategies.

Dataset Composition

The dataset primarily includes the following subsets:

• scbench_choice_eng: 58 samples, English multi-turn selection tasks.
• scbench_kv: 100 samples, basic KV cache tests.
• scbench_many_shot: 54 samples, few-shot learning tasks.
• scbench_mf: 100 samples, multi-turn dialogue tasks.
• scbench_prefix_suffix: 100 samples, prefix and suffix context tasks.
• scbench_qa_chn: 35 samples, Chinese question-answering tasks.
• scbench_qa_eng: 69 samples, English question-answering tasks.
• scbench_repoqa: 88 samples, code repository question-answering tasks.
• scbench_repoqa_and_kv: 88 samples, tasks combining code repository question answering with KV cache analysis.
• scbench_summary: 70 samples, text summarization tasks.
• scbench_summary_with_needles: 70 samples, summarization tasks involving key information retrieval.
• scbench_vt: 90 samples, specific validation tasks.

The dataset primarily includes the following fields:

• id: Unique identifier for the sample.
• context: Long context content, such as code repositories, documents, or long texts.
• multi_turns: List of multi-turn interactions, containing the input and answer for each turn.
• lang: Language information, included in some subsets.
• repo: Information related to code repositories, primarily used for code repository question-answering tasks.
• task: Task type or related information, included in some subsets.

Citation

@article{li2024scbench,
    title={SCBench: A KV cache-centric analysis of long-context methods},
    author={Li, Yucheng and Jiang, Huiqiang and Wu, Qianhui and Luo, Xufang and Ahn, Surin and Zhang, Chengruidong and Abdi, Amir H and Li, Dongsheng and Gao, Jianfeng and Yang, Yuqing and Qiu, Lili},
    journal={arXiv preprint arXiv:2412.10319},
    year={2024}
}