One-sentence Summary

Yale University researchers introduce Spider, a large-scale, cross-domain text-to-SQL dataset comprising 10,181 questions and 5,693 complex SQL queries across 200 databases spanning 138 domains, which defines a challenging generalization task where state-of-the-art models achieve only $12.4\%$12.4% exact match accuracy.

Key Contributions

• The paper introduces Spider, a large-scale, cross-domain text-to-SQL dataset comprising 10,181 questions and 5,693 complex SQL queries across 200 multi-table databases in 138 domains, labeled by 11 college students.
• The paper defines a new semantic parsing task that requires models to generalize to both unseen SQL queries and unseen database schemas, using distinct train and test splits that prevent memorization of database-specific templates.
• Experimental evaluation of state-of-the-art models on this task shows a top exact matching accuracy of only 12.4% on the database split, establishing a strong baseline and highlighting the challenge for future research.

Introduction

Semantic parsing, the task of translating natural language into executable queries such as SQL, is key to building reliable natural language interfaces for databases. Prior benchmarks suffer from two main shortcomings: complex-query datasets are tiny and reuse the same target programs across train and test splits, allowing models to memorize templates instead of learning compositionality, while large-scale datasets like WikiSQL are limited to single-table schemas and simple SELECT–WHERE clauses. The authors present Spider, a large, cross-domain dataset of 10,181 questions and 5,693 corresponding complex SQL queries spanning 200 databases with multiple tables and foreign keys. They further define a database-level split evaluation where no database overlaps between training and testing, forcing models to generalize to entirely new schemas and query patterns. Baseline experiments show that standard neural approaches reach at most 12.4% exact match accuracy, highlighting the difficulty of genuine semantic parsing and leaving substantial room for improvement.

Dataset

The authors construct the Spider dataset, a large-scale text-to-SQL benchmark designed to test generalization across 138 domains and complex SQL queries. The dataset is built from multiple sources and refined through a rigorous human annotation pipeline, resulting in 200 unique relational databases with associated natural language questions and SQL query pairs.

• Database Sources and Creation

• 70 databases collected from college database courses, SQL tutorial websites, online CSV files, and textbook examples.

• 40 databases sourced from DatabaseAnswers, which provides only schema models. Authors converted these schemas to SQLite, populated them using an online tool, and manually corrected fields to appear natural.

• 90 databases created from WikiSQL tables. About 500 tables across ~90 different domains were selected from WikiSQL dev/test splits. Multiple related tables were grouped, and the authors designed relational schemas by adding foreign keys and, where necessary, intersection tables. Since WikiSQL tables contain Wikipedia data, no further population was needed.

• All schemas were manually fixed: abbreviations like “stu id” were expanded to full words, missing foreign keys added, and nonsensical column names corrected.

• Question and SQL Annotation Process

• Per database, eight computer science students proficient in SQL wrote 20‑50 question-query pairs, with no templates used.

• Annotators ensured:

• SQL pattern coverage: Queries must include SELECT (with multiple columns and aggregations), WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, JOIN, INTERSECT, EXCEPT, UNION, NOT IN, OR, AND, EXISTS, LIKE, and nested queries. Each table appears in at least one query.

• SQL consistency: A protocol was set so that when multiple equivalent SQL queries existed, all annotators chose the same query pattern to avoid confusing training.

• Question clarity: Questions must not be vague/ambiguous (e.g., no “popular” without a defined metric) and must not require outside knowledge. The specific columns to return are always stated.

• After initial annotation, a separate reviewer checked that questions were clear and that SQL labels followed the protocol. They verified coverage of all common SQL clauses.

• Native English speakers then reviewed and corrected grammar, ensured natural phrasing, and paraphrased some questions to increase diversity.

• A final expert reviewer resolved remaining ambiguities, and a script executed every SQL label to confirm syntactic correctness.

• Dataset Composition and Statistics

• 200 databases across 138 distinct domains (e.g., college, club, TV show, government). Most domains have a single database, containing 20‑50 questions; a few larger domains (like flight information) have multiple databases and over 100 questions total.

• On average: 27.6 columns and 8.8 foreign keys per database.

• Average question length: ~13 tokens; average SQL length: ~21 tokens.

• Compared to prior datasets, Spider has roughly twice as many nested queries, ten times more ORDER BY/LIMIT clauses, and ten times more GROUP BY/HAVING clauses.

• How the Data is Used

• The dataset is split into training, development, and test sets such that no database appears in multiple splits. This forces models to generalize to entirely new database schemas and domains during evaluation.

• The authors train semantic parsing models on the training set and report cross-domain accuracy on the held-out database schemas, measuring the ability to produce correct SQL for unseen domains. No specific training mixture ratios are mentioned; the standard split provided with Spider is used.

Method

The authors leverage a rigorous five-step pipeline to construct the corpus, investing approximately 1,000 hours of human labor in total.

As shown in the figure below:

The process begins with Database Collection and Creation, gathering 200 databases. Next, in the Question and SQL Annotation phase, computer science students proficient in SQL create 20 to 50 natural questions and their corresponding SQL labels for each database. To ensure diversity and reflect real-world usage, no templates or scripts are used for generation. The annotation protocol enforces three key aspects. First, SQL pattern coverage ensures that all common SQL components, such as SELECT with aggregations, WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, JOIN, INTERSECT, EXCEPT, UNION, NOT IN, OR, AND, EXISTS, LIKE, and nested queries, are represented. Second, SQL consistency is maintained by requiring annotators to choose the same SQL query pattern when multiple equivalent queries are possible. Third, question clarity is prioritized by excluding vague questions or those requiring external common sense knowledge. Annotators utilize a web-based interface powered by the sqlite web3 tool to view schemas, execute queries, and verify returned results.

Following the initial annotation, the SQL Review step involves a different annotator checking question clarity, verifying the correctness of SQL labels under the established protocol, and ensuring comprehensive coverage of SQL clauses. Subsequently, native English speakers conduct the Question Review and Paraphrase step. They correct grammar, ensure the question accurately reflects the SQL label's meaning, and add paraphrased versions to enhance linguistic diversity. Finally, the most experienced annotator performs the Final Review and Processing, resolving any remaining ambiguities and running scripts to execute and parse all SQL labels to guarantee correctness.

Based on this corpus, the authors define a text-to-SQL task that evaluates model generalization across complex queries and diverse databases. They adapt several state-of-the-art semantic parsing models for this task. A comprehensive column list is created by concatenating columns from all tables as input, and the column selection space is restricted to the current database for each question. The Seq2Seq model is adapted with attention and copying mechanisms, and a vocabulary mask is introduced to restrict decoding to SQL keywords and the current database schema elements. SQLNet is extended beyond its original design by expanding its SELECT and WHERE modules to support ORDER BY and GROUP BY components. Similarly, TypeSQL is adapted by utilizing question type information extracted from database content and extending its modules to handle ORDER BY and GROUP BY clauses.

Experiment

The evaluation framework employs set-based component matching on SQL clauses and overall exact matching, while categorizing queries by hardness levels. Experiments show that Seq2Seq-based models can occasionally generate complex nested queries but predominantly produce invalid SQL, whereas structure-guided models like SQLNet and TypeSQL achieve valid outputs but fail on nested structures; all models struggle most with WHERE clause prediction and column selection. Performance drops substantially under database-split conditions, with TypeSQL’s accuracy falling sharply, and increasing foreign keys in the schema further degrades results, indicating significant challenges in generalizing to new databases.

Earlier text-to-SQL datasets each contain only a single database in a single domain, with many paraphrased questions for a small set of SQL templates, making it easy for models to exploit template matching. Spider breaks this pattern by providing 200 databases across 138 domains, multiple tables per database, and a much higher proportion of complex SQL components like nested queries, ORDER BY, and GROUP BY, while enforcing a database split that tests generalization to unseen schemas. Under this rigorous setup, all models perform poorly, with accuracy dropping substantially on new databases and further degrading as the number of foreign keys in the schema increases. Every prior dataset uses one database and one domain, whereas Spider spans 138 domains with an average of 27.6 columns and 8.8 foreign keys per database, and its queries contain roughly twice the nested instances and an order of magnitude more ORDER BY / GROUP BY components than all previous datasets combined. Exact match accuracy falls sharply from the example split to the database split, and performance decreases as the number of foreign keys in a database rises, revealing that current models struggle to generalize to new schemas and to capture table relationships.

Accuracy on the example split declines sharply from easy to extra hard queries for all models. Structure-guided approaches like TypeSQL substantially outperform basic Seq2Seq models, but all methods struggle with database generalization, where performance is much lower than on the example split. Easy-query accuracy is reasonably high for TypeSQL (47.5%) but falls to 14.4% on extra hard queries, while Seq2Seq models achieve only 1.3% on extra hard examples. TypeSQL leads the example split with 33.0% overall accuracy, more than double the best Seq2Seq variant (18.3%), but its performance drops the most when moving to new databases. Basic Seq2Seq, Seq2Seq+Attention, and Seq2Seq+Copying produce many invalid SQL queries and gain little from attention or copying mechanisms, with overall accuracy remaining below 19%. Hardness-level breakdown shows that even the strongest model struggles as complexity increases, leaving a large gap between easy (47.5%) and extra hard (14.4%) performance. Database split results are consistently lower than example split results across all models, indicating that generalizing to unseen databases is a key challenge.

Component-level F1 scores reveal that SQL generation models incorporating explicit query structure (SQLNet, TypeSQL) far outperform generic sequence-to-sequence approaches. The WHERE clause is the most challenging component for all methods, and all models perform substantially worse when evaluated on unseen databases. TypeSQL attains the highest F1 scores across all components on the example split, reaching 77.3 for SELECT and 78.4 for KEYWORDS. Seq2Seq, Seq2Seq+Attention, and Seq2Seq+Copying record WHERE F1 scores below 10, while SQLNet and TypeSQL reach 32.9 and 52.4 respectively. WHERE clause prediction consistently yields the lowest F1 across every model, as it often involves multiple columns and operations. Under the database split, all models drop sharply compared to the example split, with TypeSQL suffering the largest decline, indicating difficulty generalizing to new databases.

The Spider evaluation spans 200 databases across 138 domains with complex SQL queries and a strict database split that tests generalization to unseen schemas. Models achieve very low accuracy, and performance degrades sharply when moving from seen examples to new databases, with further drops as query complexity and foreign key counts increase. Structure-guided approaches like TypeSQL consistently outperform generic sequence-to-sequence methods, but all models fail substantially on new schemas; WHERE clause prediction and complex components remain the hardest. The results demonstrate that current text-to-SQL systems struggle to generalize beyond training databases and to capture table relationships, leaving a large gap in practical performance.