Talaei Shayan ; Pourreza Mohammadreza ; Chang Yu-Chen ; Mirhoseini Azalia ; Saberi Amin

Abstract

Translating natural language questions into SQL queries, known astext-to-SQL, is a long-standing research problem. Effective text-to-SQLsynthesis can become very challenging due to (i) the extensive size of databasecatalogs (descriptions of tables and their columns) and database values, (ii)reasoning over large database schemas, (iii) ensuring the functional validityof the generated queries, and (iv) navigating the ambiguities of naturallanguage questions. We introduce CHESS, a Large Language Model (LLM) basedmulti-agent framework for efficient and scalable SQL synthesis, comprising fourspecialized agents, each targeting one of the aforementioned challenges: theInformation Retriever (IR) extracts relevant data, the Schema Selector (SS)prunes large schemas, the Candidate Generator (CG) generates high-qualitycandidates and refines queries iteratively, and the Unit Tester (UT) validatesqueries through LLM-based natural language unit tests. Our framework offersconfigurable features that adapt to various deployment constraints, including1) Supporting industrial-scale databases: leveraging the Schema Selector agent,CHESS efficiently narrows down very large database schemas into manageablesub-schemas, boosting system accuracy by approximately $2\%$ and reducing thenumber of LLM tokens by $\times 5$. 2) State-of-the-Art privacy-preservingperformance: Among the methods using open-source models, CHESS achievesstate-of-the-art performance, resulting in a high-performing,privacy-preserving system suitable for industrial deployment. 3) Scalablitywith additional compute budget: In settings with high computational budgets,CHESS achieves $71.10\%$ accuracy on the BIRD test set, within $2\%$ of theleading proprietary method, while requiring approximately $83\%$ fewer LLMcalls.