Yi Cui

Abstract

This paper presents insights from evaluating 16 frontier large languagemodels (LLMs) on the WebApp1K benchmark, a test suite designed to assess theability of LLMs to generate web application code. The results reveal that whileall models possess similar underlying knowledge, their performance isdifferentiated by the frequency of mistakes they make. By analyzing lines ofcode (LOC) and failure distributions, we find that writing correct code is morecomplex than generating incorrect code. Furthermore, prompt engineering showslimited efficacy in reducing errors beyond specific cases. These findingssuggest that further advancements in coding LLM should emphasize on modelreliability and mistake minimization.