Abstract

We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUIautomation agents across Windows, macOS, Linux, iOS, Android, and Webplatforms. It comprises four levels: GUI Content Understanding, ElementGrounding, Task Automation, and Task Collaboration, covering essential skillsfor GUI agents. In addition, we propose a novel Efficiency-Quality Area (EQA)metric to assess GUI agent execution efficiency in online automation scenarios.Through MMBench-GUI, we identify accurate visual grounding as a criticaldeterminant of overall task success, emphasizing the substantial benefits ofmodular frameworks that integrate specialized grounding modules. Furthermore,to achieve reliable GUI automation, an agent requires strong task planning andcross-platform generalization abilities, with long-context memory, a broadaction space, and long-term reasoning playing a critical role. More important,task efficiency remains a critically underexplored dimension, and all modelssuffer from substantial inefficiencies, with excessive redundant steps evenwhen tasks are ultimately completed. The integration of precise localization,effective planning, and early stopping strategies is indispensable to enabletruly efficient and scalable GUI automation. Our benchmark code, evaluationdata, and running environment will be publicly available athttps://github.com/open-compass/MMBench-GUI.