Devichand Budagam Sankalp KJ Ashutosh Kumar Vinija Jain Aman Chadha

Abstract

Assessing the effectiveness of large language models (LLMs) in addressingdiverse tasks is essential for comprehending their strengths and weaknesses.Conventional evaluation techniques typically apply a single prompting strategyuniformly across datasets, not considering the varying degrees of taskcomplexity. We introduce the Hierarchical Prompting Taxonomy (HPT), a taxonomythat employs a Hierarchical Prompt Framework (HPF) composed of five uniqueprompting strategies, arranged from the simplest to the most complex, to assessLLMs more precisely and to offer a clearer perspective. This taxonomy assigns ascore, called the Hierarchical Prompting Score (HP-Score), to datasets as wellas LLMs based on the rules of the taxonomy, providing a nuanced understandingof their ability to solve diverse tasks and offering a universal measure oftask complexity. Additionally, we introduce the Adaptive Hierarchical Promptframework, which automates the selection of appropriate prompting strategiesfor each task. This study compares manual and adaptive hierarchical promptframeworks using four instruction-tuned LLMs, namely Llama 3 8B, Phi 3 3.8B,Mistral 7B, and Gemma 7B, across four datasets: BoolQ, CommonSenseQA (CSQA),IWSLT-2017 en-fr (IWSLT), and SamSum. Experiments demonstrate the effectivenessof HPT, providing a reliable way to compare different tasks and LLMcapabilities. This paper leads to the development of a universal evaluationmetric that can be used to evaluate both the complexity of the datasets and thecapabilities of LLMs. The implementation of both manual HPF and adaptive HPF ispublicly available.