Language Modelling

Language modeling is the task of predicting the next word or character in a document, and trained language models can be applied to various natural language processing tasks such as text generation, text classification, and question answering. Since the 2010s, neural language models have replaced N-gram models, and after the 2020s, large language models (LLMs) have become the sole path to achieving state-of-the-art performance. The capabilities of these models are evaluated using metrics like cross-entropy and perplexity, with common datasets including WikiText-103, One Billion Word, Text8, C4, and The Pile.