OmniBenchmark is a rich yet concise benchmark that can be used to evaluate the generalization ability of pre-trained models across semantic super-concepts/realms, e.g., across mammals to aircraft. The dataset contains 21 semantic realm-wise datasets (including 7,372 concepts and 1,074,346 images) with no concept overlap.