Can a computer determine a piano player's skill level? Is it preferable tobase this assessment on visual analysis of the player's performance or shouldwe trust our ears over our eyes? Since current CNNs have difficulty processinglong video videos, how can shorter clips be sampled to best reflect the playersskill level? In this work, we collect and release a first-of-its-kind datasetfor multimodal skill assessment focusing on assessing piano player's skilllevel, answer the asked questions, initiate work in automated evaluation ofpiano playing skills and provide baselines for future work. Dataset isavailable from: https://github.com/ParitoshParmar/Piano-Skills-Assessment.