While pre-trained large-scale vision models have shown significant promisefor semantic correspondence, their features often struggle to grasp thegeometry and orientation of instances. This paper identifies the importance ofbeing geometry-aware for semantic correspondence and reveals a limitation ofthe features of current foundation models under simple post-processing. We showthat incorporating this information can markedly enhance semanticcorrespondence performance with simple but effective solutions in bothzero-shot and supervised settings. We also construct a new challengingbenchmark for semantic correspondence built from an existing animal poseestimation dataset, for both pre-training validating models. Our methodachieves a PCK@0.10 score of 65.4 (zero-shot) and 85.6 (supervised) on thechallenging SPair-71k dataset, outperforming the state of the art by 5.5p and11.0p absolute gains, respectively. Our code and datasets are publiclyavailable at: https://telling-left-from-right.github.io/.