This paper introduces SFSORT, the world's fastest multi-object trackingsystem based on experiments conducted on MOT Challenge datasets. To achieve anaccurate and computationally efficient tracker, this paper employs atracking-by-detection method, following the online real-time tracking approachestablished in prior literature. By introducing a novel cost function calledthe Bounding Box Similarity Index, this work eliminates the Kalman Filter,leading to reduced computational requirements. Additionally, this paperdemonstrates the impact of scene features on enhancing object-track associationand improving track post-processing. Using a 2.2 GHz Intel Xeon CPU, theproposed method achieves an HOTA of 61.7% with a processing speed of 2242 Hzon the MOT17 dataset and an HOTA of 60.9% with a processing speed of 304 Hz onthe MOT20 dataset. The tracker's source code, fine-tuned object detectionmodel, and tutorials are available at\url{https://github.com/gitmehrdad/SFSORT}.