We aim to tackle the problem of point-based interactive segmentation, inwhich the key challenge is to propagate the user-provided annotations tounlabeled regions efficiently. Existing methods tackle this challenge byutilizing computationally expensive fully connected graphs or transformerarchitectures that sacrifice important fine-grained information required foraccurate segmentation. To overcome these limitations, we propose a cascadesparse feature propagation network that learns a click-augmented featurerepresentation for propagating user-provided information to unlabeled regions.The sparse design of our network enables efficient information propagation onhigh-resolution features, resulting in more detailed object segmentation. Wevalidate the effectiveness of our method through comprehensive experiments onvarious benchmarks, and the results demonstrate the superior performance of ourapproach. Code is available at\href{https://github.com/kleinzcy/CSFPN}{https://github.com/kleinzcy/CSFPN}.