The arguably most widely employed algorithm to train deep neural networkswith Differential Privacy is DPSGD, which requires clipping and noising ofper-sample gradients. This introduces a reduction in model utility compared tonon-private training. Empirically, it can be observed that this accuracydegradation is strongly dependent on the model architecture. We investigatedthis phenomenon and, by combining components which exhibit good individualperformance, distilled a new model architecture termed SmoothNet, which ischaracterised by increased robustness to the challenges of DP-SGD training.Experimentally, we benchmark SmoothNet against standard architectures on twobenchmark datasets and observe that our architecture outperforms others,reaching an accuracy of 73.5% on CIFAR-10 at $\varepsilon=7.0$ε=7.0 and 69.2% at$\varepsilon=7.0$ε=7.0 on ImageNette, a state-of-the-art result compared to priorarchitectural modifications for DP.