Differentiable Neural Architecture Search (DARTS) recently attracts a lot of research attention because of its high efficiency. However, the competition of candidate operations in DARTS introduces high uncertainty for selecting the truly important operation, thus leading to serious performance collapse. In this work, we decrease the uncertainty of differentiable architecture search (DU-DARTS) by enforcing the distribution of architecture parameters to approach the one-hot categorical distribution and by replacing the zero operation with a gate switch. Without any extra search cost, our method achieves state-of-the-art performance with 2.32%, 16.74%, and 24.1% test error on CIFAR-10, CIFAR-100, and ImageNet datasets, respectively. Moreover, DU-DARTS can robustly find an excellent architecture on NAS-Bench-1Shot1 and NAS-Bench-201, which further demonstrates the effectiveness of our method. The source code is available at https://github.com/ShunLu91/DU-DARTS.