Command Palette
Search for a command to run...
Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?
Li Zheng ; Li Yuxuan ; Zhao Penghai ; Song Renjie ; Li Xiang ; Yang Jian
Abstract
Diffusion models have recently achieved astonishing performance in generatinghigh-fidelity photo-realistic images. Given their huge success, it is stillunclear whether synthetic images are applicable for knowledge distillation whenreal images are unavailable. In this paper, we extensively study whether andhow synthetic images produced from state-of-the-art diffusion models can beused for knowledge distillation without access to real images, and obtain threekey conclusions: (1) synthetic data from diffusion models can easily lead tostate-of-the-art performance among existing synthesis-based distillationmethods, (2) low-fidelity synthetic images are better teaching materials, and(3) relatively weak classifiers are better teachers. Code is available athttps://github.com/zhengli97/DM-KD.
Code Repositories
Benchmarks
| Benchmark | Methodology | Metrics |
|---|---|---|
| few-shot-learning-on-dtd | Real-Guidance + CAL | 12-shot Accuracy: 54.5 16-shot Accuracy: 57.4 4-shot Accuracy: 41.5 8-shot Accuracy: 50.6 |
| few-shot-learning-on-fgvc-aircraft-1 | Real-Guidance + CAL | 12-shot Accuracy: 65.8 16-shot Accuracy: 72.5 4-shot Accuracy: 34.5 8-shot Accuracy: 54.6 Harmonic mean: 34.5 |
| few-shot-learning-on-stanford-cars | Real-Guidance + CAL | 12-shot Accuracy: 83.9 16-shot Accuracy: 88.3 4-shot Accuracy: 44.3 8-shot Accuracy: 73.1 |
| mitigating-contextual-bias-on-fgvc-aircraft | CAL + Real-Guidance | OOD Accuracy (%): 17.7 Top-1 Accuracy (%): 71.7 |
Build AI with AI
From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.