Deep Learning Revolutionizes Live-Cell Nanoscale Fluorescence Imaging: Up to 4x Frame Rate With Only 25% Sampling

It is well known that cells are the fundamental structural and functional units of living organisms. Various exquisite organelles, cytoskeletal elements, and protein complexes—nanoscale structures within cells—perform their respective roles to drive complex biological activities. However, to date, the scientific community still has limited understanding of how these intricate nanoscale structures dynamically interact. The primary reason is that conventional optical microscopy is constrained by the Abbe diffraction limit, leaving many fine nanoscale structures within cells and their dynamic changes shrouded in a "blurry" mist, significantly hindering progress in the study of cellular substructures.

The advent of fluorescence nanoscopy has broken this barrier. As a class of far-field fluorescence microscopy techniques that surpass the optical diffraction limit, it employs physical methods to modulate the emission states of fluorescent molecules, effectively advancing observation resolution to the nanoscale. This allows researchers to directly observe the fine conformations and dynamic evolution of microscopic structures within cells, making it a crucial tool for exploring cellular physiological principles, elucidating disease mechanisms, and decoding the mysteries of life activities.

Among the various technical approaches to fluorescence nanoscopy, Reversible Saturable Optical Linear Fluorescence Transitions (RESOLFT) nanoscopy is a representative solution. It utilizes reversibly switchable fluorescent proteins (rsFPs) and spatially structured light fields to reversibly switch fluorescent proteins between ON and OFF states, thereby creating molecular emission sparsity. Through this ingenious "selective activation" scanning, high-definition cellular images beyond the diffraction limit are ultimately reconstructed.

However, RESOLFT nanoscopy is not yet perfect. Two major obstacles hinder its widespread application: First, rsFPs suffer from photoswitching fatigue, where repeated state transitions may lead to a gradual loss of reversible switching capability, directly limiting the total duration for long-term time-lapse imaging of live cells. Second, the imaging time of RESOLFT is constrained by fluorescent molecule switching, excitation duration, and the number of spatial scanning sampling points. More sampling points result in longer imaging times, and insufficient temporal resolution can easily produce motion artifacts.

To address these challenges, a cross-institutional research team from the KTH Royal Institute of Technology in Stockholm, Sweden, the Technical University of Dresden in Germany, and the Human Technopole Foundation in Milan, Italy, combined deep learning with RESOLFT imaging to develop two complementary computational imaging strategies: the Hierarchical DivNoising (HDN) network and the Fast-MoNaLISA sparse sampling reconstruction framework. These are used to recover low signal-to-noise ratio (SNR) or undersampled images, greatly extending the experimental limits of RESOLFT imaging. With the HDN network, the light dose per single frame can be reduced by 10-fold, and the total duration of time-lapse imaging can be extended by up to 5-fold. With the Fast-MoNaLISA framework, the imaging frame rate can be increased by up to 4-fold under 25% sampling conditions.

The related findings were published in Nature Communications under the title "Time-lapse RESOLFT nanoscopy empowered by deep learning."

Research Highlights:

• Based on the HDN/HDNsup hierarchical denoising network, low-SNR RESOLFT images can be restored under 10-fold lower light dose conditions, enabling live-cell long-term nanoscopy for up to 5 times longer duration.
• Developed the Fast-MoNaLISA temporal upsampling model to perform super-resolution reconstruction on undersampled images, achieving up to 4-fold imaging acceleration and reducing scanning artifacts.
• Constructed multi-type datasets for comprehensive model validation, enabling live-cell nanoscopy of dynamic subcellular processes such as mitochondrial vesicles and actin networks.

Paper link:

https://www.nature.com/articles/s41467-026-78580-z

Multi-source Dataset Design for Training and Independent Evaluation

This study utilized several RESOLFT datasets with different purposes, covering paired datasets, unpaired datasets, and independent datasets. Specifically:

Paired Datasets

To obtain clean-noisy paired image datasets, researchers fixed the cells to prevent structural displacement between image pairs. Fixation limits the number of rsFP cycles, meaning photobleaching occurs faster than in live cells, but high-quality RESOLFT images can still be obtained.

For the fixed vimentin dataset (U2OS Vim-rsEGFP2), researchers first acquired noisy images with approximately 10%-20% ON-switching (approximately 30 W/cm²); subsequently, 5 images were acquired with approximately 80%-90% ON-switching (approximately 300 W/cm²). The pseudo-time series signal decays rapidly. Researchers utilized this signal decay process to evaluate HDN, HDNsup, and upsampling prediction results at different SNR levels, taking the average of the first 3 images of the pseudo-time series to obtain the optimal ground truth reference.

For the fixed actin meshwork dataset (U2OS LifeAct-rsEGFP2), the procedure was similar. However, the first image was acquired with approximately 50% ON-switching; subsequently, 5 images were acquired with approximately 90% ON-switching, and averaging was performed to generate high-quality ground truth reference images for training the HDNsup model.

Unpaired Datasets

Unpaired datasets are primarily used for unsupervised denoising and upsampling training.

For unsupervised HDN denoising of mitochondria and HIV Gag-IM particles, researchers employed time series with low ON-switching times, training the HDN directly on these low signal-to-noise ratio (SNR) time series. Since a single time series contains a large number of frames, a small number of cells can provide sufficient noisy images.

For the upsampling training dataset, researchers acquired conventional RESOLFT time series (containing 10-40 frames), with the duration adjusted according to each cell and structure to ensure sufficient SNR across the entire time series.

Independent Test Datasets

For HIV Gag-IM particles, researchers additionally acquired two test sets: a denoising evaluation set (containing images acquired under three conditions: low ON-switching, high ON-switching, and enhanced confocal) and an upsampling evaluation set (containing images acquired under only two conditions: high ON-switching and enhanced confocal). It is worth noting that all these datasets were acquired from live cells, thus involving the impact of cell movement.

Each dataset was randomly split into training, validation, and test sets. Image sizes included 64 × 64, 96 × 96, and 128 × 128; blocks with insufficient information were discarded. The specific datasets are shown in the figure below:

Deconstructing Core Challenges, Focusing on Image Denoising and Sparse Sampling Recovery

This study focuses on RESOLFT technology, with the basic method adopting the MoNaLISA implementation—a state-of-the-art parallelized RESOLFT imaging technique. MoNaLISA integrates multi-focus patterns for ON-switching and read-out, along with superimposed periodic line patterns for OFF-switching, enabling live-cell imaging with a spatial resolution of 45-60 nm at a frame rate of approximately 0.5 Hz within a field of view (FOV) of about 40 × 40 µm².

Building on this, researchers further proposed leveraging deep learning to recover images with low SNR or undersampled images, thereby breaking the limitations of traditional RESOLFT methods. The logic is straightforward: utilizing image denoising techniques to reduce the light dose required for imaging, thereby mitigating photobleaching; and constructing a computational framework to recover data acquired via sparse sampling, enhancing temporal resolution through computational upsampling.

Specifically, the ON-switching light intensity determines the number of activated rsFPs, thereby controlling the intensity of the acquired signal. Traditional schemes typically activate 80%-90% of rsFPs to maximize SNR; in contrast, this study proposes an opposite strategy, activating only 10%-20% of rsFPs to acquire low-SNR images, and then reconstructing high-quality images via deep learning-based denoising. This not only minimizes photobleaching but also alleviates phototoxicity issues arising from the use of short wavelengths (typically 405 nm) for ON-switching.

Based on this, researchers proposed adopting HDN, an unsupervised image denoising algorithm. This method is based on a ladder variational autoencoder (LVAE) network structure and incorporates an experimentally measured noise model. Additionally, the study provides a supervised variant of HDN—HDNsup—based on the same framework but modified for supervised training. The research team proposed both unsupervised and supervised versions so that supervised learning can be used when high-quality ground truth (GT) is available, while the unsupervised mode remains stably usable when no ground truth is available. Meanwhile, the ideal strategy must also effectively reduce noise without sacrificing spatial resolution.

In HDN, the backbone network LVAE is used to learn hierarchical structural priors of biological image data, simultaneously capturing multi-level features from coarse outlines to fine textures. More importantly, this model is a multi-modal prior model that does not output a single deterministic result; the experimental noise model is the core enabling unsupervised training in HDN, describing the conditional probability of the underlying true clean signal given the noisy observed image by the imaging system. More importantly, this framework can be easily adapted for supervised training, allowing HDN to leverage known clean ground truth information while retaining the two key characteristics of HDN: multi-modal representation and adjustable output diversity.

The second proposed solution is to construct a framework for recovering sparsely sampled images, aiming to visualize rapid live-cell dynamics. Specifically, acquisition time depends on the number of scanning steps, and combining sparse sampling with image recovery techniques promises to significantly improve imaging speed while maintaining image quality. The optimal sampling step size is typically determined by the Nyquist-Shannon sampling theorem to prevent aliasing artifacts or information redundancy, with the step size equal to twice the system's cutoff frequency, approximately half of the expected resolution. In RESOLFT imaging, the step size is approximately 20-30 nm.

Based on this, researchers proposed Fast MoNaLISA. This method shortens the single-frame acquisition time by skipping some scanning steps, then uses an upsampling network to recover undersampled images. The advantages are significant: it improves temporal resolution without changing the light dose. This concept has been applied to diffraction-limited confocal microscopy but has not yet been validated in nanoscale super-resolution imaging below 100 nm. Additionally, a key technical detail of this study is that while reducing sampling, the OFF-switching light dose remains unchanged, thereby maintaining the original imaging resolution on the sample.

Fast MoNaLISA implements two training paradigms: single-frame training and multi-frame training. Single-frame training targets super-resolution reconstruction in static scenes, providing the network with data from a single time point. Multi-frame training improves upon the single-frame method, aiming to image fast dynamic processes of intracellular molecules and organelles. In this approach, researchers use sub-sampled time-lapse images as input, rather than single snapshots, generating sub-sampled images at each time point by applying the same random down-sampling steps used in single-frame training. The network is then trained to predict intermediate time points within the time-lapse sequence, while also learning additional information provided by the new temporal context of the input images, thereby enhancing reconstruction fidelity.

Regarding network architecture, the foundational structure for Fast MoNaLISA's super-resolution reconstruction is UNet-RCAN, but it retains the original RCAN's final upsampling layer, keeping it intact. The rationale for choosing this network is straightforward: RCAN is inherently designed for image up-sampling tasks, making it naturally suited for recovering under-sampled images in this study. Additionally, in terms of structural trade-offs, the model retains the U-Net backbone to facilitate faster convergence.

10x Less Illumination, 5x Longer Time-Lapse, 4x Faster Imaging

The study first evaluated HDN's reconstruction capability for low ON-switching images. The experiments compared HDN with unsupervised methods Noise2Void (N2V) and Self-Inspired Noise2Noise (SN2N), as well as supervised methods Content-Aware Image Restoration (CARE) and UNet-RCAN.

Results showed that the ground truth (GT) had an average full width at half-maximum (FWHM) of 58 nm, while both HDN and HDNsup maintained spatial resolution with average FWHMs of 56 nm and 54 nm, respectively. In comparison, the unsupervised method SN2N had an average FWHM of 120 nm, and N2V had 93 nm; supervised methods performed relatively better, with CARE at 73 nm and UNet-RCAN at 70 nm. However, because these methods tend to smooth cellular structures, leading to loss of detail, their average FWHMs remained higher than the ground truth, and their Fourier ring correlation (FRC) curves showed declines in the low-frequency range, as shown in the figure below:

It is worth noting, however, that the limited performance of supervised denoising methods does not stem from the network architecture itself, but rather from the insufficient quality of the ground truth. Therefore, like other supervised learning methods, although HDNsup performs excellently, its performance remains highly dependent on ground truth quality.

Next, researchers validated the effectiveness of the HDN denoising method on sparser punctate structures. Results indicated that HDN accurately recovered these punctate structures and preserved super-resolution details unobservable in enhanced confocal imaging. This capability is unmatched by other unsupervised denoising methods, a conclusion further confirmed by FRC analysis. Of course, prediction accuracy decreases when the signal-to-noise ratio is extremely low; therefore, ON-switching levels must be carefully selected and validated to ensure prediction reliability. For HIV Gag-IM-rsEGFP2, a 5% active state fraction resulted in obvious reconstruction artifacts, while 10% represented a more reliable working threshold.

When validating the scheme's ability to achieve long-duration time-lapse imaging of live cells, the UV-visible light energy used for imaging was only about 1/10 of the original amount, allowing time-lapse imaging durations to be extended by up to approximately 5 times compared to conventional high ON-switching RESOLFT methods, as shown in the figure below. Notably, these experiments were conducted at endogenous expression levels of vimentin. For brighter samples, such as the rsEGFP2-labeled outer mitochondrial membrane, unsupervised HDN was used instead, enabling time-lapse imaging of over 100 frames.

Researchers continued to use HDNsup for imaging experiments on complex interwoven structures such as the actin meshwork. By collecting high-quality data from fixed cells expressing LifeAct-rsEGFP2, they first verified its ability to precisely reconstruct the actin meshwork, then transitioned to live-cell imaging. Results showed that HDNsup successfully acquired at least 50 consecutive frames, enabling long-term observation of the dynamic processes of the actin meshwork.

In the evaluation of single-frame versus multi-frame training, the study explored three sampling rates: 75%, 50%, and 25%, corresponding to frame rate improvements of 1.3x, 2x, and 4x, respectively.

Single-frame training results demonstrated that Fast MoNaLISA performed best on images with high signal-to-noise ratios; therefore, conventional ON-switching levels (80%-100%) should be adopted. This aligns with the overall goal of reducing light dosage, as sub-sampling itself reduces the dose. Furthermore, the choice of sampling rate reflects a trade-off between imaging speed and prediction reliability; adopting a 50% sampling rate in single-frame prediction represents the most robust compromise between speed and fidelity, as shown in the figure below:

Multi-frame training results demonstrate that Fast MoNaLISA can accurately reconstruct images under high-speed imaging conditions with only 25% pixel acquisition, surpassing the previous limitations of single-frame training. As shown in the figure below:

Conclusion

In summary, this study established a hybrid framework combining experimental imaging with computational reconstruction, enabling long-term dynamic observation of living cells based on parallel RESOLFT technology. With a spatial resolution of approximately 60 nanometers, the method can capture fine structures such as the cytoskeleton and mitochondrial vesicles, and track weak signals and fast-moving molecules that were previously observable only in static imaging. Compared to mainstream live-cell super-resolution techniques such as SIM, SOFI, and STED, this approach achieves a rare balance among sub-70 nm resolution, imaging speed, field of view, and photodamage.

Notably, the team has open-sourced the LISAI software framework, and the associated algorithms can be transferred to other scanning microscopy systems such as STED and spinning-disk confocal microscopy. Of course, deep learning-based reconstruction involves parameter trade-offs, requiring parameter tuning based on the signal-to-noise ratio of the samples. Future work may further optimize the model and build an integrated framework to simultaneously reduce imaging light dose and acquisition time.