Multi-frame Restoration for High-rate Lissajous Confocal Laser Endomicroscopy Minhee Lee1 , Sangyoon Lee1 , Jiwook Lee2 , Minki Hong2 , Kyuyoung Kim2 , Wonhwa Kim1 , and Jaeho Lee1 POSTECH, Pohang, Republic of Korea VPIX Medical, Seoul, Republic of Korea {mhlee02,sangyoon.lee,wonhwa,jaeho.lee}@postech.ac.kr {jw.lee,mk.hong,ky.kim}@vpixmedical.com
arXiv:2605.00527v1 [eess.IV] 1 May 2026
1
2
Abstract. Lissajous confocal laser endomicroscopy (CLE) is a promising solution for high speed in vivo optical biopsy for handheld scenarios. However, Lissajous scanning traces a resonant trajectory and samples only the visited pixels per frame; at high frame rates, many pixels remain unvisited, creating structured holes. In this work, we introduce the first benchmark for high-rate Lissajous CLE, consisting of lowquality video clips paired with high-quality reference images. The reference images are wide-FOV mosaics obtained by stitching stabilized, slowscan frames of the same tissue, enabling temporally aligned supervision. Using this dataset, we propose MIRA, a lightweight recurrent framework for Lissajous CLE restoration that iteratively aggregates temporal context through feature reuse and displacement alignment. Our experiments demonstrate that MIRA outperforms both lightweight and highcomplexity baselines in restoration quality while maintaining a favorable computational efficiency suitable for clinical deployment. Keywords: Confocal laser endomicroscopy · Lissajous CLE restoration
1
Introduction
Confocal laser endomicroscopy (CLE) enables in vivo imaging of tissue microstructures at cellular resolution, offering the potential for real-time “optical biopsy” during endoscopy [3,26]. However, since CLE forms images via sequential laser scanning over a microscopic field-of-view (FOV), it is inherently sensitive to probe-tissue motion. Even subtle motion can cause severe distortion (i.e., motion artifacts), necessitating robust, high-speed imaging for clinical use [1,13,16,23]. Increasing the frame rate is a natural way to reduce motion artifacts by shortening the acquisition window. Yet, conventional raster scanning is mechanically limited at high speeds, due to its stop-and-turn trajectory and dense sampling. In contrast, Lissajous scanning, driven by resonant oscillations along orthogonal axes, enables substantially higher frame rate with smoother, inertia-efficient motion, making it particularly attractive for high-rate CLE [6,7,8]. Despite its speed advantage, high-rate Lissajous CLE produces sparse, nonuniform pixel measurements, resulting in severe undersampling artifacts (called
2
M. Lee et al.
(a)
(b)
(c)
(d)
Fig. 1: A visual description of the high-rate Lissajous CLE frames: (a) High-rate LQ frame (10Hz) with steady probe movement; (b) High-rate LQ frame of the same scene, with shaky probe movement; (c) Scanning trajectory of the probe while taking one frame, where the white “holes” correspond to the missing pixels; (d) The low-rate HQ frame with the same field-of-view.
“holes”) and degraded quality. Simply aggregating multiple frames can improve spatial coverage, but risks reintroducing motion artifacts and misalignments [13]. To address this challenge, we propose learning to restore high-quality frames from high-rate Lissajous CLE. We construct the first open benchmark Lissajous CLE dataset, MaLissa (Matched Lissajous CLE), comprising over 2,000 paired high-rate low-quality (LQ) and low-rate high-quality (HQ) frames collected from animal tissues. Since acquiring perfectly aligned LQ-HQ pairs is difficult, we propose a matching-based strategy: we first stitch HQ frames into a large mosaic, then extract for each LQ frame the best-matching HQ subimage as its target. Using MaLissa, we propose MIRA (Multi-frame Iterative RestorAtion), a lightweight framework to integrate the current LQ frame with past frames to estimate the HQ output. To handle sparse pixels while ensuring low latency, MIRA adopts two key strategies: (1) recurrently caching and reusing encoded features across timesteps to avoid redundant computations; (2) aligning spatially mismatched features using a lightweight flow-free registration module, which estimates inter-frame displacement from sparse pixels. MIRA achieves a favorable latency–quality tradeoff (22.88dB PSNR at 236.66ms), outperforming previous baselines [2,15,20] while remaining suitable for intraoperative deployment. Contribution. (i) We introduce the task of high-rate Lissajous CLE restoration and introduce MaLissa, the first open dataset in this domain. (ii) We propose MIRA, a lightweight recurrent restoration framework for Lissajous CLE. (iii) MIRA surpasses 22dB PSNR with sub-250ms latency, providing a practical step toward real-time in vivo optical biopsy. The MaLissa dataset is available at: https://released.upon.acceptance
2
Task: High-rate Lissajous CLE Restoration
The task of high-rate Lissajous CLE restoration can be formalized as follows: Let I = {It }Tt=1 denote the input frame sequence collected with high-rate Lissajous CLE, where each It ∈ RH×W is a grayscale LQ frame with sparse, motiondistorted pixels; noisy pixel measurements are acquired along a Lissajous trajec-
Multi-frame Restoration for High-rate Lissajous CLE
3
tory, and other pixels have the value 0. Our goal is to reconstruct a HQ frame sequence O = {Ot }Tt=1 from I, where each Ot ∈ RH×W should correspond to the same scene as It , but with dense measurements and less motion artifacts. At high frame rate, each LQ frame contains substantial missing pixels, making single-frame restoration ill-posed. Thus, we consider multi-frame restoration: Given a window of preceding frames It−τ , . . . , It−1 , we estimate Ôt = f (It−τ , . . . , It−1 , It )
(1)
such that this estimate approximates the output frames well, i.e., Ot ≈ Ôt . This task differs critically from conventional video restoration problems, such as video deblurring [2,4,5], in how features across neighboring frames should be aligned. On one hand, popular association methods such as optical flow [21,25] are difficult to apply, as the frames contain substantial missing pixels. On the other hand, we can exploit the fact that object displacement is primarily caused by global motion of the probe or the tissue as a whole; consequently, the interframe displacement of most objects tends to be consistent across the scene.
3
The MaLissa Dataset
To train and evaluate our model, we introduce an open Lissajous CLE restoration dataset, termed MaLissa (Matched Lissajous CLE), consisting of paired LQ and HQ frames (It , Ot ) that share the same scene. MaLissa contains 2,060 curated pairs (1,619 for training and 441 for validation), selected from 16 raw video clips (3,988 frames total). All videos were acquired at 1024×1024 resolution using the cCeLL Lissajous-scanning CLE system [6], with a 500 × 500 µm FOV. The data were collected from porcine stomach tissue. Although non-pathological, these tissues closely resemble human gastrointestinal tract anatomically and physiologically, making them a common choice for open-to-public datasets [19,24]. To emulate clinical use, LQ frames were captured at 10Hz, resulting in more than 70% missing pixels. During acquisition, the probe was moved irregularly in random directions to mimic a clinician’s motion. HQ frames were captured at 2Hz, leaving fewer than 10% pixels unmeasured. The missing pixels were filled via bilinear interpolation to obtain dense images. The probe followed an outward spiral trajectory with small overlaps to cover the entire sample. 3.1
Matching LQ and HQ frames
As the raw LQ and HQ videos are acquired along different probe trajectories, their frames do not share the same FOV, making direct frame-wise matching infeasible. Moreover, LQ frames contain substantial missing pixels, and asynchronous acquisition introduces local intensity inconsistencies. To obtain paired frames (It , Ot ) with identical FOV, we construct an HQ mosaic by stitching HQ frames and match each LQ frame to a corresponding subimage of this mosaic. Our method consists of four steps:
4
M. Lee et al.
Fig. 2: Overview of dataset construction procedure for the MaLissa dataset.
Step 1: HQ stitching. We select 25 HQ frames with approximately 20% spatial overlap and stitch them to generate the HQ mosaic Omos . To maintain intensity continuity while suppressing edge artifacts, all frames are mapped to a unified coordinate system and blended using a Hann window. Step 2: Augmenting LQ frames. As LQ frames contain many missing pixels, direct comparison with HQ subimages is unreliable. We therefore construct augmented LQ frames Ĩ = {I˜t }Tt=1 , by aggregating information from neighboring frames. For each It , we estimate the translations to the past (It−4 , . . . , It−1 ) and future (It+1 , . . . , It+4 ) frames using FFT-based phase correlation [27]: r(I, I ′ ) = F −1 norm F(I) ⊙ F(I ′ ) ,
(2)
where F (·) denotes the 2D FFT, (·) the complex conjugate, ⊙ the Hadamard product, and norm the elementwise normalization for complex values. The translation maximizing this correlation is selected to align neighboring frames with It . The aligned frames are then averaged with It , using only measured pixels while ignoring unobserved locations, yielding the augmented frame I˜t . Step 3: Matching via phase correlation. We match augmented LQ frames I˜t to cropped subimages of the HQ mosaic Omos using FFT-based phase correlation. To mitigate the effect of missing pixels, we apply max-pooling downsampling to ds both I˜t and Omos , and compute the correlation r(I˜tds , Omos ). Pairs with correlation above a threshold (0.05 for the MaLissa dataset) are retained. If a match is found, the corresponding subimage is assigned to the HQ label Ot for It . Step 4: Temporal expansion. The pairs (It , Ot ) identified in Step 3 guide further matching, as temporally adjacent LQ frames are likely to correspond to nearby regions in Omos . For each (It , Ot ), we search for subimages that align with It−1 and It+1 . Since unmatched frames often exhibit motion artifacts that degrade structural cues and hinder phase correlation, we instead employ template matching in this step. The process is propagated to neighboring frames until no additional matches are found.
Multi-frame Restoration for High-rate Lissajous CLE
5
Fig. 3: Overview of the MIRA architecture. Encoder features from previous frames Et−4 , . . . , Et−1 and the previous FAM output d˜t−1 are cached and aligned to the current frame via global registration.
4
Method: Multi-frame Iterative Restoration (MIRA)
MIRA adopts a lightweight U-Net architecture [22] for low-latency processing (Fig. 3). Both encoder and decoder blocks use ResFFT modules [17], which combine frequency and spatial branches to suppress motion artifacts that are often separable in the frequency domain [9]. At timestep t, the encoder extracts multiscale features Etl from It and stores them in a memory bank with a window size of 4. Each decoder block aggregates current features with cached features of l {Et−i }4i=0 to compensate for pixel sparsity. Concretely, each block conducts l/2
Et
= enc Etl ,
l/2 l Dtl = dec Dt , Etl , {shift(Et−i , ∆st,t−i )}4i=1 ,
(3)
where Dtl denote the decoder features at step t and scale l, and shift(·) denotes the registration (described below). A 1 × 1 convolution is added at each decoder block to enhance the representational capacity. The decoded features are further refined by a lightweight feature attention module (FAM) [4,17], which recursively incorporates the previous FAM output dˆt−1 to stabilize high-frequency details. Flow-free registration of cached features. Before aggregation, the cached feal tures {Et−i }4i=1 , d˜t−1 are aligned to the current frame It . This is done by estimating the global inter-frame displacement via phase correlation: ∆st,t′ = argmax(x,y) [r(I˜tds , I˜tds ′ )]x,y
(4)
This strategy avoids registration methods such as optical flow which are sensitive to missing pixels and computationally expensive, and exploits the contactconstrained nature of CLE, where motion is predominantly lateral [10,18].
6
M. Lee et al.
Table 1: Quality and efficiency of various video restoration models, with the best in bold and the runner-up in underlined. We use the patch size 256, 5 frame window. The latency is measured in ms, on an NVIDIA RTX 6000 Ada. Model
Backbone PSNR
MedVSR [12]
SSIM MS-SSIM Latency GFLOPs Params
SSM
9.95
0.0597
0.2747
Transformer Transformer
21.81 21.45
0.5957 0.5885
0.7037 1668.69 0.6948 491.77
9280.94 59.08M 1363.74 13.57M
BasicVSR++ [2] EMVD [15]
CNN CNN
20.89 18.71
0.5813 0.5441
0.6876 333.19 0.6349 165.63
1968.80 7.39M 10.41 0.02M
MIRA (Ours)
CNN 22.88 0.6008
Turtle [5] RVRT [11]
4.1
0.7197
290.03
236.66
1428.92
16.57
7.12M
7.29M
Training with patch-wise rejection sampling
MIRA is trained with the joint loss ℓ(Ot , Ôt ) = ℓchar (Ot , Ôt ) + λ · ∥F(Ot ) − F (Ôt )∥1 ,
(5)
where the Charbonnier loss ℓchar ensures spatial robustness and the latter loss enhances structural details [4,17], which are balanced by λ = 0.01. During training, we randomly crop 256 × 256 patches from paired LQ-HQ frames. As phase-correlation-based alignment can introduce brightness mismatches that destabilize the training, we apply patch-wise rejection sampling to filter out problematic crops: For each crop, we compute a binary inconsistency mask between the HQ patch and the (downsampled) augmented LQ frame I˜tds , which aggregates aligned neighboring frames to reduce missing pixels. The patch is divided into 8 × 8 blocks, and block-wise MSE is evaluated between LQ and HQ blocks. Blocks above a threshold are marked inconsistent. The patch is rejected and resampled if inconsistent regions exceed 12.5% of the area.
5
Experiments & Results
5.1
Experimental setup
MIRA was trained with Adam with the step size 2e-4 and the batch size 32. We trained for 600 epochs with the cosine annealing. Baselines were trained using the hyperparameters in their original papers. We have also applied random cropping, patch-wise rejection and temporal reversal augmentation to all methods. 5.2
Main results on image restoration
We compare MIRA with state-of-the-art video restoration models on the MaLissa validation split (Table 1). MIRA achieves the highest restoration quality while
Multi-frame Restoration for High-rate Lissajous CLE
Turtle
RVRT
BasicVSR++
EMVD
MIRA
7
HQ
Fig. 4: Qualitative comparison of LQ frames restored by MIRA and the baselines.
Table 2: Ablation of key technical components of MIRA. “Registration” denotes the flow-free registration, and “Rej. sampling” denotes the patch-wise rejection sampling. Variant
PSNR↑ SSIM↑ MS-SSIM↑
w/o Registration 22.87 0.6006 w/o FAM 22.90 0.6000 w/o rej. sampling 22.70 0.5966
0.7093 0.7083 0.7050
MIRA
0.7197
22.88 0.6008
Fig. 5: Qualitative comparison of ablations.
maintaining substantially lower latency than transformer-based methods. Compared with convolutional approaches, MIRA delivers markedly better reconstruction with only a modest increase in computation. Overall, MIRA offers a favorable accuracy-efficiency tradeoff, bridging the gap between heavy and lightweight models. Qualitatively, MIRA recovers sharper structures and preserves edge better than BasicVSR++ and Turtle, which tend to over-smooth details (Fig. 4). 5.3
Ablation studies
We conduct an ablation study for each technical component of MIRA in Table 2. Removing the rejection sampling leads to a substantial quality decrease in all metrics. Removing either registration or FAM does not noticeably decrease the
8
M. Lee et al.
Table 3: Downstream task performances. Acc@1 (Full) and segmentation metrics (95% upper-trimmed mean) are reported. Bold indicates the best performance. Retrieval Model
Segmentation
Acc@1 ↑
Dice ↑
HD95 ↓
BFScore ↑
BasicVSR++ Turtle
59.4 60.6
0.8670 ± 0.0953 0.8683 ± 0.0958
16.3700 ± 7.9237 15.9190 ± 7.0681
0.2849 ± 0.1614 0.2752 ± 0.1452
MIRA (Ours)
70.7
0.8688 ± 0.1004 15.2473 ± 7.2288 0.2919 ± 0.1711
PSNR or SSIM, but has visible impacts on MS-SSIM; Indeed, the removal does degrade the perceptual quality of the images, as shown in Fig. 5. 5.4
Downstream applications
To evaluate predictive quality from restored frames, we consider two downstream tasks: 1) sample region retrieval and 2) semantic segmentation. Retrieval. We evaluate the task of retrieving the correct HQ mosaic given a restored LQ frame as the query; see Table 3. MIRA achieves 70.7% accuracy, outperforming BasicVSR++ and Turtle by 11.3%p and 10.1%p, respectively. Segmentation. We assess structural consistency via segmentation, measuring the overlap between segmentation maps from restored LQ and their corresponding HQ frames. Since MaLissa lacks clinician-annotated masks, we construct pseudo-GT masks on HQ images by prompting MedSAM [14] with humanannotated bounding boxes on visible objects. The same pipeline is applied to restored LQ frames. We report Dice, HD95, and BFScore, using the 95% uppertrimmed mean to mitigate the influence of a small number of outlier frames with degenerate predictions. MIRA outperforms the baselines across all metrics. Discussion. The results show that MIRA’s advantage goes beyond pixel-level fidelity to improved reconstruction of structural properties. Such improvements may facilitate more reliable clinical assessment during endoscopic procedures.
6
Conclusion
In this work, we introduce the task of high-rate Lissajous CLE restoration and present MaLissa, the first publicly available dataset constructed via a matchingbased strategy to address field-of-view inconsistencies. We also propose MIRA, a lightweight U-Net-based recurrent framework that effectively aligns and aggregates sparse low-quality frames. Trained on MaLissa, MIRA outperforms existing video restoration methods while maintaining low latency. Future work. Despite these advances, inference latency remains a bottleneck for fully real-time deployment. For a target frame rate of 10Hz, the per-frame latency should ideally be below 100 ms. A promising direction is to incorporate model compression techniques to further accelerate MIRA. In particular, since
Multi-frame Restoration for High-rate Lissajous CLE
9
the current implementation operates in FP32, transitioning to lower-precision formats (e.g., FP8/FP16) provides a natural starting point. With careful algorithmic design to minimize quality degradation, such precision scaling could deliver substantial efficiency gains while preserving restoration performance.
References 1. Carbone, F., Fochi, N.P., Di Perna, G., Wagner, A., Schlegel, J., Ranieri, E., Spetzger, U., Armocida, D., Cofano, F., Garbossa, D., et al.: Confocal laser endomicroscopy: enhancing intraoperative decision making in neurosurgery. Diagnostics (2025) 2. Chan, K.C., Zhou, S., Xu, X., Loy, C.C.: BasicVSR++: Improving video superresolution with enhanced propagation and alignment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022) 3. Chauhan, S.S., Dayyeh, B.K.A., Bhat, Y.M., Gottlieb, K.T., Hwang, J.H., Komanduri, S., Konda, V., Lo, S.K., Manfredi, M.A., Maple, J.T., et al.: Confocal laser endomicroscopy. Gastrointestinal Endoscopy (2014) 4. Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021) 5. Ghasemabadi, A., Janjua, M.K., Salameh, M., Niu, D.: Learning truncated causal history model for video restoration. In: Advances in Neural Information Processing Systems (2024) 6. Hwang, K., Seo, Y.H., Ahn, J., Kim, P., Jeong, K.H.: Frequency selection rule for high definition and high frame rate Lissajous scanning. Scientific Reports (2017) 7. Hwang, K., Seo, Y.H., Kim, D.Y., Ahn, J., Lee, S., Han, K.H., Lee, K.H., Jon, S., Kim, P., Yu, K.E., et al.: Handheld endomicroscope using a fiber-optic harmonograph enables real-time and in vivo confocal imaging of living cell morphology and capillary perfusion. Microsystems & Nanoengineering (2020) 8. Jeon, J., Kim, H., Jang, H., Hwang, K., Kim, K., Park, Y.G., Jeong, K.H.: Handheld laser scanning microscope catheter for real-time and in vivo confocal microscopy using a high definition high frame rate Lissajous mems mirror. Biomedical Optics Express (2022) 9. Kong, L., Dong, J., Ge, J., Li, M., Pan, J.: Efficient frequency domain-based transformers for high-quality image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 10. Latt, W.T., Newton, R.C., Visentini-Scarzanella, M., Payne, C.J., Noonan, D.P., Shang, J., Yang, G.Z.: A hand-held instrument to maintain steady tissue contact during probe-based confocal laser endomicroscopy. IEEE Transactions on Biomedical Engineering (2011) 11. Liang, J., Fan, Y., Xiang, X., Ranjan, R., Ilg, E., Green, S., Cao, J., Zhang, K., Timofte, R., Gool, L.V.: Recurrent video restoration transformer with guided deformable attention. In: Advances in Neural Information Processing Systems (2022) 12. Liu, X., Sun, G., Wang, C., Yuan, Y., Konukoglu, E.: MedVSR: Medical video super-resolution with cross state-space propagation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2025) 13. Loewke, N.O., Qiu, Z., Mandella, M.J., Ertsey, R., Loewke, A., Gunaydin, L.A., Rosenthal, E.L., Contag, C.H., Solgaard, O.: Software-based phase control, videorate imaging, and real-time mosaicing with a Lissajous-scanned confocal microscope. IEEE Transactions on Medical Imaging (2019)
10
M. Lee et al.
14. Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications (2024) 15. Maggioni, M., Huang, Y., Li, C., Xiao, S., Fu, Z., Song, F.: Efficient multistage video denoising with recurrent spatio-temporal fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021) 16. Mahé, J., Linard, N., Tafreshi, M.K., Vercauteren, T., Ayache, N., Lacombe, F., Cuingnet, R.: Motion-aware mosaicing for confocal laser endomicroscopy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2015) 17. Mao, X., Liu, Y., Liu, F., Li, Q., Shen, W., Wang, Y.: Intriguing findings of frequency selection for image deblurring. In: Proceedings of the AAAI Conference on Artificial Intelligence (2023) 18. Marques, M.J., Hughes, M.R., Vyas, K., Thrapp, A., Zhang, H., Bradu, A., Gelikonov, G., Giataganas, P., Payne, C.J., Yang, G.Z., et al.: En-face optical coherence tomography/fluorescence endomicroscopy for minimally invasive imaging using a robotic scanner. Journal of Biomedical Optics (2019) 19. Ozyoruk, K.B., Gokceler, G.I., Bobrow, T.L., Coskun, G., Incetan, K., Almalioglu, Y., Mahmood, F., Curto, E., Perdigoto, L., Oliveira, M., Sahin, H., Araujo, H., Alexandrino, H., Durr, N.J., Gilbert, H.B., Turan, M.: EndoSLAM dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videos. Medical Image Analysis (2021) 20. Qi, C., Chen, J., Yang, X., Chen, Q.: Real-time streaming video denoising with bidirectional buffers. In: Proceedings of the ACM International Conference on Multimedia (2022) 21. Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017) 22. Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2015) 23. Stoeve, M., Aubreville, M., Oetter, N., Knipfer, C., Neumann, H., Stelzle, F., Maier, A.: Motion artifact detection in confocal laser endomicroscopy images. Bildverarbeitung für die Medizin (2018) 24. Studier-Fischer, A., Seidlitz, S., Sellner, J., Bressan, M., Özdemir, B., Ayala, L., Odenthal, J., Knoedler, S., Kowalewski, K.F., Haney, C.M., Salg, G., Dietrich, M., Kenngott, H., Gockel, I., Hackert, T., Müller-Stich, B.P., Maier-Hein, L., Nickel, F.: HeiPorSPECTRAL - the Heidelberg porcine hyperspectral imaging dataset of 20 physiological organs. Scientific Data (2023) 25. Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-net: Cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018) 26. Wallace, M.B., Fockens, P.: Probe-based confocal laser endomicroscopy. Gastroenterology (2009) 27. Zitova, B., Flusser, J.: Image registration methods: A survey. Image and Vision Computing (2003)