基于自监督约束的双尺度真实图像盲去噪算法
作者:
作者简介:

王迪(1992-),女,博士生,主要研究领域为图像复原与增强;潘金山(1985-),男,教授,博士生导师,CCF高级会员,主要研究领域为图像去模糊,图像/视频分析和增强,计算机视觉;唐金辉(1981-),男,教授,博士生导师,CCF杰出会员,主要研究领域为多媒体内容分析

通讯作者:

潘金山,jspan@njust.edu.cn

中图分类号:

TP391

基金项目:

科技创新2030“新一代人工智能”重大项目(2018AAA0102001);国家自然科学基金(61922043,61872421)


Two-scale Real Image Blind Denoising with Self-supervised Constraints
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [36]
  • |
  • 相似文献 [9]
  • | | |
  • 文章评论
    摘要:

    现存的图像去噪算法在处理加性高斯白噪声上已经取得令人满意的效果,然而其在未知噪声强度的真实噪声图像上泛化性能较差.鉴于深度卷积神经网络极大地促进了图像盲去噪技术的发展,针对真实噪声图像提出一种基于自监督约束的双尺度真实图像盲去噪算法.首先,所提算法借助小尺度网络分支得到的初步去噪结果为大尺度分支的图像去噪提供额外的有用信息,以帮助后者实现良好的去噪效果.其次,用于去噪的网络模型由噪声估计子网络和图像非盲去噪子网络构成,其中噪声估计子网络用于预测输入图像的噪声强度,非盲去噪子网络则在所预测的噪声强度指导下进行图像去噪.鉴于真实噪声图像通常缺少对应的清晰图像作为标签,提出了一种基于全变分先验的边缘保持自监督约束和一个基于图像背景一致性的背景自监督约束,前者可通过调节平滑参数将网络泛化到不同的真实噪声数据集上并取得良好的无监督去噪效果,后者则可借助多尺度高斯模糊图像之间的差异信息辅助双尺度网络完成去噪.此外,还提出一种新颖的结构相似性注意力机制,用于引导网络关注图像中微小的结构细节,以便复原出纹理细节更加清晰的真实去噪图像.相关实验结果表明在SIDD,DND和Nam这3个真实基准数据集上,所提的基于自监督的双尺度盲去噪算法无论在视觉效果上还是在量化指标上均优于多种有监督图像去噪方法,且泛化性能也得到了较为明显的提升.

    Abstract:

    Existing image denoising algorithms have achieved decent performance on the images with the additive white Gaussian noise (AWGN), while their generalization ability is not good on real-world images with unknown noise. Motivated by the significant progress of deep convolution neural networks (CNNs) in image denoising, a novel two-scale blind image denoising algorithm is proposed based on self-supervised constraints. Firstly, the proposed algorithm leverages the denoised results from the small-scale network branch to provide additional useful information for the large-scale image denoising, so as to achieve favorable denoised results. Secondly, the used network is composed of a noise estimation subnetwork and an image non-blind denoising subnetwork. The noise estimation subnetwork is firstly used to predict noise map, and then image denoising is carried out through the non-blind denoising subnetwork under the guidance of the corresponding noise map. In view of the fact that the real noise image lacks the corresponding clean image as the label, an edge-preserving self-supervised constraint is proposed based on the total variation (TV) priori, which generalizes the network to different real noisy datasets by adjusting smoothing parameters. To keep the consistency of the image background, a background guidance module (BGM) is proposed, which builds a self-supervised constraint based on the information difference between multi-scale Gaussian blurred images and thus assists the network to complete image denoising. In addition, the structural similarity attention mechanism (SAM) is proposed to guide the network to pay attention to trivial structural details in images, so as to recover real denoised images with cleaner texture details. The relevant experimental results on the SIDD, DND, and Nam benchmarks indicate that the proposed self-supervised blind denoising algorithm is superior to some deep supervised denoising methods, and the generalization performance of the network is improved significantly.

    参考文献
    [1] Dabov K, Foi A, Katkovnik V, Egiazarian K. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 2007, 16(8): 2080–2095. [doi: 10.1109/TIP.2007.901238]
    [2] Dong WS, Zhang L, Shi GM, Li X. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing, 2013, 22(4): 1620-1630. [doi: 10.1109/TIP.2012.2235847]
    [3] Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 2006, 15(12): 3736-3745. [doi: 10.1109/TIP.2006.881969]
    [4] Roth S, Black MJ. Fields of experts: A framework for learning image priors. In: Proc. of the 2005 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. San Diego: IEEE, 2005. 860−867.
    [5] Zoran D, Weiss Y. From learning models of natural image patches to whole image restoration. In: Proc. of the 2011 Int’l Conf. on Computer Vision. Barcelona: IEEE, 2011. 479−486.
    [6] Guo S, Yan ZF, Zhang K, Zuo WM, Zhang L. Toward convolutional blind denoising of real photographs. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 1712−1722.
    [7] Plötz T, Roth S. Benchmarking denoising algorithms with real photographs. In: Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 2750−2759.
    [8] Abdelhamed A, Lin S, Brown MS. A high-quality denoising dataset for smartphone cameras. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 1692−1700.
    [9] Anwar S, Barnes N. Real image denoising with feature attention. In: Proc. of the 2019 IEEE/CVF Int’l Conf. on Computer Vision. Seoul: IEEE, 2019. 3155−3164.
    [10] Buades A, Coll B, Morel JM. A non-local algorithm for image denoising. In: Proc. of the 2005 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. San Diego: IEEE, 2005. 60−65.
    [11] Xing XX, Wang HL, Li J, Zhang XD. Asymptotic non-local means image denoising algorithm. Acta Automatica Sinica, 2020, 46(9): 1952-1960 (in Chinese with English abstract). [doi: 10.16383/j.aas.c190294] 邢笑笑, 王海龙, 李健, 张选德. 渐近非局部平均图像去噪算法. 自动化学报, 2020, 46(9): 1952-1960. [doi: 10.16383/j.aas.c190294]
    [12] Burger HC, Schuler CJ, Harmeling S. Image denoising: Can plain neural networks compete with BM3D? In: Proc. of the the 2012 IEEE Conf. on Computer Vision and Pattern Recognition. Providence: IEEE, 2012. 2392−2399.
    [13] Jiang ZT, Wu X, Zhang SQ. Low-illumination image enhancement based on MR-VAE. Chinese Journal of Computers, 2021, 43(7): 1328-1339 (in Chinese with English abstract). [doi: 10.11897/SP.J.1016.2020.01328] 江泽涛, 伍旭, 张少钦. 一种基于MR-VAE的低照度图像增强方法. 计算机学报, 2020, 43(7): 1328-1339. [doi: 10.11897/SP.J.1016.2020.01328]
    [14] Zhang K, Zuo WM, Chen YJ, Meng DY, Zhang L. Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 2017, 26(7): 3142-3155. [doi: 10.1109/TIP.2017.2662206]
    [15] Zhang K, Zuo WM, Zhang L. FFDnet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing, 2018, 27(9): 4608-4622. [doi: 10.1109/TIP.2018.2839891]
    [16] Liu YN, Zhang SS, Wang CP, Li GY, Yang J. Image restoration based on cascading dense network in contourlet transform domain. Ruan Jian Xue Bao/Journal of Software, 2020, 31(12): 3968-3980 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5866.htm 刘宇男, 张姗姗, 王春鹏, 李广宇, 杨建. 基于级联密集网络的轮廓波变换域图像复原. 软件学报, 2020, 31(12): 3968-3980. http://www.jos.org.cn/1000-9825/5866.htm
    [17] Liu YC, Ma W, Xu SB, Zhang XP. Edge-fidelity image denoising based on convolutional neural network. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(11): 1822-1831 (in Chinese with English abstract). [doi: 10.3724/SP.J.1089.2020.18170] 刘一畅, 马伟, 徐士彪, 张晓鹏. 基于卷积神经网络的边缘保真图像去噪算法. 计算机辅助设计与图形学学报, 2020, 32(11): 1822-1831. [doi: 10.3724/SP.J.1089.2020.18170]
    [18] Mildenhall B, Barron JT, Chen JW, Sharlet D, Ng R, Carroll R. Burst denoising with kernel prediction networks. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 2502−2510.
    [19] Zhao YY, Jiang ZQ, Men AD, Ju GD. Pyramid real image denoising network. In: Proc. of the 2019 IEEE Visual Communications and Image Processing. Sydney: IEEE, 2019. 1−4.
    [20] Yue ZS, Yong HW, Zhao Q, Meng DY, Zhang L. Variational denoising network: Toward blind noise modeling and removal. In: Proc. of the 2019 Annual Conf. on Neural Information Processing Systems. Vancouver, 2019. 1688−1699.
    [21] Chen JW, Chen JW, Chao HY, Yang M. Image blind denoising with generative adversarial network based noise modeling. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 3155−3164.
    [22] Lin K, Li TH, Liu S, Li G. Real photographs denoising with noise domain adaptation and attentive generative adversarial network. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops. Long Beach: IEEE, 2019. 1717−1721.
    [23] Lehtinen J, Munkberg J, Hasselgren J, Laine S, Karras T, Aittala M, Aila T. Noise2Noise: Learning image restoration without clean data. In: Proc. of the 35th Int’l Conf. on Machine Learning. Stockholm: PMLR, 2018. 2971−2980.
    [24] Krull A, Buchholz TO, Jug F. Noise2Void—Learning denoising from single noisy images. In: Proc. of 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 2124−2132.
    [25] Laine S, Karras T, Lehtinen J, Aila T. High-quality self-supervised deep image denoising. In: Proc. of the 2019 Annual Conf. on Neural Information Processing Systems. Vancouver, 2019. 6968−6978.
    [26] Batson J, Royer L. Noise2Self: Blind denoising by self-supervision. In: Proc. of the 36th Int’l Conf. on Machine Learning. Long Beach: PMLR, 2019. 524−533.
    [27] Soh JW, Cho NI. Variational deep image denoising. arXiv:2104.00965, 2021.
    [28] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. In: Proc. of the 18th Int’l Conf. on Medical Image Computing and Computer-assisted Intervention. Munich: Springer, 2015. 234−241.
    [29] Zhang YL, Tian YP, Kong Y, Zhong BN, Fu Y. Residual dense network for image super-resolution. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 2472−2481.
    [30] Nam S, Hwang Y, Matsushita Y, Kim SJ. A holistic approach to cross-channel image noise modeling and its application to image denoising. In: Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 1683−1691.
    [31] Chen YJ, Pock T. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1256-1272. [doi: 10.1109/TPAMI.2016.2596743]
    [32] Chen GY, Zhu FY, Heng PA. An efficient statistical method for image noise level estimation. In: Proc. of the 2015 IEEE Int’l Conf. on Computer Vision. Santiago: IEEE, 2015. 477−485.
    [33] Abdelhamed A, Lin S, Brown MS. A high-quality denoising dataset for smartphone cameras. In: Proc. of the 2018 IEEE Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 1692–1700.
    [34] Gu SH, Zhang L, Zuo WM, Feng XC. Weighted nuclear norm minimization with application to image denoising. In: Proc. of the 2014 IEEE Conf. on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014. 2862−2869.
    [35] Mittal A, Soundararajan R, Bovik AC. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 2013, 20(3): 209−212.
    [36] Mittal A, Moorthy AK, Bovik AC. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing, 2012, 21(12): 4695-4708. [doi: 10.1109/TIP.2012.2214050]
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

王迪,潘金山,唐金辉.基于自监督约束的双尺度真实图像盲去噪算法.软件学报,2023,34(6):2942-2958

复制
分享
文章指标
  • 点击次数:720
  • 下载次数: 2593
  • HTML阅读次数: 2048
  • 引用次数: 0
历史
  • 收稿日期:2021-03-22
  • 最后修改日期:2021-07-16
  • 在线发布日期: 2022-12-22
  • 出版日期: 2023-06-06
文章二维码
您是第20060275位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号