Research on Dual-adversarial MR Image Fusion Network Using Pre-trained Model for Feature Extraction
Author:
Affiliation:

  • Article
  • | |
  • Metrics
  • |
  • Reference [51]
  • |
  • Related
  • |
  • Cited by
  • | |
  • Comments
    Abstract:

    With the popularization of multimodal medical images in clinical diagnosis and treatment, fusion technology based on spatiotemporal correlation characteristics has been developed rapidly. The fused medical images not only retain the unique features of source images with various modalities but also strengthen the complementary information, which can facilitate image reading. At present, most methods perform feature extraction and feature fusion by manually defining constraints, which can easily lead to the loss of useful information and unclear details in the fused images. In light of this, a dual-adversarial fusion network using a pre-trained model for feature extraction is proposed in this study to fuse MR-T1/MR-T2 images. The network consists of a feature extraction module, a feature fusion module, and two discriminator network modules. Due to the small scale of the registered multimodal medical image dataset, the feature extraction network cannot be fully trained. In addition, as the pre-trained model has powerful data representation ability, a pre-trained convolutional neural network model is embedded into the feature extraction module to generate the feature map. Then, the feature fusion network fuses the deep features and outputs fused images. Through accurate classification of the source and fused images, the two discriminator networks establish adversarial relations with the feature fusion network separately and eventually encourage it to learn the optimal fusion parameters. The experimental results illustrate the effectiveness of pre-trained technology in this method. Compared with six existing typical fusion methods, the proposed method can generate the fused results of optimal performance in visual effects and quantitative metrics.

    Reference
    [1] Liu H, Xu J, Wu Y, Guo Q, Ibragimov B, Xing L. Learning deconvolutional deep neural network for high resolution medical image reconstruction. Information Sciences. 2018, 468: 142–154.
    [2] Liu H, Wang HO, Wu Y, Xing L. Superpixel region merging based on deep network for medical image segmentation. ACM Trans. on Intelligent Systems and Technology. 2020, 11(4): 1–22.
    [3] 于晓, 刘慧, 吴彦, 张彩明. 基于本质自表示的多视角子空间聚类. 中国科学: 信息科学, 2021, 51(10): 1625–1639.
    Yu X, Liu H, Wu Y, Zhang CM. Intrinsic self-representation for multi-view subspace clustering. Scienta Sinica Informations, 2021, 51(10): 1625–1639 (in Chinese with English abstract).
    [4] Hermessi H, Mourali O, Zagrouba E. Multimodal medical image fusion review: Theoretical background and recent advances. Signal Processing, 2021, 183: 108036. [doi: 10.1016/j.sigpro.2021.108036]
    [5] Zhang H, Xu H, Tian X, Jiang JJ, Ma JY. Image fusion meets deep learning: A survey and perspective. Information Fusion, 2021, 76: 323–336. [doi: 10.1016/j.inffus.2021.06.008]
    [6] Li Y, Zhao JL, Lv ZH, Li JH. Medical image fusion method by deep learning. International Journal of Cognitive Computing in Engineering, 2021, 2: 21–29. [doi: 10.1016/j.ijcce.2020.12.004]
    [7] Prabhakar KR, Srikar VS, Babu RV. DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proc. of the 2017 IEEE Int’l Conf. on Computer Vision. Venice: IEEE, 2017. 4714–4722.
    [8] Li H, Wu XJ. DenseFuse: A fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 2019, 28(5): 2614–2623. [doi: 10.1109/TIP.2018.2887342]
    [9] Han X, Zhang ZY, Ding N, Gu YX, Liu X, Huo YQ, Qiu JZ, Yao Y, Zhang A, Zhang L, Han WT, Huang ML, Jin Q, Lan YY, Liu Y, Liu ZY, Lu ZW, Qiu XP, Song RH, Tang J, Wen JR, Yuan JH, Zhao WX, Zhu J. Pre-trained models: Past, present and future. AI Open, 2021, 2: 225–250. [doi: 10.1016/j.aiopen.2021.08.002]
    [10] Iglovikov V, Shvets A. TernausNet: U-Net with VGG11 encoder pre-trained on ImageNet for image segmentation. arXiv:1801.05746, 2018.
    [11] Chen YC, Li LJ, Yu LC, El Kholy A, Ahmed F, Gan Z, Cheng Y, Liu JJ. UNITER: Universal image-text representation learning. In: Proc. of the 2020 European Conf. on Computer Vision. Glasgow: Springer, 2020. 104–120.
    [12] Deng J, Dong W, Socher R, Li JL, Li K, Li FF. ImageNet: A large-scale hierarchical image database. In: Proc. of the 2009 IEEE Conf. on Computer Vision and Pattern Recognition. Miami: IEEE, 2009. 20–25.
    [13] Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. In: Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 2261–2269.
    [14] Sanjay AR, Soundrapandiyan R, Karuppiah M, Ganapathy R. CT and MRI image fusion based on discrete wavelet transform and Type-2 fuzzy logic. International Journal of Intelligent Engineering and Systems, 2017, 10(3): 355–362. [doi: 10.22266/ijies2017.0630.40]
    [15] Bhatnagar G, Wu QMJ, Liu Z. Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Transactions on Multimedia, 2013, 15(5): 1014–1024. [doi: 10.1109/TMM.2013.2244870]
    [16] Zhang Q, Liu Y, Blum RS, Han JG, Tao DC. Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review. Information Fusion, 2018, 40: 57–75. [doi: 10.1016/j.inffus.2017.05.006]
    [17] Bavirisetti DP, Xiao G, Liu G. Multi-sensor image fusion based on fourth order partial differential equations. In: Proc. of the 20th Int’l Conf. on Information Fusion. Xi’an: IEEE, 2017. 1–9.
    [18] Han JG, Pauwels EJ, de Zeeuw P. Fast saliency-aware multi-modality image fusion. Neurocomputing, 2013, 111: 70–80. [doi: 10.1016/j.neucom.2012.12.015]
    [19] 郑有志, 覃征. 基于二维经验模态分解的医学图像融合算法. 软件学报, 2009, 20(5): 1096–1105. http://www.jos.org.cn/1000-9825/3542.htm
    Zheng YZ, Tan Z. Medical image fusion algorithm based on bidimensional empirical mode decomposition. Ruan Jian Xue Bao/Journal of Software, 2009, 20(5): 1096–1105 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/3542.htm
    [20] Yin M, Liu XN, Liu Y, Chen X. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Transactions on Instrumentation and Measurement, 2019, 68(1): 49–64. [doi: 10.1109/TIM.2018.2838778]
    [21] Zhang H, Xu H, Xiao Y, Guo XJ, Ma JY. Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity. In: Proc. of the 34th AAAI Conf. on Artificial Intelligence. New York: AAAI, 2019. 12797–12804.
    [22] Lahoud F, Süsstrunk S. Zero-learning fast medical image fusion. In: Proc. of the 22nd Int’l Conf. on Information Fusion. Ottawa: IEEE, 2019. 1–8.
    [23] Zhang Y, Liu Y, Sun P, Yan H, Zhao XL, Zhang L. IFCNN: A general image fusion framework based on convolutional neural network. Information Fusion, 2020, 54: 99–118. [doi: 10.1016/j.inffus.2019.07.011]
    [24] Ma JY, Yu W, Liang PW, Li C, Jiang JJ. FusionGAN: A generative adversarial network for infrared and visible image fusion. Information Fusion, 2019, 48: 11–26. [doi: 10.1016/j.inffus.2018.09.004]
    [25] Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. In: Proc. of the 27th Int’l Conf. on Neural Information Processing Systems. Montreal: MIT Press, 2014. 2672–2680.
    [26] Pan SJ, Yang Q. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345–1359. [doi: 10.1109/TKDE.2009.191]
    [27] Raina R, Battle A, Lee H, Packer B, Ng AY. Self-taught learning: Transfer learning from unlabeled data. In: Proc. of the 24th Int’l Conf. on Machine Learning. Corvalis: Association for Computing Machinery, 2007. 759–766.
    [28] Gao J, Fan W, Jiang J, Han JW. Knowledge transfer via multiple model local structure mapping. In: Proc. of the 14th ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data Mining. Las Vegas: Association for Computing Machinery, 2008. 283–291.
    [29] 杜鹏飞, 李小勇, 高雅丽. 多模态视觉语言表征学习研究综述. 软件学报, 2021, 32(2): 327–348. http://www.jos.org.cn/1000-9825/6125.htm
    Du PF, Li XY, Gao YL. Survey on multimodal visual language representation learning. Ruan Jian Xue Bao/Journal of Software, 2021, 32(2): 327–348 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6125.htm
    [30] Hendrycks D, Lee K, Mazeika M. Using pre-training can improve model robustness and uncertainty. In: Proc. of the 36th Int’l Conf. on Machine Learning. Long Beach: PMLR, 2019. 2712–2721.
    [31] 陈佛计, 朱枫, 吴清潇, 郝颖明, 王恩德, 崔芸阁. 生成对抗网络及其在图像生成中的应用研究综述. 计算机学报, 2021, 44(2): 347–369. [doi: 10.11897/SP.J.1016.2021.00347]
    Chen FJ, Zhu F, Wu QX, Hao YM, Wang ED, Cui YG. A survey about image generation with generative adversarial nets. Chinese Journal of Computers, 2021, 44(2): 347–369 (in Chinese with English abstract). [doi: 10.11897/SP.J.1016.2021.00347]
    [32] Ratliff LJ, Burden SA, Sastry SS. Characterization and computation of local Nash equilibria in continuous games. In: Proc. of the 51st Annual Allerton Conf. on Communication, Control, and Computing. Monticello: IEEE, 2013. 917–924.
    [33] Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434, 2015.
    [34] Arjovsky M, Chintala S, Bottou L. Wasserstein GAN. arXiv:1701.07875, 2017.
    [35] Mao XD, Li Q, Xie HR, Lau RYK, Wang Z, Smolley SP. Least squares generative adversarial networks. In: Proc. of the 2017 IEEE Int’l Conf. on Computer Vision. Venice: IEEE, 2017. 2794–2802.
    [36] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.
    [37] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proc. of the 32nd Int’l Conf. on Machine Learning. Lille: PMLR, 2015. 448–456.
    [38] Huynh-Thu Q, Ghanbari M. Scope of validity of PSNR in image/video quality assessment. Electronics Letters, 2008, 44(13): 800–801. [doi: 10.1049/el:20080522]
    [39] Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. [doi: 10.1109/TIP.2003.819861]
    [40] Qu GH, Zhang DL, Yan PF. Information measure for performance of image fusion. Electronics Letters, 2002, 38(7): 313–315. [doi: 10.1049/el:20020212]
    [41] Roberts JW, van Aardt JA, Ahmed FB. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing, 2008, 2(1): 023522. [doi: 10.1117/1.2945910]
    [42] Eskicioglu AM, Fisher PS. Image quality measures and their performance. IEEE Transactions on Communications, 1995, 43(12): 2959–2965. [doi: 10.1109/26.477498]
    [43] Zhao ZB, Yuan JS, Gao Q, Kong YH. Wavelet image de-noising method based on noise standard deviation estimation. In: Proc. of the 2007 Int’l Conf. on Wavelet Analysis and Pattern Recognition. Beijing: IEEE, 2007. 1910–1914.
    [44] Han Y, Cai YZ, Cao Y, Xu XM. A new image fusion performance metric based on visual information fidelity. Information Fusion, 2013, 14(2): 127–135. [doi: 10.1016/j.inffus.2011.08.002]
    [45] Xydeas CS, Petrović V. Objective image fusion performance measure. Electronics Letters, 2000, 36(4): 308–309. [doi: 10.1049/el:20000267]
    [46] Perra C, Massidda F, Giusto DD. Image blockiness evaluation based on Sobel operator. In: Proc. of the 2005 IEEE Int’l Conf. on Image Processing. Genova: IEEE, 2005. 386–389.
    [47] Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Proc. of the 31st Int’l Conf. on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017. 6629–6640.
    Related
    Cited by
Get Citation

刘慧,李珊珊,高珊珊,邓凯,徐岗,张彩明.预训练模型特征提取的双对抗磁共振图像融合网络研究.软件学报,2023,34(5):2134-2151

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:April 18,2022
  • Revised:May 29,2022
  • Online: September 20,2022
  • Published: May 06,2023
You are the first2033798Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address:4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code:100190
Phone:010-62562563 Fax:010-62562533 Email:jos@iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063