分组并行的轻量化实时微观三维形貌重建方法
作者:
通讯作者:

钱宇华,E-mail:jinchengqyh@sxu.edu.cn

基金项目:

国家自然科学基金(62136005, 62006146); 科技创新2030—“新一代人工智能”重大项目(2021ZD0112400); 中央引导地方科技发展资金(YDZJSX20231C001, YDZJSX20231B001)


Grouping Parallel Lightweight Real-time Microscopic 3D Shape Reconstruction Method
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [37]
  • |
  • 相似文献 [20]
  • | | |
  • 文章评论
    摘要:

    微观三维形貌重建作为精密制造领域生产制造的关键环节,其重建过程依赖于高分辨率稠密图像的采集.而面对复杂应用场景的高时效性需求, 高分辨率稠密图像的输入会导致运算量与计算复杂度呈几何倍增加, 无法实现高效率低延时的实时微观三维形貌重建. 针对上述现状, 提出一种分组并行的轻量级实时微观三维形貌重建方法GPLWS-Net. GPLWS-Net以U型网络为基础构造轻量化主干网络, 以并行分组式查询加速三维形貌重建过程, 并针对神经网络结构进行重参数化设计避免重建微观结构的精度损失. 另外, 为弥补现有微观三维重建数据集的缺失, 公开了一组多聚焦微观三维重建数据集(Micro 3D), 其标签数据利用多模态数据融合的方式获取场景高精度的三维结构. 结果表明, 所提出的GPLWS-Net网络不仅可以保证重建精度, 而且在三组公开数据集中相比于其他5类深度学习方法平均耗时降低39.15%, 在Micro 3D数据集中平均耗时降低50.55%, 能够实现复杂微观场景的实时三维形貌重建.

    Abstract:

    Microscopic three-dimensional (3D) shape reconstruction is a crucial step in the field of precision manufacturing. The reconstruction process relies on the acquisition of high-resolution and dense images. Nevertheless, in the face of high efficiency requirements in complex application scenarios, inputting high-resolution dense images will result in geometrically increased computation and complexity, making it difficult to achieve efficient and low-latency real-time microscopic 3D shape reconstruction. In response to this situation, this study proposes a grouping parallelism lightweight real-time microscopic 3D shape reconstruction method GPLWS-Net. The GPLWS-Net constructs a lightweight backbone network based on a U-shaped network and accelerates the 3D shape reconstruction process with parallel group-querying. In addition, the neural network structure is re-parameterized to avoid the accuracy loss of reconstructing the microstructure. Furthermore, to supplement the lack of existing microscopic 3D reconstruction datasets, this study publicly releases a set of multi-focus microscopic 3D reconstruction dataset called Micro 3D. The label data uses multi-modal data fusion to obtain a high- precision 3D structure of the scene. The results show that the GPLWS-Net network can not only guarantee the reconstruction accuracy, but also reduce the average time of 39.15% in the three groups of public datasets and 50.55% in the Micro 3D dataset compared with the other five types of deep learning-based methods, which can achieve real-time 3D shape reconstruction of complex microscopic scenes.

    参考文献
    [1] Huang B, Wang W, Bates M, et al. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science, 2008, 319(5864):810-813.
    [2] Nayar S, Nakagawa Y. Shape from focus. IEEE Trans. on Pattern Analysis and Machine Intelligence, 1994, 16(8):824-831.
    [3] Yan T, Qian YH, Li FJ, et al. Intelligent microscopic 3D shape reconstruction method based on 3D time-frequency transformation. Sci Sin Inform, 2023, 53:282-308 (in Chinese with English abstract).[doi:10.1360/SSI-2021-0386]
    [4] Zhang JF, Yan T, Wang KQ, et al. 3D shape reconstruction from multi depth of filed images:datasets and models. Chinese Journal of Computers, 2023, 46(8):1734-1752 (in Chinese with English abstract).[doi:10.11897/SP.J.1016.2023.01734]
    [5] Yang F, Huang X, Zhou Z. Deep depth from focus with differential focus volume. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2022. 12642-12651.
    [6] Hazirbas C, Soyer SG, Staab MC, et al. Deep depth from focus. In:Proc. of the Asian Conf. on Computer Vision. 2018. 525-541.
    [7] Maximov M, Galim K, Leal-Taixé L. Focus on defocus:bridging the synthetic to real domain gap for depth estimation. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2020. 1071-1080.
    [8] Wang NH, Wang R, Liu YL, et al. Bridging unsupervised and supervised depth from focus via all-in-focus supervision. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2021. 12621-12631.
    [9] Howard AG, Zhu M, Chen B, et al. MobileNets:Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017.
    [10] Zhang X, Zhou X, Lin M, et al. ShuffleNet:An extremely efficient convolutional neural network for mobile devices. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2018. 6848-6856.
    [11] Han K, Wang Y, Tian Q, et al. GhostNet:More features from cheap operations. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2020. 1577-1586.
    [12] Mehta S, Mohammad R. MobileViT:Light-weight, general-purpose, and mobile-friendly vision transformer. arXiv:2110.02178, 2021.
    [13] Lee JY, Park RH. Complex-valued disparity:Unified depth model of depth from stereo, depth from focus, and depth from defocus based on the light field gradient. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2021, 43(3):830-841.
    [14] Muhammad M, Choi TS. Sampling for shape from focus in optical microscopy. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2012, 34(3):564-573.
    [15] Jeon HG, Surh J, Im S, et al. Ring difference filter for fast and noise robust depth from focus. IEEE Trans. on Image Processing, 2020, 29:1045-1060.
    [16] Yan T, Hu Z, Qian YH, et al. 3D shape reconstruction from multifocus image fusion using a multidirectional modified laplacian operator. Pattern Recognition, 2020, 98:107065.
    [17] Yan T, Wu P, Qian YH, et al. Multiscale fusion and aggregation pcnn for 3D shape recovery. Information Sciences, 2020, 536:277-297.
    [18] Minhas R, Mohammed AA, Wu QM. Shape from focus using fast discrete curvelet transform. Pattern Recognition, 2011, 44(4):839-853.
    [19] Ali U, Muhammad TM. Robust focus volume regularization in shape from focus. IEEE Trans. on Image Processing, 2021, 30:7215-7227.
    [20] Moeller M, Benning M, Schnlieb C, et al. Variational depth from focus reconstruction. IEEE Trans. on Image Processing, 2015, 24(12):5369-5378.
    [21] Hu, J, Li S, Gang S. Squeeze-and-excitation networks. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2018. 7132-7141.
    [22] Daquan Z, Hou Q, Chen Y, et al. Rethinking bottleneck structure for efficient mobile network design. arXiv:2007.02269, 2020.
    [23] Chollet F. Xception:Deep learning with depthwise separable convolutions. In:Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition. 2017. 1800-1807.
    [24] Zhang T, Qi GJ, Xiao B, et al. Interleaved group convolutions for deep neural networks. arXiv:1707.02725, 2017.
    [25] Tan M, Le QV. EfficientNet:Rethinking model scaling for convolutional neural networks. arXiv:1905.11946, 2019.
    [26] Han K, Wang Y, Zhang Q, et al. Model rubik's cube:Twisting resolution, depth and width for TinyNets. arXiv:2010.14819, 2020.
    [27] Ma N, Zhang X, Zheng H, et al. ShuffleNet V2:Practical guidelines for efficient CNN architecture design. In:Proc. of the European Conf. on Computer Vision. 2018. 122-138.
    [28] Andrew H, Mark S, Grace C, et al. Searching for MobileNetV3. In:Proc. of the IEEE/CVF Int'l Conf. on Computer Vision. 2019. 1314-1324.
    [29] Chen J, Kao S, He H, et al. Run, don't walk:Chasing higher FLOPS for faster neural networks. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2023.
    [30] Pavan K, Vasu A, Gabriel J, et al. MobileOne:An improved one millisecond mobile backbone. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2023.
    [31] Chen Y, Dai X, Chen D, et al. Mobile-former:Bridging MobileNet and transformer. In:Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2022. 5260-5269.
    [32] Pentland AP. A new sense for depth of field. IEEE Trans. on Pattern Analysis and Machine Intelligence, 1987, 4:523-531.
    [33] Won C, Jeon H. Learning depth from focus in the wild. In:Proc. of the European Conf. on Computer Vision. 2022. 1-18.
    [34] Krizhevsky A, Sutskever I, Hinton G. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(624):84-90.
    附中文参考文献:
    [3] 闫涛, 钱宇华, 李飞江, 等. 三维时频变换视角的智能微观三维形貌重建方法. 中国科学:信息科学, 2023, 53:282-308.[doi:10.1360/SSI-2021-0386]
    [4] 张江峰, 闫涛, 王克琪, 等. 多景深图像聚焦信息的三维形貌重建:数据集与模型. 计算机学报, 2023, 46(8):1734-1752.[doi:10.11897/SP.J.1016.2023.01734]
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

闫涛,高浩轩,张江峰,钱宇华,张临垣.分组并行的轻量化实时微观三维形貌重建方法.软件学报,2024,35(4):1717-1731

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2023-05-14
  • 最后修改日期:2023-07-07
  • 在线发布日期: 2023-09-11
  • 出版日期: 2024-04-06
文章二维码
您是第19727817位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号