高清几何缓存多尺度特征融合的渲染超分方法
作者:
作者简介:

张浩南(2000-), 男, 硕士生, 主要研究领域为计算机图形学.
过洁(1986-), 男, 博士, 副研究员, CCF高级会员, 主要研究领域为计算机图形学, 虚拟现实.
覃浩宇(1998-), 男, 硕士生, 主要研究领域为计算机图形学.
傅锡豪(1997-), 男, 硕士生, 主要研究领域为实时渲染.
郭延文(1980-), 男, 博士, 教授, 博士生导师, CCF专业会员, 主要研究领域为计算机图形学, 三维视觉

通讯作者:

过洁, E-mail: guojie@nju.edu.cn

中图分类号:

TP391

基金项目:

国家自然科学基金(61972194, 62032011); 江苏省自然科学基金(BK20211147)


Super-resolution Method for Rendered Contents by Multi-scale Feature Fusion with High-resolution Geometry Buffers
Author:
  • 摘要
  • | |
  • 访问统计
  • |
  • 参考文献 [46]
  • |
  • 相似文献 [20]
  • | | |
  • 文章评论
    摘要:

    人们对图像显示设备高分辨率和逼真视觉感知的需求随着现代信息技术的发展日益增长, 这对计算机软硬件提出了更高要求, 也为渲染技术在性能与工作负载上带来更多挑战. 利用深度神经网络等机器学习技术对渲染图像进行质量改进和性能提升成为了计算机图形学热门的研究方向, 其中通过网络推理将低分辨率图像进行上采样获得更加清晰的高分辨率图像是提升图像生成性能并保证高清细节的一个重要途径. 而渲染引擎在渲染流程中产生的几何缓存(geometry buffer, G-buffer)包含较多的语义信息, 能够帮助网络有效地学习场景信息与特征, 从而提升上采样结果的质量. 设计一个基于深度神经网络的低分辨率渲染内容的超分方法. 除了当前帧的颜色图像, 其使用高分辨率的几何缓存来辅助计算并重建超分后的内容细节. 所提方法引入一种新的策略来融合高清缓存与低清图像的特征信息, 在特定的融合模块中对不同种特征信息进行多尺度融合. 实验验证所提出的融合策略和模块的有效性, 并且, 在和其他图像超分辨率方法的对比中, 所提方法体现出明显的优势, 尤其是在高清细节保持方面.

    Abstract:

    With the development of modern information technology, people’s demand for high resolution and realistic visual perception of image display devices has increased, which has put forward higher requirements for computer software and hardware and brought many challenges to rendering technology in terms of performance and workload. Using machine learning technologies such as deep neural networks to improve the quality and performance of rendered images has become a popular research method in computer graphics, while upsampling low-resolution images through network inference to obtain clearer high-resolution images is an important way to improve image generation performance and ensure high-resolution details. The geometry buffers (G-buffers) generated by the rendering engine in the rendering process contain much semantic information, which help the network learn scene information and features effectively and then improve the quality of upsampling results. In this study, a super-resolution method for rendered contents in low resolution based on deep neural networks is designed. In addition to the color image of the current frame, the method uses high-resolution G-buffers to assist in the calculation and reconstruct the high-resolution content details. The method also leverages a new strategy to fuse the features of high-resolution buffers and low-resolution images, which implements a multi-scale fusion of different feature information in a specific fusion module. Experiments demonstrate the effectiveness of the proposed fusion strategy and module, and the proposed method shows obvious advantages, especially in maintaining high-resolution details, when compared with other image super-resolution methods.

    参考文献
    [1] Kaplanyan AS, Sochenov A, Leimkühler T, Okunev M, Goodall T, Rufo G. DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Transactions on Graphics, 2019, 38(6): 212. [doi: 10.1145/3355089.3356557]
    [2] Jo Y, Oh SW, Kang J, Kim SJ. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 3224–3232.
    [3] Dong C, Loy CC, Tang XO. Accelerating the super-resolution convolutional neural network. In: Proc. of the 14th European Conf. on Computer Vision. Amsterdam: Springer, 2016. 391–407.
    [4] Epic Games. 2022. https://www.unrealengine.com/.UnrealEngine
    [5] Burnes A. NVIDIA DLSS 2.0: A big leap in AI rendering. 2020. https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-big-leap-in-ai-rendering/.NVIDIA
    [6] DirectX-Specs, Variable Rate Shading. 2022. https://microsoft.github.io/DirectX-Specs/d3d/VariableRateShading.html.Microsoft
    [7] Xiao K, Liktor G, Vaidyanathan K. Coarse pixel shading with temporal supersampling. In: Proc. of the 2018 ACM SIGGRAPH Symp. on Interactive 3D Graphics and Games. Montreal: ACM, 2018.
    [8] Sakai H, Nabata K, Yasuaki S, Iwasaki K. Error estimation for many-light rendering with supersampling. In: Proc. of the 2018 SIGGRAPH Asia Technical Briefs. Tokyo: ACM, 2018.
    [9] Xiao L, Nouri S, Chapman M, Fix A, Lanman D, Kaplanyan A. Neural supersampling for real-time rendering. ACM Transactions on Graphics, 2020, 39(4): 142. [doi: 10.1145/3386569.3392376]
    [10] Kalantari NK, Bako S, Sen P. A machine learning approach for filtering Monte Carlo noise. ACM Transactions on Graphics, 2015, 34(4): 122. [doi: 10.1145/2766977].
    [11] Nalbach O, Arabadzhiyska E, Mehta D, Seidel HP, Ritschel T. Deep shading: Convolutional neural networks for screen space shading. Computer Graphics Forum, 2017, 36(4): 65–78. [doi: 10.1111/cgf.13225]
    [12] Isobe T, Zhu F, Jia X, Wang SJ. Revisiting temporal modeling for video super-resolution. arXiv:2008.05765, 2020.
    [13] Haris M, Shakhnarovich G, Ukita N. Space-time-aware multi-resolution video enhancement. In: Proc. of the 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020. 2856–2865.
    [14] Liao RJ, Tao X, Li RY, Ma ZY, Jia JY. Video super-resolution via deep draft-ensemble learning. In: Proc. of the 2015 IEEE Int’l Conf. on Computer Vision. Santiago: IEEE, 2015. 531–539.
    [15] Chan KCK, Zhou SC, Xu XY, Loy CC. BasicVSR++: Improving video super-resolution with enhanced propagation and alignment. In: Proc. of the 2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022. 5962–5971.
    [16] Dong C, Loy CC, He KM, Tang XO. Learning a deep convolutional network for image super-resolution. In: Proc. of the 13th European Conf. on Computer Vision. Zurich: Springer, 2014. 184–199.
    [17] Lim B, Son S, Kim H, Nah S, Lee KM. Enhanced deep residual networks for single image super-resolution. In: Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE. 2017. 1132–1140.
    [18] Zhang YL, Li KP, Li K, Wang LC, Zhong BN, Fu, Y. Image super-resolution using very deep residual channel attention networks. In: Proc. of the 15th European Conf. on Computer Vision (ECCV). Munich: Springer, 2018. 294–310.
    [19] Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, AitkenA, Tejani A, Totz J, Wang ZH and Shi WZ. Photo-realistic single image super-resolution using a generative adversarial network. In: Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 105–114.
    [20] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.
    [21] Weng WH, Zhu X. INet: Convolutional networks for biomedical image segmentation. IEEE Access, 2021, 9: 16591–166603. [doi: 10.1109/ACCESS.2021.3053408]
    [22] 丁玲, 丁世飞, 张健, 张子晨. 使用VGG能量损失的单图像超分辨率重建. 软件学报, 2021, 32(11): 3659–3668. http://www.jos.org.cn/1000-9825/6053.htm
    Ding L, Ding SF, Zhang J, Zhang ZC. Single image super-resolution reconstruction based on VGG energy loss. Ruan Jian Xue Bao/Journal of Software, 2021, 32(11): 3659–3668. (in Chinese with English abstract) http://www.jos.org.cn/1000-9825/6053.htm
    [23] 潘宗序, 禹晶, 肖创柏, 孙卫东. 基于自适应多字典学习的单幅图像超分辨率算法. 电子学报, 2015, 43(2): 209–216. [doi: 10.3969/j.issn.0372-2112.2015.02.001]
    Pan ZX, Yu J, Xiao CB, Sun WD. Single image super resolution based on adaptive multi-dictionary learning. Acta Electronica Sinica, 2015, 43(2): 209–216. (in Chinese with English abstract) [doi: 10.3969/j.issn.0372-2112.2015.02.001]
    [24] Caballero J, Ledig C, Aitken A, Acosta A, Totz J, Wang ZH, Shi WZ. Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 2848–2857.
    [25] Wang XT, Chan KCK, Yu K, Dong C, Loy CC. EDVR: Video restoration with enhanced deformable convolutional networks. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW). Long Beach: IEEE, 2019. 1954–1963.
    [26] Yi P, Wang ZY, Jiang K, Jiang JJ, Ma JY. Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. In: Proc. of the 2019 IEEE/CVF Int’l Conf. on Computer Vision. Seoul: IEEE, 2019. 3106–3115.
    [27] Lu YF, Xie N, Shen HT. DMCR-GAN: Adversarial denoising for Monte Carlo renderings with residual attention networks and hierarchical features modulation of auxiliary buffers. In: SIGGRAPH Asia 2020 Technical Communications. ACM, 2020. 5.
    [28] Chaitanya CRA, Kaplanyan AS, Schied C, Salvi M, Lefohn A, Nowrouzezahrai D, Aila T. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics, 2017, 36(4): 98. [doi: 10.1145/3072959.3073601]
    [29] Guo J, Fu XH, Lin LQ, Ma HJ, Guo YW, Liu SQ, Yan LQ. ExtraNet: Real-time extrapolated rendering for low-latency temporal supersampling. ACM Transactions on Graphics, 2021, 40(6): 278. [doi: 10.1145/3478513.3480531]
    [30] 杜文俊, 冯结青. 面向延迟着色的统一反走样算法. 计算机辅助设计与图形学学报, 2016, 28(1): 58–67. [doi: 10.3969/j.issn.1003-9775.2016.01.008]
    Du WJ, Feng JQ. A unified anti-aliasing method for deferred shading. Journal of Computer-Aided Design & Computer Graphics, 2016, 28(1): 58–67. (in Chinese with English abstract) [doi: 10.3969/j.issn.1003-9775.2016.01.008]
    [31] 邵鹏, 周伟, 李光泉, 吴志健. 一种后处理式的改进抗锯齿算法. 计算机科学, 2018, 45(11A): 218–221, 225. (查阅网上资料, 期不确定是否正确, 请确认)
    Shao P, Zhou W, Li GQ, Wu ZJ. Improved anti-aliasing algorithm based on deferred shading. Computer Science, 2018, 45(11A): 218–221, 225. (in Chinese with English abstract)
    [32] Briedis KM, Djelouah A, Meyer M, McGonigal I, Gross M, Schroers C. Neural frame interpolation for rendered content. ACM Transactions on Graphics, 2021, 40(6): 239. [doi: 10.1145/3478513.3480553]
    [33] Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 7132–7141.
    [34] He KM, Zhang XY, Ren SQ, Sun, J. Deep residual learning for image recognition. In: Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 770–778.
    [35] Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. [doi: 10.1109/TIP.2003.819861]
    [36] Zhang R, Isola P, Efros AA, Shechtman E, Wang O. The unreasonable effectiveness of deep features as a perceptual metric. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018. 586–595.
    [37] Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen J, Lin ZM, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai JJ, Chintala, S. PyTorch: An imperative style, high-performance deep learning library. In: Proc. of the 33rd Int’l Conf. on Neural Information Processing Systems. Vancouver: Curran Associates Inc., 2019. 721.
    [38] Kingma DP, Ba J. Adam: A method for stochastic optimization. In: Proc. of the 3rd Int’l Conf. on Learning Representations. San Diego: ICLR, 2015.
    [39] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Proc. of the 13th Int’l Conf. on Artificial Intelligence and Statistics. Sardinia: JMLR.org, 2010. 249–256.
    [40] Open Neural Network Exchange. 2022. https://onnx.ai/.ONNX
    [41] NVIDIA TensorRT. 2022. https://github.com/NVIDIA/TensorRT.NVIDIA
    [42] NVIDIA. 2022. https://developer.nvidia.com/cudnn.NVIDIAcuDNN
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

张浩南,过洁,覃浩宇,傅锡豪,郭延文.高清几何缓存多尺度特征融合的渲染超分方法.软件学报,2024,35(6):3052-3068

复制
分享
文章指标
  • 点击次数:662
  • 下载次数: 1832
  • HTML阅读次数: 884
  • 引用次数: 0
历史
  • 收稿日期:2022-08-20
  • 最后修改日期:2022-10-08
  • 在线发布日期: 2023-08-09
  • 出版日期: 2024-06-06
文章二维码
您是第20060719位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号