Research Progress of Video Saliency Detection
Author:
Affiliation:

Fund Project:

National Natural Science Foundation of China (61722112, 61520106002, 61332016, 61620106009, 61602344); National Key Research and Development Program of China (2017YFB1002900)

  • Article
  • | |
  • Metrics
  • |
  • Reference [88]
  • |
  • Related [20]
  • | | |
  • Comments
    Abstract:

    As a hot topic in computer vision community, video saliency detection aims at continuously discovering the motion-related salient objects from the video sequences by considering the spatial and temporal information jointly. Due to the complex backgrounds, diverse motion patterns, and camera motions in video sequences, video saliency detection is a more challenging task than image saliency detection. This paper summarizes the existing methods of video saliency detection, introduces the relevant experimental datasets, and analyze the performance of some state-of-the-art methods on different datasets. First, an introduction of low-level cues based video saliency detection methods including transform analysis based method, sparse representation based method, information theory based method and visual prior based method, is presented. Then, the learning-based video saliency detection methods, which mainly include traditional methods and depth learning based methods, are discussed. Subsequently, the commonly used datasets for video saliency detection are presented, and four evaluation measures are introduced. Moreover, some state-of-the-art methods with qualitative and quantitative comparisons on different datasets are analyzed in experiments. Finally, the key issues of video saliency detection are summarized, and the future development trend is discussed.

    Reference
    [1] Gao Y, Shi MJ, Tao D, Xu C. Database saliency for fast image retrieval. IEEE Trans. on Multimedia, 2015,17(3):359-369.
    [2] Ren ZX, Gao SH, Chia LT, Tsang IWH. Region-Based saliency detection and its application in object recognition. IEEE Trans. on Circuits and Systems for Video Technology, 2014,24(5):769-779.
    [3] Fu HZ, Xu D, Lin S, Liu J. Object-Based RGBD image cosegmentation with mutex constraint. In:Proc. of the CVPR. 2015. 4428-4436.
    [4] Lei JJ, Wu M, Zhang CQ, Wu F, Ling N, Hou CP. Depth-Preserving stereo image retargeting based on pixel fusion. IEEE Trans. on on Multimedia, 2017,19(7):1442-1453.
    [5] Lei JJ, Zhang CC, Fang YM, Gu ZY, Ling N, Hou CP. Depth sensation enhancement for multiple virtual view rendering. IEEE Trans. on Multimedia, 2015,17(4):457-469.
    [6] Xiao DG, Xin C, Zhang T, Zhu H, Li XL. Saliency texture structure descriptor and its application in pedestrian detection. Ruan Jian Xue Bao/Journal of Software, 2014,25(3):675-689(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/4438.htm[doi:10.13328/j.cnki.jos.004438]
    [7] Gu K, Wang SQ, Yang H, Lin WS, Zhai GT, Yang XK, Zhang WJ. Saliency-Guided quality assessment of screen content images. IEEE Trans. on Multimedia, 2016,18(6):1098-1110.
    [8] Han S, Vasconcelos N. Image compression using object-based regions of interest. In:Proc. of the ICIP. 2006. 3097-3100.
    [9] Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. on Pattern Analysis and Machine Intelligence, 1998,20(11):1254-1259.
    [10] Ma YF, Zhang HJ. Contrast-Based image attention analysis by using fuzzy growing. In:Proc. of the ACM MM. 2003. 374-381.
    [11] Zhang P, Wang RS. Detecting salient regions based on location shift and extent trace. Ruan Jian Xue Bao/Journal of Software, 2004,15(6):891-898(in Chinese with English abstract). http://www.jos.org.cn/1000-9825/15/891.htm
    [12] Harel J, Koch C, Perona P. Graph-Based visual saliency. In:Proc. of the ANIPS. 2006. 545-552.
    [13] Zhai Y, Shah M. Visual attention detection in video sequences using spatiotemporal cues. In:Proc. of the ACM MM. 2006. 815-824.
    [14] Hou XD, Zhang LQ. Saliency detection:A spectral residual approach. In:Proc. of the CVPR. 2007.
    [15] Liu T, Sun J, Zheng NN, Tang XO. Shum HY. Learning to detect a salient object. In:Proc. of the CVPR. 2007.
    [16] Achanta R, Hemami S, Estrada F, Susstrunk S. Frequency-Tuned salient region detection. In:Proc. of the CVPR. 2009. 1597-1604.
    [17] Cheng MM, Zhang GX, Mitra NJ, Huang X, Hu SM. Global contrast based salient region detection. In:Proc. of the CVPR. 2011. 409-416.
    [18] Zhu WJ, Liang S, Wei YC, Sun J. Saliency optimization from robust background detection. In:Proc. of the CVPR. 2014. 2814-2821.
    [19] Zhou L, Yang ZH, Yuan Q, Zhou ZT, Hu DW. Salient region detection via integrating diffusion-based compactness and local contrast. IEEE Trans. on Image Processing, 2015,24(11):3308-3320.
    [20] Lei JJ, Wang BR, Fang YM, Lin WS, Callet PL, Ling N, Hou CP. A universal framework for salient object detection. IEEE Trans. on Multimedia, 2016,18(9):1783-1795.
    [21] Li XH, Lu HC, Zhang LH, Ruan X, Yang MH. Saliency detection via dense and sparse reconstruction. In:Proc. of the ICCV. 2013. 2976-2983.
    [22] Chen TS, Lin L, Liu LB, Luo XN, Li XL. DISC:Deep image saliency computing via progressive representation learning. IEEE Trans. on Neural Networks and Learning Systems, 2015,27(6):1135-1149.
    [23] He SF, Lau RW, Liu WX, Huang Z, Yang QX. SuperCNN:A superpixelwise convolutional neural network for salient object detection. International Journal of Computer Vision, 2015,115(3):330-344.
    [24] Lee G., Ta YW, Kim J. Deep saliency with encoded low level distance map and high level features. In:Proc. of the CVPR. 2016. 660-668.
    [25] Li GB, Yu YZ. Deep contrast learning for salient object detection. In:Proc. of the CVPR. 2016. 478-487.
    [26] Liu N, Han JW. DHSNet:Deep hierarchical saliency network for salient object detection. In:Proc. of the CVPR. 2016. 678-686.
    [27] Zhang J, Dai YC, Porikli F. Deep salient object detection by integrating multi-level cues. In:Proc. of the WACV. 2017. 1-10.
    [28] Hou QB, Cheng MM, Hu XW, Borji A, Tu ZW, Torr P. Deeply supervised salient object detection with short connections. In:Proc. of the CVPR. 2017. 5300-5309.
    [29] Qin Y, Lu HC, Xu YQ, Wang H. Saliency detection via cellular automata. In:Proc. of the CVPR. 2015. 110-119.
    [30] Li CY, Yuan YC, Cai WD, Xia Y, Feng DD. Robust saliency detection via regularized random walks ranking. In:Proc. of the CVPR. 2015. 2710-2717.
    [31] Kim J, Han D, Tai YW, Kim J. Salient region detection via high-dimensional color transform and local spatial support. IEEE Trans. on Image Processing, 2015,25(1):9-23.
    [32] Guo F, Shen JB, Li XL. Learning to detect stereo saliency. In:Proc. of the ICME. 2014. 1-6.
    [33] Lei JJ, Zhang HL, You L, Hou CP, Wang LH. Evaluation and modeling of depth feature incorporated visual attention for salient object segmentation. Neurocomputing, 2013,120:24-33.
    [34] Cong RM, Lei JJ, Zhang CQ, Huang QM, Cao XC, Hou CP. Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Processing Letters, 2016,23(6):819-823.
    [35] Ju R, Liu Y, Ren TW, Ge L, Wu GS. Depth-Aware salient object detection using anisotropic center-surround difference. Signal Processing:Image Communication, 2015,38:115-126.
    [36] Feng D, Barnes N, You SD, McCarthy C. Local background enclosure for RGB-D salient object detection. In:Proc. of the CVPR. 2016. 2343-2350.
    [37] Fu HZ, Cao XC, Tu ZW. Cluster-Based co-saliency detection. IEEE Trans. on Image Processing, 2013,22(10):3766-3778.
    [38] Cao XC, Tao ZQ, Zhang B, Fu HZ, Feng W. Self-Adaptively weighted co-saliency detection via rank constraint. IEEE Trans. on Image Processing, 2014,23(9):4175-4186.
    [39] Li YJ, Fu KR, Liu Z, Yang J. Efficient saliency-model-guided visual co-saliency detection. IEEE Signal Processing Letters, 2015, 22(5):588-592.
    [40] Huang R, Feng W, Sun JZ. Saliency and co-saliency detection by low-rank multiscale fusion. In:Proc. of the ICME. 2015. 1-6.
    [41] Song HK, Liu Z, Xie YF, Wu L, Huang MK. RGBD co-saliency detection via bagging-based clustering. IEEE Signal Processing Letters, 2016,23(12):1722-1726.
    [42] Cong RM, Lei JJ, Fu HZ, Huang QM, Cao XC, Hou CP. Co-Saliency detection for RGBD images based on multi-constraint feature matching and cross label propagation. IEEE Trans. on Image Processing, 2018,27(2):568-579.
    [43] Hou XD, Zhang LQ. Dynamic visual attention:Searching for coding length increments. In:Proc. of the NIPS. 2008. 681-688.
    [44] Seo HJ, Milanfar P. Static and space-time visual saliency detection by self-resemblance. Journal of Vision, 2009,9(12):1-27.
    [45] Guo CL, Ma Q, Zhang LM. Spatio-Temporal saliency detection using phase spectrum of quaternion Fourier transform. In:Proc. of the CVPR. 2008. 1-8.
    [46] Guo C, Zhang L. A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans. on Image Processing, 2010,19(1):185-198.
    [47] Cui XY, Liu QS, Metaxas DN. Temporal spectral residual:Fast motion saliency detection. In:Proc. of the ACM MM. 2009. 617-620.
    [48] Fang YM, Lin WS, Chen ZZ, Tsai CM, Lin CW. A video saliency detection model in compressed domain. IEEE Trans. on Circuits and Systems for Video Technology, 2014,24(1):27-38.
    [49] Liu YG, Chen YW. Video saliency detection algorithm based on motion spectral residual. Computer Engineering, 2014,40(12):247-250,257(in Chinese with English abstract).
    [50] Qiu GP, Gu XD, Chen ZB, Chen QQ, Wang C. An information theoretic model of patiotemporal visual saliency. In:Proc. of the ICME. 2007. 1806-1809.
    [51] Liu C, Yuen PC, Qiu GP. Object motion detection using information theoretic spatio-temporal saliency. Pattern Recognition, 2009, 42(11):2897-2906.
    [52] Li Y, Zhou Y, Yan JC, Niu ZB, Yang J. Visual saliency based on conditional entropy. In:Proc. of the ACCV. 2009. 246-257.
    [53] Lu HC, Li XH, Zhang LH, Ruan X, Yang MH. Dense and sparse reconstruction error based saliency descriptor. IEEE Trans. on Image Processing, 2016,25(4):1592-1603.
    [54] Li NY, Sun BL, Yu JY. A weighted sparse coding framework for saliency detection. In:Proc. of the CVPR. 2015. 5216-5223.
    [55] Yuan YC, Li CY, Kim J, Cai WD, Feng DD. Dense and sparse labeling with multi-dimensional features for saliency detection. IEEE Trans. on Circuits and Systems for Video Technology, 2018,28(5):1130-1143.[doi:10.1109/TCSVT.2016.2646720]
    [56] Li Y, Zhou Y, Xu L, Yang XC, Yang J. Incremental sparse saliency detection. In:Proc. of the ICIP. 2009. 3093-3096.
    [57] Luo Y, Tian Q. Spatio-Temporal enhanced sparse feature selection for video saliency estimation. In:Proc. of the CVPRW. 2012. 33-38.
    [58] Ren ZX, Chia LT, Rajan D. Video saliency detection with robust temporal alignment and local-global spatial contrast. In:Proc. of the ACM ICMR. 2012. 1-8.
    [59] Ren ZX, Gao SH, Rajan D, Chia LT, Huang Y. Spatiotemporal saliency detection via sparse representation. In:Proc. of the ICME. 2012. 158-163.
    [60] Ren ZX, Gao SH, Chia LT, Rajan D. Regularized feature reconstruction for spatiotemporal saliency detection. IEEE Trans. on Image Processing, 2013,22(8):3120-3132.
    [61] Xue YW, Guo XJ, Cao XC. Motion saliency detection using low-rank and sparse decomposition. In:Proc. of the ICASSP. 2012. 1485-1488.
    [62] Chen CLZ, Li S, Wang YG, Qin H, Hao AM. Video saliency detection via spatial-temporal fusion and low-rank coherency diffusion. IEEE Trans. on Image Processing, 2017,26(7):3156-3170.
    [63] Itti L, Baldi P. A principled approach to detecting surprising events in video. In:Proc. of the CVPR. 2005. 631-637.
    [64] Mahadevan V, Vasconcelos N. Spatiotemporal saliency in dynamic scenes. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2010,32(1):171-177.
    [65] Kim W, Kim C. Spatiotemporal saliency detection using textural contrast and its applications. IEEE Trans. on Circuits and Systems for Video Technology, 2014,24(4):646-659.
    [66] Kim W, Han JJ. Video saliency detection using contrast of spatiotemporal directional coherence. IEEE Signal Processing Letters, 2014,21(10):1250-1254.
    [67] Zhou F, Kang SB, Cohen MF. Time-Mapping using space-time saliency. In:Proc. of the CVPR. 2014. 3358-3365.
    [68] Le TN, Sugimoto A. Region-Based multiscale spatiotemporal saliency for video. arXiv:1708.01589, 2017.
    [69] Xi T, Zhao W, Wang H, Lin WS. Salient object detection with spatiotemporal background priors for video. IEEE Trans. on Image Processing, 2017,26(7):3425-3436.
    [70] Liu Z, Zhang X, Luo SH, Meur OL. Superpixel-Based spatiotemporal saliency detection. IEEE Trans. on Circuits and Systems for Video Technology, 2014,24(9):1522-1540.
    [71] Wang WG, Shen JB, Shao L. Consistent video saliency using local gradient flow optimization and global refinement. IEEE Trans. on Image Processing, 2015,24(11):4185-4196.
    [72] Wang WG, Shen JB, Porikli F. Saliency-Aware geodesic video object segmentation. In:Proc. of the CVPR. 2015. 3395-3402.
    [73] Wang WG, Shen JB, Yang RG, Porikli F. A unified spatiotemporal prior based on geodesic distance for video object segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2018,40(1):20-33.
    [74] Kim H, Kim Y, Sim JY, Kim CS. Spatiotemporal saliency detection for video sequences based on random walk with restart. IEEE Trans. on Image Processing, 2015,24(8):2552-2564.
    [75] Liu Z, Li JH, Ye LW, Sun GL, Shen LQ. Saliency detection for unconstrained videos using superpixel-level graph and spatiotemporal propagation. IEEE Trans. on Circuits and Systems for Video Technology, 2017,27(12):2527-2542.[doi:10.1109/TCSVT.2016.2595324]
    [76] Fang ZM, Cui RY, Jin JX. Video saliency detection algorithm based on biological visual feature and visual psychology theory. Acta Physica Sinica, 2017,66(10):1-14(in Chinese with English abstract).
    [77] Liu T, Yuan ZJ, Sun J, Wang JD, Zheng NN, Tang XO, Shum HY. Learning to detect a salient object. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2011,33(2):353-367.
    [78] Huang CR, Chang YJ, Yang ZX, Lin YY. Video saliency map detection by dominant camera motion removal. IEEE Trans. on Circuits and Systems for Video Technology, 2014,24(8):1336-1349.
    [79] Wang WG, Shen JB, Shao L. Video salient object detection via fully convolutional networks. IEEE Trans. on Image Processing, 2018,27(1):38-49.
    [80] Le TN, Sugimoto A. Video salient object detection using spatiotemporal deep features. arXiv:1708.01447, 2017. 1-13.
    [81] Tsai D, Flagg M, Rehg JM. Motion coherent tracking with multi-label MRF optimization. In:Proc. of the BMVC. 2010. 1-11.
    [82] Li FX, Kim T, Humayun A, Tsai D, Rehg JM. Video segmentation by tracking many figure-ground segments. In:Proc. of the ICCV. 2013. 2192-2199.
    [83] Perazzi F, Pont-Tuset J, McWilliams B, Gool LV, Gross M, Sorkine-Hornung A. A benchmark dataset and evaluation methodology for video object segmentation. In:Proc. of the CVPR. 2016. 724-732.
    附中文参考文献:
    [6] 肖德贵,辛晨,张婷,朱欢,李小乐.显著性纹理结构特征及车载环境下的行人检测.软件学报,2014,25(3):675-689. http://www.jos.org.cn/1000-9825/4438.htm[doi:10.13328/j.cnki.jos.004438]
    [11] 张鹏,王润生.基于视点转移和视区追踪的图像显著区域检测.软件学报,2004,15(6):891-898. http://www.jos.org.cn/1000-9825/15/891.htm
    [49] 刘宇光,陈耀武.基于运动谱残差的视频显著性检测算法.计算机工程,2014,40(12):247-250,257.
    [76] 方志明,崔荣一,金璟璇.基于生物视觉特征和视觉心理学的视频显著性检测算法.物理学报,2017,66(10):1-14.
    Cited by
    Comments
    Comments
    分享到微博
    Submit
Get Citation

丛润民,雷建军,付华柱,王文冠,黄庆明,牛力杰.视频显著性检测研究进展.软件学报,2018,29(8):2527-2544

Copy
Share
Article Metrics
  • Abstract:4937
  • PDF: 12476
  • HTML: 4534
  • Cited by: 0
History
  • Received:October 30,2017
  • Revised:January 04,2018
  • Online: February 08,2018
You are the first2032355Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address:4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code:100190
Phone:010-62562563 Fax:010-62562533 Email:jos@iscas.ac.cn
Technical Support:Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063