微信服务号

微信订阅号

2025-4-3- 17
Home > Archive>Volume 33, Issue 1, 2022 >102-128. DOI:10.13328/j.cnki.jos.006304
PDF HTML XML Export Cite reminder
Research on Text Representation in Natural Language Processing
Author:
Affiliation:

Clc Number:

TP391

Fund Project:

National Natural Science Foundation of China (61773276; 61836007)

  • Article
    • | |
  • Metrics
    • |
  • Reference [157]
    • |
  • Related [20]
    • | | |
  • Comments
    Abstract:

    Natural language processing is the core technology of artificial intelligence. Text representation is the basic and necessary work of natural language processing, which affects or even determines the quality and performance of natural language processing systems. This study discusses the basic principle of text representation, the formalization of natural language, the language model, and the connotation and extension of text representation. The technical classification of text representation on a macro level is analyzed. The mainstreams of text representation technologies and methods are analyzed, induced and summarized, including vector space model, topic model, graph-based model, neural network-based model, and representation learning. Event-based, semantic-based, and knowledge-based text representation technologies are also introduced. The development trends and directions of text representation technology are predicted and further discussed. Neural network-based deep learning and representation learning on text will play an important role in natural language processing. The strategy of pre-training and fine-tune optimization will gradually become the mainstream technology. Text representation needs specific analysis according to specific problems. The integration of technology and application is the driving force.

    Reference
    [1] Ray J, Johnny O, Trovati M, Sotiriadis S, Bessis N. The rise of big data science: A survey of techniques, methods and approaches in the field of natural language processing and network theory. Big Data and Cognitive Computing, 2018, 2(3): 22. [doi: 10.3390/bdcc2030022]
    [2] Kim Y, Lee J, Lee EB, Lee JH. Application of natural language processing (NLP) and text-mining of big-data to engineering-procurement-construction (EPC) bid and contract documents. In: Proc. of the 6th Conf. on Data Science and Machine Learning Applications (CDMA). Riyadh: IEEE, 2020. 123–128. [doi: 10.1109/CDMA47397.2020.00027]
    [3] Friederici AD, Chomsky N, Berwick RC, Moro A, Bolhuis JJ. Language, mind and brain. Nature Human Behaviour, 2017, 1(10): 713–722. [doi: 10.1038/s41562-017-0184-4]
    [4] 于剑. 语言与图灵测试. 自动化学报, 2016, 42(5): 668–669. [doi: 10.16383/j.aas.2016.y000004]
    Yu J. Language and Turing test. Acta Automatica Sinica, 2016, 42(5): 668–669 (in Chinese with English abstract). [doi: 10.16383/j.aas.2016.y000004]
    [5] Desaussure F. Course in General Linguistics. New York: McGraw-Hill, Inc., 1965.
    [6] 梁君英, 刘海涛. 语言学的交叉学科研究: 语言普遍性、人类认知、大数据. 浙江大学学报(人文社会科学版), 2016, 46(1): 108–118. [doi: 10.3785/j.issn.1008-942X.CN33-6000/C.2015.10.231]
    Liang JY, Liu HT. Interdisciplinary studies of linguistics: Language universals, human cognition and big-data analysis. Journal of Zhejiang University (Humanities and Sciences), 2016, 46(1): 108–118 (in Chinese with English abstract). [doi: 10.3785/j.issn.1008-942X.CN33-6000/C.2015.10.231]
    [7] Lai YY, Li C, Goldwasser D, Neville J. Better together: Combining language and social interactions into a shared representation. In: Proc. of the TextGraphs-10: The Workshop on Graph-based Methods for Natural Language Processing. San Diego: Association for Computational Linguistics, 2016. 29–33. [doi: 10.18653/v1/W16-1405]
    [8] 孙茂松, 刘挺, 姬东鸿, 穗志方, 赵军, 张钹, 吾守尔·斯拉木, 俞士汶, 朱军, 李建民, 刘洋, 王厚峰, 吐尔根·依布拉音, 刘群, 刘知远. 语言计算的重要国际前沿. 中文信息学报, 2014, 28(1): 1–8. [doi: 10.3969/j.issn.1003-0077.2014.01.001]
    Sun MS, Liu T, Ji DH, Sui ZF, Zhao J, Zhang B, Wushouer S, Yu SW, Zhu J, Li JM, Liu Y, Wang HF, Turgun I, Liu Q, Liu ZY. Frontiers of language computing. Journal of Chinese Information Processing, 2014, 28(1): 1–8 (in Chinese with English abstract). [doi: 10.3969/j.issn.1003-0077.2014.01.001]
    [9] 宗成庆. 统计自然语言处理. 第2版, 北京: 清华大学出版社, 2013. 8.
    Zong CQ. Statistical Natural Language Processing. 2nd ed., Beijing: Tsinghua University Press, 2013. 8 (in Chinese).
    [10] Taskin Z, Al U. Natural language processing applications in library and information science. Online Information Review, 2019, 43(4): 676–690. [doi: 10.1108/OIR-07-2018-0217]
    [11] 褚晓敏, 朱巧明, 周国栋. 自然语言处理中的篇章主次关系研究. 计算机学报, 2017, 40(4): 842–860. [doi: 10.11897/SP.J.1016.2017.00842]
    Chu XM, Zhu QM, Zhou GD. Discourse primary-secondary relationships in natural language processing. Chinese Journal of Computers, 2017, 40(4): 842–860 (in Chinese with English abstract). [doi: 10.11897/SP.J.1016.2017.00842]
    [12] 李仕春. 论世界语言学学术思想变迁之大势. 东岳论丛, 2017, 38(8): 163–168. [doi: 10.15981/j.cnki.dongyueluncong.2017.08.022]
    Li SC. General trend of the change in world linguistic academic thinking. Dongyue Tribune, 2017, 38(8): 163–168 (in Chinese). [doi: 10.15981/j.cnki.dongyueluncong.2017.08.022](查阅所有网上资料, 未找到本条文献英文翻译, 请联系作者确认)
    [13] 赵永刚. 语言的进化与生物语言学进路诠疏——兼评《为什么只有我们: 语言与进化》. 学术探索, 2018, (6): 107–116. [doi: 10.3969/j.issn.1006-723X.2018.06.016]
    Zhao YG. Explanations on the evolution of language and the forward road of bio-linguistic: Concurrent comments on why only us: Language and evolution. Academic Exploration, 2018, (6): 107–116 (in Chinese with English abstract). [doi: 10.3969/j.issn.1006-723X.2018.06.016]
    [14] 冯志伟. 自然语言处理的形式模型. 合肥: 中国科学技术大学出版社, 2010. 1.
    Feng ZW. Formal Models of Natural Language Processing. Hefei: China University of Science and Technology Press, 2010. 1 (in Chinese).
    [15] 张磊, 卫乃兴. 局部语法的演进、现状与前景. 当代语言学, 2018, 20(1): 103–116.
    Zhang L, Wei NX. Local grammar: Evolution, status quo, and prospects. Contemporary Linguistics, 2018, 20(1): 103–116 (in Chinese with English abstract).
    [16] Minsky M. Semantic Information Processing. Cambridge: MIT Press, 1968. 440–441.
    [17] Schank RC. Conceptual Information Processing. Amsterdam: Elsevier Science Inc., 1975. 5–21.
    [18] Berwick RC, Chomsky N. Why Only Us: Language and Evolution. Cambridge: MIT Press, 2015.
    [19] Bendersky M, Croft WB. Modeling higher-order term dependencies in information retrieval using query hypergraphs. In: Proc. of the 35th Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. Virtual Event: Association for Computing Machinery, 2012. 941–950. [doi: 10.1145/2348283.2348408]
    [20] Sidorov G. Syntactic N-Grams in Computational Linguistics. Cham: Springer, 2019. 3–86. [doi: 10.1007/978-3-030-14771-6]
    [21] Barkovich A. Informational linguistics: Computer, internet, artificial intelligence and language. In: Proc. of the ICAIIC. 2019. 8–13.
    [22] Liu J, Lin L, Ren HL, Gu MH, Wang J, Youn G, Kim JU. Building neural network language model with POS-based negative sampling and stochastic conjugate gradient descent. Soft Computing, 2018, 22(20): 6705–6717. [doi: 10.1007/s00500-018-3181-2]
    [23] Bengio Y, Ducharme R, Vincent P, Janvin C. A neural probabilistic language model. The Journal of Machine Learning Research, 2003, 3: 1137–1155.
    [24] Mikolov T, Karafiát M, Burget L, Černocký JH, Khudanpur S. Recurrent neural network based language model. In: Proc. of the INTERSPEECH. Chiba: ISCA, 2010. 1045–1048.
    [25] Sundermeyer M, Schlüter R, Ney H. LSTM neural networks for language modeling. In: Proc. of the INTERSPEECH. 2012. 194–197.
    [26] Wang P, Xu JM, Xu B, Liu CL, Zhang H, Wang FY, Hao HW. Semantic clustering and convolutional neural network for short text categorization. In: Proc. of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Int’l Joint Conf. on Natural Language Processing (Volume 2: Short Papers). Beijing: Association for Computational Linguistics, 2015. 352–357. [doi: 10.3115/v1/P15-2058]
    [27] Jing K, Xu JG, He B. A survey on neural network language models. arXiv: 1906.03591, 2019.
    [28] Samuel S, Roehr-Brackin K, Pak H, Kim H. Cultural effects rather than a bilingual advantage in cognition: A review and an empirical study. Cognitive Science, 2018, 42(7): 2313–2341. [doi: 10.1111/cogs.12672]
    [29] Siew CSQ, Wulff DU, Beckage NM, Kenett YN. Cognitive network science: A review of research on cognition through the Lens of network representations, processes, and dynamics. Complexity, 2019, 2019: 2108423. [doi: 10.1155/2019/2108423]
    [30] Xu YJ. Wittgenstein, phenomenology and cognitive linguistics. Fudan Journal of the Humanities and Social Sciences, 2018, 11(2): 219–236. [doi: 10.1007/s40647-017-0182-y]
    [31] Brenda M. A cognitive perspective on the semantics of near. Review of Cognitive Linguistics, 2017, 15(1): 121–153. [doi: 10.1075/rcl.15.1.06bre]
    [32] Feiman R, Snedeker J. The logic in language: How all quantifiers are alike, but each quantifier is different. Cognitive Psychology, 2016, 87: 29–52. [doi: 10.1016/j.cogpsych.2016.04.002]
    [33] Ali I, Melton A. Graph-based semantic learning, representation and growth from text: A systematic review. In: Proc. of the 13th Int’l Conf. on Semantic Computing (ICSC). Newport Beach: IEEE, 2019. 118–123. [doi: 10.1109/ICOSC.2019.8665592]
    [34] Liu L, Chen J, Fieguth P, Zhao GY, Chellappa R, Pietikäinen M. From BOW to CNN: Two decades of texture representation for texture classification. International Journal of Computer Vision, 2019, 127(1): 74–109. [doi: 10.1007/s11263-018-1125-z]
    [35] Liu ZY, Lin YK, Sun MS. Representation Learning for Natural Language Processing. Singapore: Springer, 2020. [doi: 10.1007/978-981-15-5573-2]
    [36] Mikolov T, Sutskever I, Chen K, Corrado G, Dean J. Distributed representations of words and phrases and their compositionality. In: Proc. of the 26th Int’l Conf. on Neural Information Processing Systems. Sydney: Curran Associates Inc., 2013. 3111–3119.
    [37] Salton G, Wong A, Yang CS. A vector space model for automatic indexing. Communications of the ACM, 1975, 18(11): 613–620. [doi: 10.1145/361219.361220]
    [38] 徐通锵. “字本位”和语言研究. 语言教学与研究, 2005, (6), 1–11.
    Xu TQ. Zi as the basic structural unit and linguistic studies. Language Teaching and Linguistic Studies, 2005, (6): 1–11 (in Chinese with English abstract).
    [39] 张若男. 字本位理论视角下的对外汉语教学研究述评. 现代语文, 2018, (4): 134–139.
    Zhang RN. Review of the research on teaching Chinese as a foreign language under the character-based theory. Modern Chinese, 2018, (4): 134–139 (in Chinese with English abstract).
    [40] Bengio Y, Courville A, Vincent P. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 1798–1828. [doi: 10.1109/TPAMI.2013.50]
    [41] Granados A. Analysis and study on text representation to improve the accuracy of the normalized compression distance. AI Communications, 2012, 25(4): 381–384. [doi: 10.3233/AIC-2012-0529]
    [42] Dourado ÍC, Galante R, Gonçalves MA, da Silva Torres R. Bag of textual graphs (BoTG): A general graph‐based text representation model. Journal of the Association for Information Science and Technology, 2019, 70(8): 817–829. [doi: 10.1002/asi.24167]
    [43] Harris ZS. Distributional structure. WORD, 1954, 10(2–3): 146–162. [doi: 10.1080/00437956.1954.11659520]
    [44] Firth J. A synopsis of linguistic theory, 1930-55. In: Studies in Linguistic Analysis. Special Volume of the Philological Society. Oxford: Blackwell, 1957. 1–31.
    [45] Wang SP, Cai JY, Lin QH, Guo WZ. An overview of unsupervised deep feature representation for text categorization. IEEE Transactions on Computational Social Systems, 2019, 6(3): 504–517. [doi: 10.1109/TCSS.2019.2910599]
    [46] 邱锡鹏. 神经网络与深度学习. 北京: 机械工业出版社, 2020. 4.
    Qiu XP. Neural Networks and Deep Learning. Beijing: China Machine Press, 2020. 4 (in Chinese).
    [47] Chew PA, Bader BW, Helmreich S, Abdelali A, Verzi SJ. An information-theoretic, vector-space-model approach to cross-language information retrieval. Natural Language Engineering, 2011, 17(1): 37–70. [doi: 10.1017/S1351324910000185]
    [48] Tsatsaronis G, Panagiotopoulou V. A generalized vector space model for text retrieval based on semantic relatedness. In: Proc. of the 12th Conf. of the European Chapter of the Association for Computational Linguistics: Student Research Workshop. Athens: Association for Computational Linguistics, 2009. 70–78.
    [49] 董蕊芳, 柳长安, 杨国田. 一种基于改进TF-IDF的SLAM回环检测算法. 东南大学学报(自然科学版), 2019, 49(2): 251–258. [doi: 10.3969/j.issn.1001-0505.2019.02.008]
    Dong RF, Liu CA, Yang GT. TF-IDF based loop closure detection algorithm for SLAM. Journal of Southeast University (Natural Science Edition), 2019, 49(2): 251–258 (in Chinese with English abstract). [doi: 10.3969/j.issn.1001-0505.2019.02.008]
    [50] Niu FG. Basic Co-occurrence latent semantic vector space mode. Journal of Classification, 2019, 36(2): 277–294. [doi: 10.1007/s00357-018-9283-9]
    [51] Hajjem M, Latiri C. Combining IR and LDA topic modeling for filtering microblogs. Procedia Computer Science, 2017, 112: 761–770. [doi: 10.1016/j.procs.2017.08.166]
    [52] Liu ZY, Huang WY, Zheng YB, Sun MS. Automatic keyphrase extraction via topic decomposition. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing (EMNLP’10). Cambridge: Association for Computational Linguistics, 2010. 366–376.
    [53] Ding ZY, Qiu XP, Qi Z, Huang XJ. Topical translation model for microblog hashtag suggestion. In: Proc. of the 23rd Int’l Joint Conf. on Artificial Intelligence. Beijing: AAAI Press, 2013. 2078–2084.
    [54] Blei DM, Ng AY, Jordan MI. Latent dirichlet allocation. The Journal of Machine Learning Research, 2003, 3: 993–1022.
    [55] Pu XJ, Jin R, Wu GS, Han DY, Xue GR. Topic modeling in semantic space with keywords. In: Proc. of the 24th ACM Int’l on Conf. on Information and Knowledge Management. Melbourne: Association for Computing Machinery, 2015. 1141–1150. [doi: 10.1145/2806416.2806584]
    [56] Siu MH, Gish H, Chan A, Belfield W, Lowe S. Unsupervised training of an HMM-based self-organizing unit recognizer with applications to topic classification and keyword discovery. Computer Speech & Language, 2014, 28(1): 210–223. [doi: 10.1016/j.csl.2013.05.002]
    [57] Schenker A, Last M, Bunke H, Kandel A. Graph representations for Web document clustering. In: Proc. of the 1st Iberian Conf. on Pattern Recognition and Image Analysis. Puerto de Andratx: Springer, 2003. 935–942. [doi: 10.1007/978-3-540-44871-6_108]
    [58] Sonawane SS, Kulkarni PA. Graph based representation and analysis of text document: A survey of techniques. International Journal of Computer Applications, 2014, 96(19): 1–8. [doi: 10.5120/16899-6972]
    [59] Chen YY, Lu H, Qiu J, Wang L. A tutorial of graph representation. In: Proc. of the 5th Int’l Conf. on Artificial Intelligence and Security. New York: Springer, 2019. 368–378. [doi: 10.1007/978-3-030-24274-9_33]
    [60] 赵京胜, 张丽, 朱巧明, 周国栋. 中文文学作品中的社会网络抽取与分析. 中文信息学报, 2017, 31(2): 99–106, 116.
    Zhao JS, Li Z, Zhu QM, Zhou GD. Extracting and analyzing social networks from Chinese literary. Journal of Chinese Information Processing, 2017, 31(2): 99–106, 116 (in Chinese with English abstract).
    [61] Mihalcea R, Tarau P. TextRank: Bringing order into texts. In: Proc. of the EMNLP. Barcelona: ACM, 2004. 404–411.
    [62] Page L, Brin S, Motwani R, Winograd T. The PageRank citation ranking: Bringing order to the Web. Stanford: Stanford InfoLab, 1999.
    [63] 赵京胜, 朱巧明, 周国栋, 张丽. 自动关键词抽取研究综述. 软件学报, 2017, 28(9): 2431–2449. http://www.jos.org.cn/1000-9825/5301.htm
    Zhao JS, Zhu QM, Zhou GD, Zhang L. Review of research in automatic keyword extraction. Ruan Jian Xue Bao/Journal of Software, 2017, 28(9): 2431–2449 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5301.htm
    [64] Kleinberg JM. Authoritative sources in a hyperlinked environment. Journal of the ACM, 1999, 46(5): 604–632. [doi: 10.1145/324133.324140]
    [65] Watts DJ, Strogatz SH. Collective dynamics of ‘small-world’ networks. Nature, 1998, 393(6684): 440–442. [doi: 10.1038/30918]
    [66] Barabási AL, Albert R. Emergence of scaling in random networks. Science, 1999, 286(5439): 509–512. [doi: 10.1126/science.286.5439.509]
    [67] Cancho RFI, Solé RV. The small world of human language. Proceedings of the Royal Society B: Biological Sciences, 2001, 268(1482): 2261–2265. [doi: 10.1098/rspb.2001.1800]
    [68] Holovatch Y, Palchykov V. Complex networks of words in fables. In: Kenna R, MacCarron M, MacCarron P. Maths Meets Myths: Quantitative Approaches to Ancient Narratives. Cham: Springer, 2017. 159–175. [doi: 10.1007/978-3-319-39445-9_9]
    [69] Balinsky H, Balinsky A, Simske SJ. Automatic text summarization and small-world networks. In: Proc. of the 11th ACM Symp. on Document Engineering. Limerick: ACM, 2011. 175–184. [doi: 10.1145/2034691.2034731]
    [70] Pardo TAS, Antiqueira L, Nunes MDGV, Oliveira ON, Da Fontoura Costa L. Using complex networks for language processing: The case of summary evaluation. In: Proc. of the 2006 Int’l Conf. on Communications, Circuits and Systems. Guilin: IEEE, 2006. 2678–2682. [doi: 10.1109/ICCCAS.2006.285222]
    [71] Lozano S, Calzada-Infante L, Adenso-Díaz B, García S. Complex network analysis of keywords co-occurrence in the recent efficiency analysis literature. Scientometrics, 2019, 120(2): 609–629. [doi: 10.1007/s11192-019-03132-w]
    [72] Yan JH, Wang CY, Cheng WL, Gao M, Zhou AY. A retrospective of knowledge graphs. Frontiers of Computer Science, 2018, 12(1): 55–74. [doi: 10.1007/s11704-016-5228-9]
    [73] Wang Q, Mao ZD, Wang B, Guo L. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 2017, 29(12): 2724–2743. [doi: 10.1109/TKDE.2017.2754499]
    [74] Liu H, Zhang YZ, Wang YP, Lin Z, Chen YG. Joint character-level word embedding and adversarial stability training to defend adversarial text. Proc. of the AAAI Conf. on Artificial Intelligence, 2020, 34(5): 8384–8391. [doi: 10.1609/aaai.v34i05.6356]
    [75] Hinton GE. Learning distributed representations of concepts. In: Proc. of the 8th Annual Conf. of the Cognitive Science Society. Hillsdale: Lawrence Erlbaum, 1986. 1–12.
    [76] Levy O, Goldberg Y. Neural word embedding as implicit matrix factorization. In: Proc. of the 27th Int’l Conf. on Neural Information Processing Systems. Montreal: MIT Press, 2014. 2177–2185.
    [77] Pennington J, Socher R, Manning C. Glove: Global vectors for word representation. In: Proc. of the 2014 Conf. on Empirical Methods in Natural Language Processing (EMNLP). Doha: Association for Computational Linguistics, 2014. 1532–1543. [doi: 10.3115/v1/D14-1162]
    [78] Le Q, Mikolov T. Distributed representations of sentences and documents. In: Proc. of the 31st Int’l Conf. on Machine Learning. Beijing, 2014. II-1188–II-1196.
    [79] Joulin A, Grave E, Bojanowski P, Mikolov T. Bag of tricks for efficient text classification. arXiv: 1607.01759, 2016.
    [80] Vinyals O, Le Q. A neural conversational model. arXiv: 1506.05869, 2015.
    [81] Shen DH, Wang GY, Wang WL, Min MR, Su QL, Zhang YZ, Li CY, Henao R, Carin L. Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. In: Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne: Association for Computational Linguistics, 2018. 440–450. [doi: 10.18653/v1/P18-1041]
    [82] Peters ME, Ammar W, Bhagavatula C, Power R. Semi-supervised sequence tagging with bidirectional language models. arXiv: 1705.00108, 2017.
    [83] Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L. Deep contextualized word representations. arXiv: 1802.05365, 2018.
    [84] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Proc. of the 31st Int’l Conferenc. on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017. 6000–6010.
    [85] Dai ZH, Yang ZL, Yang YM, Carbonell J, Le QV, Salakhutdinov R. Transformer-XL: Attentive language models beyond a fixed-length context. arXiv: 1901.02860, 2019.
    [86] Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training. https://www.cs.ubcca/~amuham01/LING530/papers/radford2018improving.pdf.
    [87] Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf
    [88] Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis: Association for Computational Linguistics, 2019. 4171–4186. [doi: 10.18653/v1/N19-1423]
    [89] Yang ZL, Dai ZH, Yang YM, Carbonell J, Salakhutdinov R, Le QV. XLNet: Generalized autoregressive pretraining for language understanding. In: Proc. of the 33rd Conf. on Neural Information Processing Systems. Vancouver, 2019. 5754–5764.
    [90] 张政馗, 庞为光, 谢文静, 吕鸣松, 王义. 面向实时应用的深度学习研究综述. 软件学报, 2020, 31(9): 2654–2677. http://www.jos.org.cn/1000-9825/5946.htm
    Zhang ZK, Pang WG, Xie WJ, Lv MS, Wang Y. Deep learning for real-time applications: A survey. Ruan Jian Xue Bao/Journal of Software, 2020, 31(9): 2654–2677. (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5946.htm
    [91] Wei W, Wu JS, Zhu CS. Special issue on deep learning for natural language processing. Computing, 2020, 102(3): 601–603. [doi: 10.1007/s00607-019-00788-3]
    [92] 李枫林, 柯佳. 基于深度学习的文本表示方法. 情报科学, 2019, 37(1): 156–164. [doi: 10.13833/j.issn.1007-7634.2019.01.024]
    Li FL, Ke J. Text representation method based on deep learning. Information Science, 2019, 37(1): 156–164 (in Chinese with English abstract). [doi: 10.13833/j.issn.1007-7634.2019.01.024]
    [93] Malte A, Ratadiya P. Evolution of transfer learning in natural language processing. arXiv: 1910.07370, 2019.
    [94] Yilmaz S, Toklu S. A deep learning analysis on question classification task using Word2Vec representations. Neural Computing and Applications, 2020, 32(7): 2909–2928. [doi: 10.1007/s00521-020-04725-w]
    [95] Hewitt J, Manning CD. A structural probe for finding syntax in word representations. In: Proc. of NAACL-HLT. Minneapolis: Association for Computational Linguistics, 2019. 4129–4138.
    [96] Ltaifa IB, Hlaoua L, Romdhane LB. Hybrid deep neural network-based text representation model to improve microblog retrieval. Cybernetics and Systems, 2020, 51(2): 115–139. [doi: 10.1080/01969722.2019.1705548]
    [97] Schmidhuber J. Deep learning in neural networks: An overview. Neural Networks, 2015, 61: 85–117. [doi: 10.1016/j.neunet.2014.09.003]
    [98] Bengio Y, Lamblin O, Popovici D, Larochelle H. Greedy layer-wise training of deep networks. In: Proc. of the 19th Int’l Conf. on Neural Information Processing Systems. Cambridge: MIT Press, 2006. 153–160.
    [99] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436–444. [doi: 10.1038/nature14539]
    [100] Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G. The graph neural network model. IEEE Transactions on Neural Networks, 2009, 20(1): 61–80. [doi: 10.1109/TNN.2008.2005605]
    [101] Goyal P, Ferrara E. Graph embedding techniques, applications, and performance: A survey. Knowledge-Based Systems, 2018, 151: 78–94. [doi: 10.1016/j.knosys.2018.03.022]
    [102] Niepert M, Ahmed M, Kutzkov K. Learning convolutional neural networks for graphs. In: Proc. of the 33rd Int’l Conf. on Machine Learning. New York, 2016. 2014–2023.
    [103] Gui J, Sun ZN, Wen YG, Tao DC, Ye JP. A review on generative adversarial networks: Algorithms, theory, and applications. Journal of Latex Classifiers, 2015, 14(8): 1–28.
    [104] Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial networks. In: Advances in Neural Information Processing Systems. Montreal, 2014. 2672–2680.
    [105] Zhang XM, Zhu XB, Zhang XY, Zhang NG, Li P, Wang L. SegGAN: Semantic segmentation with generative adversarial network. In: Proc. of the 4th IEEE Int’l Conf. on Multimedia Big Data (BigMM). Xi’an: IEEE, 2018. 1–5. [doi: 10.1109/BigMM.2018.8499105]
    [106] Wang HW, Wang JL, Wang JL, Zhao M, Zhang WN, Zhang FZ, Xie X, Guo MY. GraphGAN: Graph representation learning with generative adversarial nets. IEEE Transactions on Knowledge and Data Engineering, 2017: 99–117. (查阅所有网上资料, 未找到本条文献卷期页码信息, 请联系作者补充并核对文献类型是否正确)
    [107] Wang C, Deng CY, Vladimir I. SAG-VAE: End-to-end joint inference of data representations and feature relations. In: Proc. of the 2020 Int’l Joint Conf. on Neural Networks (IJCNN). Glasgow: IEEE, 2020. 1–9. [doi: 10.1109/IJCNN48605.2020.9207154]
    [108] Iancu B, Ruiz L, Ribeiro A, Isufi E. Graph-adaptive activation functions for graph neural networks. In: Proc. of 30th IEEE Int’l Workshop on Machine Learning for Signal Processing (MLSP). Espoo: IEEE, 2020. 1–6. [doi: 10.1109/MLSP49062.2020.9231732]
    [109] Lu RK, Liu JW, Wang YF, Xie HJ, Zuo X. Auto-encoder based co-training multi-view representation learning. In: Proc. of the 23rd Pacific-Asia Conf. on Advances in Knowledge Discovery and Data Mining. Macau: Springer, 2019. 119–130. [doi: 10.1007/978-3-030-16142-2_10].
    [110] Vlachostergiou A, Caridakis G, Mylonas P, Stafylopatis A. Learning representations of natural language texts with generative adversarial networks at document, sentence, and aspect level. Algorithms, 2018, 11(10): 164. [doi: 10.3390/a11100164]
    [111] Sutskever I, Vinyals O, Le QV. Sequence to sequence learning with neural networks. In: Proc. of the 27th Int’l Conf. on Neural Information Processing Systems. Bangkok: MIT Press, 2014. 3104–3112.
    [112] Zhu X, Hu JT, Song LC, Suo GL, Zhan Y. Attention-based encoder-decoder model for photovoltaic power generation prediction. Journal of Physics: Conference Series, 2020, 1575: 012025. [doi: 10.1088/1742-6596/1575/1/012025]
    [113] Nie YP, Han Y, Huang JM, Jiao B, Li AP. Attention-based encoder-decoder model for answer selection in question answering. Frontiers of Information Technology & Electronic Engineering, 2017, 18(4): 535–544. [doi: 10.1631/FITEE.1601232]
    [114] Tian T, Fang Z. Attention-based autoencoder topic model for short texts. Procedia Computer Science, 2019, 151: 1134–1139. [doi: 10.1016/j.procs.2019.04.161]
    [115] Liang ZQ, Pan D, Xu RJ. Knowledge representation framework of accounting event in corpus-based financial report text. Cluster Computing, 2019, 22(4): 9335–9346. [doi: 10.1007/s10586-018-2153-8]
    [116] 王先传, 刘宗田. 新闻文本中事件语义表示. 上海大学学报(自然科学版), 2019, 25(5): 733–741. [doi: 10.12066/j.issn.1007-2861.1989]
    Wang XC, Liu ZT. Event semantic representation for news texts. Journal of Shanghai University (Natural Science), 2019, 25(5): 733–741 (in Chinese with English abstract). [doi: 10.12066/j.issn.1007-2861.1989]
    [117] Giallonardo E, Poggi F, Rossi D, Zimeo E. Semantics-driven programming of self-adaptive reactive systems. International Journal of Software Engineering and Knowledge Engineering, 2020, 30(6): 805–834. [doi: 10.1142/S0218194020400082]
    [118] Wang XL, Feng A, Golshan B, Halevy A, Mihaila G, Oiwa H, Tan WC. Scalable semantic querying of text. Proceedings of the VLDB Endowment, 2018, 11(9): 961–974. [doi: 10.14778/3213880.3213887]
    [119] Liu FL, Liu YX, Ren XC, He XD, Sun X. Aligning visual regions and textual concepts for semantic-grounded image representations. In: Proc. of the 33rd Conf. on Neural Information Processing Systems (NeurIPS). Vancouver, 2019. 6847–6857.
    [120] Liu J, Yang YH, He HH. Multi-level semantic representation enhancement network for relationship extraction. Neurocomputing, 2020, 403: 282–293. [doi: 10.1016/j.neucom.2020.04.056]
    [121] Franco-Salvador M. A cross-domain and cross-language knowledge-based representation of text and its meaning. Procesamiento del Lenguaje Natural, 2019, (62): 111–114.
    [122] Tang X, Chen L, Cui J, Wei BG. Knowledge representation learning with entity descriptions, hierarchical types, and textual relations. Information Processing & Management, 2019, 56(3): 809–822. [doi: 10.1016/j.ipm.2019.01.005]
    [123] Liu CF, Zhang Y, Yu M, Li XW, Zhao MK, Xu TY, Yu J, Yu RG. Text-enhanced knowledge representation learning based on gated convolutional networks. In: Proc. of the 31st Int’l Conf. on Tools with Artificial Intelligence (ICTAI). Portland: IEEE, 2019. 308–315. [doi: 10.1109/ICTAI.2019.00051]
    [124] Piad-Morffis A, Muñoz R, Almeida-Cruz Y, Gutiérrez Y, Estevez-Velarde D, Montoyo A. A neural network component for knowledge-based semantic representations of text. In: Proc. of the Recent Advances in Natural Language Processing. Varna, 2019. 904–911.
    [125] Li HR, Zhu JN, Ma C, Zhang JJ, Zong CQ. Read, watch, listen, and summarize: Multi-modal summarization for asynchronous text, image, audio and video. IEEE Transactions on Knowledge and Data Engineering, 2019, 31(5): 996–1009. [doi: 10.1109/TKDE.2018.2848260]
    [126] Li S, Tao ZQ, Li K, Fu Y. Visual to text: Survey of image and video captioning. IEEE Transactions on Emerging Topics in Computational Intelligence, 2019, 3(4): 297–312. [doi: 10.1109/TETCI.2019.2892755].
    [127] Ibrahim ZAA, Saab M, Sbeity I. VideoToVecs: A new video representation based on deep learning techniques for video classification and clustering. SN Applied Sciences, 2019, 1(6): 560. [doi: 10.1007/s42452-019-0573-6]
    [128] Yu LY, Yu J, Ling Q. Deep neural network based 3D articulatory movement prediction using both text and audio inputs. In: Proc. of the 2019 Int’l Conf. on Multimedia Modeling. Thessaloniki: Springer, 2019. 68–79. [doi: 10.1007/978-3-030-05710-7_6]
    [129] Han ZZ, Shang MY, Wang XY, Liu YS, Zwicker M. Y2Seq2Seq: Cross-modal representation learning for 3D shape and text by joint reconstruction and prediction of view and word sequences. Proc. of the AAAI Conf. on Artificial Intelligence, 2018, 33(1): 126–133. [doi: 10.1609/aaai.v33i01.3301126]
    [130] Suwela N. Ranking index berita new normal dengan metode information retrieval menggunakan vector space model. STRING (Satuan Tulisan Riset dan Inovasi Teknologi), 2020, 5(1): 61–69. [doi: 10.30998/string.v5i1.6479]
    [131] Zhang T, Shen S, Cheng CX, Su K, Zhang XX. A topic model based framework for identifying the distribution of demand for relief supplies using social media data. International Journal of Geographical Information Science, 2021, (10): 1–22. [doi: 10.1080/13658816.2020.1869746](查阅所有网上资料, 未找到本条文献卷期页码信息, 请联系作者确认并补充)
    [132] Shah SMA, Ge HW, Haider SA, Irshad M, Noman SM, Arshad J, Ahmad A, Younas T. A quantum spatial graph convolutional network for text classification. Computer Systems Science and Engineering, 2021, 36(2): 369–382. [doi: 10.32604/csse.2021.014234]
    [133] Jiang Z, Gao S, Chen L. Study on text representation method based on deep learning and topic information. Journal of Computing, 2020, 102(3): 623–642. [doi：10.1007/s00607-019-00755-y]
    [134] 潘俊, 吴宗大. 词汇表示学习研究进展. 情报学报, 2019, 38(11): 1222–1240. [doi: 10.3772/j.issn.1000-0135.2019.11.010]
    Pan J, Wu ZD. A review of word representation learning. Journal of the China Society for Scientific and Technical Information, 2019, 38(11): 1222–1240 (in Chinese with English abstract). [doi: 10.3772/j.issn.1000-0135.2019.11.010]
    [135] Sun YM, Lin L, Tang DY, Yang N, Ji ZZ, Wang XL. Modeling mention, context and entity with neural networks for entity disambiguation. In: Proc. of the 24th Int’l Conf. on Artificial Intelligence (IJCAI). Buenos Aires: AAAI Press, 2015. 1333–1339.
    [136] Rezaeinia SM, Rahmani R, Ghodsi A, Veisi H. Sentiment analysis based on improved pre-trained word embeddings. Expert Systems with Applications, 2019, 117: 139–147. [doi: 10.1016/j.eswa.2018.08.044]
    [137] Wang Y, Sun YN, Ma ZC, Gao LS, Xu Y. Named entity recognition in Chinese medical literature using pretraining models. Scientific Programming, 2020, 2020: 8812754. [doi: 10.1155/2020/8812754]
    [138] Bhattacharjee A, Hasan T, Samin K, Rahman MS, Iqbal A, Shahriyar R. BanglaBERT: Combating embedding barrier for low-resource language understanding. arXiv: 2101.00204, 2021.
    Cited by
Get Citation

赵京胜,宋梦雪,高祥,朱巧明.自然语言处理中的文本表示研究.软件学报,2022,33(1):102-128

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:December 01,2020
  • Revised:January 10,2021
  • Online: April 21,2021
  • Published: January 06,2022
Links:Institute of Software, CASChina Computer FederationCASNSFC
You are the first2032315Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address：4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code：100190
Phone：010-62562563 Fax：010-62562533 Email：jos@iscas.ac.cn
Technical Support：Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063