微信服务号

微信订阅号

2025年8月13日 13:57 星期三
首页 > 过刊浏览>2023年第34卷第2期 >761-785. DOI:10.13328/j.cnki.jos.006713
PDF HTML阅读 XML下载 导出引用 引用提醒
HTAP数据库关键技术综述
作者:
作者简介:

张超(1990－),男,博士,助理研究员,CCF专业会员,主要研究领域为数据库与大数据管理技术;李国良(1980－),男,博士,教授,博士生导师,CCF杰出会员,主要研究领域为数据库,大数据分析和挖掘,群体计算;冯建华(1967－),男,博士,教授,博士生导师,CCF杰出会员,主要研究领域为数据库,数据安全与隐私保护,信息;张金涛(2000－),男,硕士生,主要研究领域为数据库与机器学习的交叉技术

通讯作者:

李国良，liguoliang@tsinghua.edu.cn

基金项目:

国家自然科学基金（61925205，62072261，62232009）


Survey of Key Techniques of HTAP Databases
Author:
  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [64]
    • |
  • 相似文献 [20]
    • | | |
  • 文章评论
    摘要:

    混合事务与分析处理（hybrid transactional analytical processing，HTAP）技术是一种基于一站式架构同时处理事务请求与查询分析请求的技术.HTAP技术不仅消除了从关系型事务数据库到数据仓库的数据抽取、转换和加载过程，还支持实时地分析最新事务数据.然而，为了同时处理OLTP与OLAP，HTAP系统也需要在系统性能与数据分析新鲜度之间做出取舍，这主要是因为高并发、短时延的OLTP与带宽密集型、高时延的OLAP访问模式不同且互相干扰.目前，主流的HTAP数据库主要以行列共存的方式来支持混合事务与分析处理，但是由于该类数据库面向不同的业务场景，所以它们的存储架构与处理技术各有不同.首先，全面调研HTAP数据库，总结它们主要的应用场景与优缺点，并根据存储架构对它们进行分类、总结与对比.现有综述工作侧重于基于行/列单格式存储的HTAP数据库以及基于Spark的松耦合HTAP系统，而这里侧重于行列共存的实时HTAP数据库.特别地，凝炼了主流HTAP数据库关键技术，包括数据组织技术、数据同步技术、查询优化技术、资源调度技术这4个部分.同时总结分析了HTAP数据库构建技术与评测基准.最后，讨论了HTAP技术未来的研究方向与挑战.

    Abstract:

    Hybrid transactional analytical processing (HTAP) relies on a single system to process the mixed workloads of transactions and analytical queries simultaneously. It not only eliminates the extract-transform-load (ETL) process, but also enables real-time data analysis. Nevertheless, in order to process the mixed workloads of OLTP and OLAP, such systems must balance the trade-off between workload isolation and data freshness. This is mainly because of the interference of highly-concurrent short-lived OLTP workloads and bandwidth-intensive, long-running OLAP workloads. Most existing HTAP databases leverage the best of row store and column store to support HTAP. As there are different requirements for different HTAP applications, HTAP databases have disparate storage strategies and processing techniques. This study comprehensively surveys the HTAP databases. The taxonomy of state-of-the-art HTAP databases is introduced according to their storage strategies and architectures. Then, their pros and cons are summarized and compared. Different from previous works that focus on single-model and spark-based loosely-coupled HTAP systems, real-time HTAP databases with a row-column dual store are focused on. Moreover, a deep dive into their key techniques is accomplished regarding data organization, data synchronization, query optimization, and resource scheduling. The existing HTAP benchmarks are also introduced. Finally, the research challenges and open problems are discussed for HTAP.

    参考文献
    [1] Özcan F, Tian YY, Tözün P. Hybrid transactional/analytical processing:A survey. In:Salihoglu S, Zhou WC, Chirkova R, Yang J, Suciu D, eds. Proc. of the Int'l Conf. on Management of Data. ACM, 2017. 1771-1775.
    [2] Hieber D, Grambow G. Hybrid transactional and analytical processing databases:A systematic literature review. In:Popescu M, Agosto A, Giudici P, Sztandera L, eds. Proc. of the Data Analytics. Nice:IARIA XPS, 2020. 90-98.
    [3] Lahiri T, Chavan S, Colgan M, Das D, Ganesh A, Gleeson M, Hase S, Holloway A, Kamp J, Lee TH. Oracle database in-memory:A dual format in-memory database. In:Gehrke J, Lehner W, Shim K, Cha SK, Lohman GM, eds. Proc. of the Int'l Conf. on Data Engineering. Seoul:IEEE, 2015. 1253-1258.
    [4] Larson PÅ, Birka A, Hanson EN, Huang W, Nowakiewicz M, Papadimos V. Real-time analytical processing with SQL server. Proc. of the VLDB Endowment, 2015, 8(12):1740-1751.
    [5] Raman V, Attaluri G, Barber R, Chainani N, Kalmuk D, KulandaiSamy V, Leenstra J, Lightstone S, Liu S, Lohman GM. DB2 with BLU acceleration:So much more than just a column store. Proc. of the VLDB Endowment, 2013, 6(11):1080-1091.
    [6] Sikka V, Färber F, Lehner W, Cha SK, Peh T, Bornhövd C. Efficient transaction processing in SAP HANA database:The end of a column store myth. In:Candan KS, Chen Y, Snodgrass RT, Gravano L, Fuxman A, eds. Proc. of the Int'l Conf. on Management of Data. Scottsdale:ACM, 2012. 731-742.
    [7] Huang DX, Liu Q, Cui Q, et al. TiDB:A raft-based HTAP database. Proc. of the VLDB Endowment, 2020, 13(12):3072-3084.
    [8] Yang J, Rae I, Xu J, et al. F1 lightning:HTAP as a service. Proc. of the VLDB Endowment, 2020, 13(12):3313-3325.
    [9] MySQL Heatwave. Real-time Analytics for MySQL Database Service. Public Documentation Version 3.0. 2021. 1-20.
    [10] Oracle. Database In-Memory Guide. Public Documentation 21c. 2021. 71-88.
    [11] Boissier M, Schlosser R, Uflacker M. Hybrid data layouts for tiered HTAP databases with pareto-optimal data placements. In:Proc. of the Int'l Conf. on Data Engineering. Paris:IEEE, 2018. 209-220.
    [12] Alagiannis I, Idreos S, Ailamaki A. H2O:A hands-free adaptive store. In:Dyreson CE, Li FF, Ozsu MT, eds. Proc. of the Int'l Conf. on Management of Data. Snowbird:ACM, 2014. 1103-1114.
    [13] Athanassoulis M, Bøgh KS, Idreos S. Optimal column layout for hybrid workloads. Proc. of the VLDB Endowment, 2019, 12(13):2393-2407.
    [14] Arulraj J, Pavlo A, Menon P. Bridging the archipelago between row-stores and column-stores for hybrid workloads. In:Ozscan F, Koutrika G, Madden S, eds. Proc. of the Int'l Conf. on Management of Data. San Francisco:ACM, 2016. 583-598.
    [15] Abebe M, Lazu H, Daudjee K. Proteus:Autonomous adaptive storage for mixed workloads. In:Bonifati A, Abbadi AE, Barcelo P, eds. Proc. of the Int'l Conf. on Management of Data. ACM, 2022. 1-15.
    [16] Appuswamy R, Karpathiotakis M, Porobic D, Ailamaki A. The case for heterogeneous HTAP. In:Proc. of the 8th Biennial Conf. on Innovative Data Systems Research. Chaminade:CIDR, 2017. 1-11.
    [17] Lee R, Zhou MH, Li C, Hu SG, Teng JP, Li DY, Zhang XD. The art of balance:A RateupDB experience of building a CPU/GPU hybrid database product. Proc. of the VLDB Endowment, 2021, 14(12):2999-3013.
    [18] Sun YH, Blelloch GE, Lim WS, Pavlo A. On supporting efficient snapshot isolation for hybrid workloads with multi-versioned indexes. Proc. of the VLDB Endowment, 2019, 13(2):211-225.
    [19] Riegger C, Vinçon T, Gottstein R, Petrov I. MV-PBT:Multi-version indexing for large datasets and HTAP workloads. In:Bonifati, Zhou YL, Salles MAV, Bohm A, Olteanu D, eds. Proc. of the Int'l Conf. on Extending Database Technology. Copenhagen:OpenProceedings. org, 2020. 217-228.
    [20] Psaroudakis I, Scheuer T, May N, Ailamaki A. Task scheduling for highly concurrent analytical and transactional main-memory workloads. In:Bordawekar R, Lang CA, Gedik B, eds. Proc. of the Int'l Workshop on Accelerating Data Management Systems Using Modern Processor and Storage Architectures. Trento:VLDB, 2013. 36-45.
    [21] Sirin U, Dwarkadas S, Ailamaki A. Performance characterization of HTAP workloads. In:Proc. of the Int'l Conf. on Data Engineering. Chania:IEEE, 2021. 1829-1834.
    [22] Raza A, Chrysogelos P, Anadiotis AC, Ailamaki A. Adaptive HTAP through elastic resource scheduling. In:Maier D, Pottinger R, Doan AH, Tan WC, Alawini A, Ngo HQ, eds. Proc. of the Int'l Conf. on Management of Data. Portland:ACM, 2020. 2043-2054.
    [23] Kemper A, Neumann T. HyPer:A hybrid OLTP & OLAP main memory database system based on virtual memory snapshots. In:Abiteboul S, Bohm K, Koch C, Tan KL, eds. Proc. of the Int'l Conf. on Data Engineering. Hannover:IEEE, 2011. 195-206.
    [24] Makreshanski D, Giceva J, Barthels C, Alonso G. BatchDB:Efficient isolated execution of hybrid OLTP+OLAP workloads for interactive applications. In:Salihoglu S, Zhou WC, Chirkova R, Yang J, Suciu D, eds. Proc. of the Int'l Conf. on Management of Data. Chicago:ACM, 2017. 37-50.
    [25] Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I. Spark:Cluster computing with working sets. In:Naham EM, Xu DY, eds. Proc. of the 2nd USENIX Workshop on Hot Topics in Cloud Computing. Boston:USENIX, 2010. 1-7.
    [26] Barber R, Garcia-Arellano C, Grosman R, et al. Evolving databases for new-gen big data applications. In:Proc. of the 8th Biennial Conf. on Innovative Data Systems Research. Chaminade:CIDR, 2017. 1-8.
    [27] Mozafari B, Ramnarayan J, Menon S, Mahajan Y, Chakraborty S, Bhanawat H, Bachhav K. SnappyData:A unified cluster for streaming, transactions and interactice analytics. In:Proc. of the 8th Biennial Conf. on Innovative Data Systems Research. Chaminade:CIDR, 2017. 1-8.
    [28] Apache HBase. Reference Guide. Pulic Documentation Version 3.0. 2021. 444-455.
    [29] Bouzeghoub M, Peralta V. A framework for analysis of data freshness. In:Nauman F, Scannapieco M, eds. Proc. of the Int'l Workshop on Information Quality in Information Systems. Paris:ACM, 2004. 59-67.
    [30] Dziedzic A, Wang JJ, Das S, Ding BL, Narasayya VR, Syamala M. Columnstore and B+ tree-Are hybrid physical designs important? In:Das G, Jermaine CM, Bernstein PA, eds. Proc. of the Int'l Conf. on Management of Data. Houston:ACM, 2018. 177-190.
    [31] Diaconu C, Freedman C, Ismert E, Larson PA, Mittal P, Stonecipher R, Verma N, Zwilling M. Hekaton:SQL server's memory-optimized OLTP engine. In:Ross KA, Srivastava D, Papadias D, eds. Proc. of the Int'l Conf. on Management of Data. New York:ACM, 2013. 1243-1254.
    [32] Lamport L. Paxos made simple. ACM SIGACT News (Distributed Computing Column), 2001, 32(4):51-58.
    [33] Diego O, Ousterhout JK. In search of an understandable consensus algorithm. In:Gibson G, Zeldovich N, eds. Proc. of the USENIX Annual Technical Conf. Philadelphia:USENIX, 2014. 305-319.
    [34] Corbett JC, Dean J, Epstein M, et al. Spanner:Google's globally distributed database. ACM Trans. on Computer System, 2013, 31(3):1-22.
    [35] Goel AK, Pound J, Auch N, Bumbulis P, MacLean S. Towards scalable real-time analytics:An architecture for scale-out of OLxP workloads. Proc. of the VLDB Endowment, 2015, 8(12):1716-1727.3.
    [36] Färber F, May N, Lehner W, Große P, Müller I, Rauhe H, Dees J. The SAP HANA database-An architecture overview. IEEE Database Engineering Bulletin, 2012, 35(1):28-33.
    [37] Shen SJ, Chen R, Chen HB, Zang BY. Retrofitting high availability mechanism to tame hybrid transaction/analytical processing. In:Brown AD, Lorch JR, eds. Proc. of the 15th USENIX Symp. on Operating Systems Design and Implementation. Virtual Event:USENIX, 2021. 219-238.
    [38] Neumann T, Mühlbauer T, Kemper A. Fast serializable multi-version concurrency control for main-memory database systems. In:Sellis TK, Davison SB, I, eds. Proc. of the Int'l Conf. on Management of Data. Melbourne:ACM, 2015. 677-689.
    [39] Grund M, Krüger J, Plattner H, Zeier A, Cudre-Mauroux P, Madden S. Hyrise:A main memory hybrid storage engine. Proc. of the VLDB Endowment, 2010, 4(2):105-116.
    [40] Borthakur D. HDFS architecture guide. Hadoop Apache Project, 2008. 1-14.
    [41] Vohra D. Apache Parquet. Practical Hadoop Ecosystem. Berkeley:Springer, 2016. 325-335.
    [42] Pei W, Li ZH, Pan W. Survey of key technologies in GPU database system. Ruan Jian Xue Bao/Journal of Software, 2021, 32(3):859-885 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6175.htm[doi:10.13328/j.cnki.jos.006175] 裴威, 李战怀, 潘巍. GPU数据库核心技术综述. 软件学报, 2021, 32(3):859-885. http://www.jos.org.cn/1000-9825/6175.htm[doi:10.13328/j.cnki.jos.006175].
    [43] Gallet B, Gowanlock M. Heterogeneous CPU-GPU epsilon grid joins:Static and dynamic work partitioning strategies. Data Science and Engineering, 2021, 6(1):39-62.
    [44] Psaroudakis I, Wolf F, May N, Neumann T, Böhm A, Ailamaki A, Sattler KU. Scaling up mixed workloads:A battle of data freshness, flexibility, and scheduling. In:Nambiar R, Posess M, eds. Proc. of the Technology Conf. on Performance Evaluation and Benchmarking. Hangzhou:Sprinder, 2014. 97-112.
    [45] Cole R, Funke F, Giakoumakis L, Guy W, Kemper A, Krompass S, Kuno H, Nambiar R, Neumann T, Poess M. The mixed workload CH-benCHmark. In:Graefe. G, Salem K, eds. Proc. of the 4th Int'l Workshop on Testing Database Systems. Athen:ACM, 2011. 1-6.
    [46] Coelho F, Paulo J, Vilaça R, Pereira J, Oliveira R. HTAPBench:Hybrid transactional and analytical processing benchmark. In:Binder W, Cortellessa V, Koziolek A, Smirni E, Poess M, eds. Proc. of the 8th ACM/SPEC on Int'l Conf. on Performance Engineering. ACM, 2017. 293-304.
    [47] Bog A, Plattner H, Zeier A. A mixed transaction processing and operational reporting benchmark. Information Systems Frontiers, 2011, 13(3):321-335.
    [48] Bog A, Sachs K, Plattner H. Interactive performance monitoring of a composite OLTP and OLAP workload. In:Candan KS, Chen Y, Snodgrass RT, Gravano L, Fuxman A, eds. Proc. of the 2012 ACM SIGMOD Int'l Conf. on Management of Data. Scottsdale:ACM, 2012. 645-648.
    [49] Chai MK, Fan J, Du XY. Learnable database systems:Challenges and opportunities. Ruan Jian Xue Bao/Journal of Software, 2020, 31(3):806-830 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/5908.htm[doi:10.13328/j.cnki.jos.005908] 柴茗珂, 范举, 杜小勇. 学习式数据库系统:挑战与机遇. 软件学报, 2020, 31(3):806-830. http://www.jos.org.cn/1000-9825/5908.htm[doi:10.13328/j.cnki.jos.005908].
    [50] Li GL, Zhou XH. Survey of data management techniques for supporting artificial intelligence. Ruan Jian Xue Bao/Journal of Software, 2021, 32(1):21-40 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6121.htm[doi:10.13328/j.cnki. jos.006121] 李国良, 周煊赫. 面向AI的数据管理技术综述. 软件学报, 2021, 32(1):21-40. http://www.jos.org.cn/1000-9825/6121.htm[doi:10.13328/j.cnki.jos.006121].
    [51] Li GL, Zhou XH, Li SF, Gao B. Qtune:A query-aware database tuning system with deep reinforcement learning. Proc. of the VLDB Endowment, 2019, 12(12):2118-2130.
    [52] Zhang J, Liu Y, Zhou K, Li GL, Xiao ZL, Cheng B, Xing JS, Wang YT, Cheng TH, Liu L, Ran MW, Li ZK. An end-to-end automatic cloud database tuning system using deep reinforcement learning. In:Bonce PA, Manegold S, Ailamaki A, Deshpande A, Kraska T, eds. Proc. of the Int'l Conf. on Management of Data. Amasterdam:ACM, 2019. 415-432.
    [53] Yu X, Li GL, Chai CL, Tang N. Reinforcement learning with tree-LSTM for join order selection. In:Proc. of the Int'l Conf. on Data Engineering. Dallas:IEEE, 2020. 1297-1308.
    [54] Yuan HT, Li GL, Feng L, Sun J, Han Y. Automatic view generation with deep learning and reinforcement learning. In:Proc. of the Int'l Conf. on Data Engineering. Dallas:IEEE, 2020. 1501-1512.
    [55] Boissier M. Robust and budget-constrained encoding configurations for in-memory database systems. Proc. of the VLDB Endowment, 2021, 15(4):780-793.
    [56] Li GL, Zhou XH, Cao L. AI meets database:AI4DB and DB4AI. In:Li GL, Li ZH, Idreos S, Srivastava, eds. Proc. of the Int'l Conf. on Management of Data. Virtual Event:ACM, 2021. 2859-2866.
    [57] Li GL, Zhou XH. Machine learning for data management:A system view. In:Proc. of the Int'l Conf. on Data Engineering. Virtual Event:IEEE, 2022. 1-4.
    [58] Zhou XH, Chai CL, Li GL, Sun J. Database meets artificial intelligence:A survey. IEEE Trans. on Knowledge Data Engineering, 2022, 34(3):1096-1116.
    [59] Li GL, Zhou XH, Sun J, Yu X, Han Y, Jin LY, Li WB, Wang TQ, Li SF. openGauss:An autonomous database system. Proc. of the VLDB Endowment, 2021, 14(12):3028-3041.
    [60] Marcus R, Negi P, Mao HZ, Tatbul N, Alizadeh M, Kraska T. Bao:Making learned query optimization practical. In:Li GL, Li ZH, Idreos S, Srivastava, eds. Proc. of the Int'l Conf. on Management of Data. Virtual Event:ACM, 2021. 1275-1288.
    [61] Marcus R, Negi P, Mao HZ, Zhang C, Alizadeh M, Kraska T, Papaemmanouil O, Tatbul N. Neo:A learned query optimizer. Proc. of the VLDB Endowment, 2019, 12(11):1705-1718.
    [62] Lan H, Bao ZF, Peng YW. A survey on advancing the DBMS query optimizer:Cardinality estimation, cost model, and plan enumeration. Data Science and Engineering, 2021, 6(1):86-101.
    [63] Boncz P, Anatiotis AC, Kläbe S. JCC-H:Adding join crossing correlations with skew to TPC-H. In:Nambiar R, Poess M, eds. Proc. of the Technology Conf. on Performance Evaluation and Benchmarking. Munich:Springer, 2017. 103-119.
    [64] Leis V, Gubichev A, Mirchev A, Boncz P, Kemper A, Neumann T. How good are query optimizers, really? Proc. of the VLDB Endowment, 2015, 9(3):204-215.
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

张超,李国良,冯建华,张金涛. HTAP数据库关键技术综述.软件学报,2023,34(2):761-785

复制
分享
文章指标
  • 点击次数:3346
  • 下载次数: 7935
  • HTML阅读次数: 5296
  • 引用次数: 0
历史
  • 收稿日期:2022-02-18
  • 最后修改日期:2022-05-08
  • 在线发布日期: 2022-07-22
  • 出版日期: 2023-02-06
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会CCF数字图书馆Int'l J of Softw Info计算机系统应用
您是第20541251位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京市海淀区中关村南四街4号,邮政编码：100190
电话：010-62562563 传真：010-62562533 Email：jos@iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号