微信服务号

微信订阅号

2025年7月30日 3:10 星期三
首页 > 过刊浏览>2024年第35卷第2期 >899-926. DOI:10.13328/j.cnki.jos.006952
PDF HTML阅读 XML下载 导出引用 引用提醒
云原生数据库综述
作者:
基金项目:

国家自然科学基金(62232009,61925205,62072261)


Survey on Cloud-native Databases
Author:
  • 摘要
    • | |
  • 访问统计
    • |
  • 参考文献 [65]
    • |
  • 相似文献 [20]
    • | | |
  • 文章评论
    摘要:

    云基础设施的虚拟化、高可用、可弹性调度等特点, 为云数据库提供了开箱即用、可靠可用、按需计费等优势. 云数据库按照架构可以划分为云托管数据库(cloud-hosted database)以及云原生数据库(cloud-native database). 云托管数据库将数据库系统直接部署到云上虚拟机环境中, 具备低成本、易运维、高可靠的优势. 在此基础上, 云原生数据库充分利用云基础设施弹性伸缩的特点, 采用计算存储分离的架构, 实现了计算资源和存储资源的独立伸缩, 进一步提升数据库性价比. 然而计算存储分离的架构为数据库系统设计带来了新的挑战. 深入分析云原生数据库系统的架构和技术. 首先将云原生OLTP和云原生OLAP的数据库架构按照资源分离模式的差异分别进行归类分析, 并对比各类架构的优势与局限. 其次, 基于计算存储分离的架构, 按照各个功能模块深入探讨云原生数据库的关键技术: 主要包括云原生OLTP关键技术(数据组织、副本一致性、主备同步、故障恢复以及混合负载处理)和云原生OLAP关键技术(存储管理、查询处理、无服务器感知计算、数据保护以及机器学习优化). 最后, 总结现有云原生数据库的技术挑战并展望未来研究方向.

    Abstract:

    The virtualization, high availability, high scheduling elasticity, and other characteristics of cloud infrastructure provide cloud databases with many advantages, such as the out-of-the-box feature, high reliability and availability, and pay-as-you-go model. Cloud databases can be divided into two categories according to the architecture design: cloud-hosted databases and cloud-native databases. Cloud-hosted databases, deploying the database system in the virtual machine environment on the cloud, offer the advantages of low cost, easy operation and maintenance, and high reliability. Besides, cloud-native databases take full advantage of the characteristic elastic scaling of the cloud infrastructure. The disaggregated compute and storage architecture is adopted to achieve the independent scaling of computing and storage resources and further increase the cost-performance ratio of the databases. However, the disaggregated compute and storage architecture poses new challenges to the design of database systems. This survey is an in-depth analysis of the architecture and technology of the cloud-native database system. Specifically, the architectures of cloud-native online transaction processing (OLTP) and online analytical processing (OLAP) databases are classified and analyzed, respectively, according to the difference in the resource disaggregation mode, and the advantages and limitations of each architecture are compared. Then, on the basis of the disaggregated compute and storage architectures, this study explores the key technologies of cloud-native databases in depth by functional modules. The technologies under discussion include those of cloud-native OLTP (data organization, replica consistency, main/standby synchronization, failure recovery, and mixed workload processing) and those of cloud-native OLAP (storage management, query processing, serverless-aware compute, data protection, and machine learning optimization). At last, the study summarizes the technical challenges for existing cloud-native databases and suggests the directions for future research.

    参考文献
    [1] Global database management system (DBMS) market outlook. 2023. https://www.expertmarketresearch.com/reports/database-management-system-market
    [2] Database-as-a-service (DBaaS): Global strategic business report. 2023. https://www.researchandmarkets.com/reports/4804281/database-as-a-service-dbaas-global-strategic
    [3] 2023-2028年中国数据库行业市场深度研究及投资前景展望报告. 2022. https://m.huaon.com/detail/840559.html#reportmethod
    China database industry market depth research and investment outlook report on 2023-2028. 2022 (in Chinese). https://m.huaon.com/detail/840559.html#reportmethod
    [4] Li GL, Dong HW, Zhang C. Cloud databases: New techniques, challenges, and opportunities. Proceedings of the VLDB Endowment, 2022, 15(12): 3758-3761. [doi: 10.14778/3554821.3554893]
    [5] Mathew S, Varia J. Overview of amazon web services. Amazon Whitepapers, 2014, 105: 1-22. (查阅网上资料, 请确认本条文献信息)
    [6] Mazumdar P, Agarwal S, Banerjee A, Mazumdar P, Agarwal S, Banerjee A. Azure SQL database. Pro SQL Server on Microsoft Azure. 2016. 129-156.
    [7] Krishnan SP, Gonzalez JL, Krishnan SP, Gonzalez JL. Google Cloud SQL. Building Your Next Big Thing with Google Cloud Platform: A Guide for Developers and Enterprise Architects. New York: Apress, 2015.
    [8] Verbitski A, Gupta A, Saha D, Brahmadesam M, Gupta K, Mittal R, Krishnamurthy S, Maurice S, Kharatishvili T, Bao XF. Amazon aurora: Design considerations for high throughput cloud-native relational databases. In: Proc. of the 2017 ACM Int’l Conf. on Management of Data. Chicago: ACM, 2017. 1041-1052.
    [9] Dageville B, Cruanes T, Zukowski M, Antonov V, Avanes A, Bock J, Claybaugh J, Engovatov D, Hentschel M, Huang JS, Lee AW, Motivala A, Munir AQ, Pelley S, Povinec P, Rahn G, Triantafyllis S, Unterbrunner P. The snowflake elastic data warehouse. In: Proc. of the 2016 Int’l Conf. on Management of Data. San Francisco: ACM, 2016. 215-226.
    [10] Bisong E. Google BigQuery. In: Building Machine Learning and Deep Learning Models on Google Cloud Platform: A Comprehensive Guide for Beginners. Berkeley: Apress, 2019. 485-517.
    [11] Cecchet E, Singh R, Sharma U, Shenoy P. Dolly: Virtualization-driven database provisioning for the cloud. ACM SIGPLAN Notices, 2011, 46(7): 51-62. [doi: 10.1145/2007477.1952691]
    [12] Pritchett D. BASE: An Acid Alternative: In partitioned databases, trading some consistency for availability can lead to dramatic improvements in scalability. Queue, 2008, 6(3): 48-55. [doi: 10.1145/1394127.1394128]
    [13] Soror AA, Minhas UF, Aboulnaga A, Salem K, Kokosielis P, Kamath S. Automatic virtual machine configuration for database workloads. ACM Transactions on Database Systems, 2008, 35(1): 7. [doi: 10.1145/1670243.1670250]
    [14] Antonopoulos P, Budovski A, Diaconu C, Hernandez Saenz A, Hu J, Kodavalla H, Kossmann D, Lingam S, Minhas UF, Prakash N, Purohit V, Qu H, Ravella CS, Reisteter K, Shrotri S, Tang DX, Wakade V. Socrates: The new SQL server in the cloud. In: In: Proc. of the 2019 Int’l Conf. on Management of Data. Amsterdam: ACM, 2019. 1743-1756.
    [15] Cao W, Zhang YQ, Yang XJ, Li FF, Wang S, Hu QD, Cheng XT, Chen ZZ, Liu ZJ, Fang J, Wang B, Wang YH, Sun HQ, Yang Z, Cheng ZS, Chen S, Wu J, Hu W, Zhao JW, Gao YS, Cai SL, Zhang YY, Tong JW. PolarDB serverless: A cloud native database for disaggregated data centers. In: Proc. of the 2021 Int’l Conf. on Management of Data. ACM, 2021. 2477-2489.
    [16] Gupta A, Agarwal D, Tan D, Kulesza J, Pathak R, Stefani S, Srinivasan V. Amazon redshift and the case for simpler data warehouses. In: Proc. of the 2015 ACM SIGMOD Int’l Conf. on Management of Data. Melbourne: ACM, 2015. 1917-1923.
    [17] Pandis I. The evolution of Amazon redshift. Proceedings of the VLDB Endowment, 2021, 14(12): 3162-3174. [doi: 10.14778/3476311.3476391]
    [18] Levandoski J, Lomet D, Sengupta S. LLAMA: A cache/storage subsystem for modern hardware. Proceedings of the VLDB Endowment, 2013, 6(10): 877-888. [doi: 10.14778/2536206.2536215]
    [19] Levandoski JJ, Lomet DB, Sengupta S. The Bw-Tree: A B-tree for new hardware platforms. In: Proc. of the 29th IEEE Int’l Conf. on Data Engineering (ICDE). Brisbane: IEEE, 2013. 302-313.
    [20] Levandoski J, Lomet D, Sengupta S, Stutsman R, Wang R. High performance transactions in deuteronomy. In: Proc. of the 2015 Conf. on Innovative Data Systems Research (CIDR). 2015.
    [21] Gutmans A. Cloud blog databases. Introducing AlloyDB for PostgreSQL: Free yourself from expensive, legacy databases. 2022. https://cloud.google.com/blog/products/databases/introducing-alloydb-for-postgresql
    [22] Azure Managed Disks pricing. 2022. https://azure.microsoft.com/en-us/pricing/details/managed-disks/
    [23] Depoutovitch A, Chen C, Chen J, Larson P, Lin S, Ng J, Cui WL, Liu Q, Huang W, Xiao Y, He YJ. Taurus database: How to be fast, available, and frugal in the cloud. In: Proc. of the 2020 ACM SIGMOD Int’l Conf. on Management of Data. Portland: ACM, 2020. 1463-1478.
    [24] Verbitski A, Gupta A, Saha D, Corey J, Gupta K, Brahmadesam M, Mittal R, Krishnamurthy S, Maurice S, Kharatishvilli T, Bao XF. Amazon aurora: On avoiding distributed consensus for I/Os, commits, and membership changes. In: Proc. of the 2018 Int’l Conf. on Manage-ment of Data. Houston: ACM, 2018. 789-796.
    [25] Cao W, Liu ZJ, Wang P, Chen S, Zhu CF, Zheng S, Wang YH, Ma GQ. PolarFS: An ultra-low latency and failure resilient distributed file system for shared storage cloud database. Proceedings of the VLDB Endowment, 2018, 11(12): 1849-1862. [doi: 10.14778/3229863.3229872]
    [26] Zhang YQ, Ruan CY, Li C, Yang XJ, Cao W, Li FF, Wang B, Fang J, Wang YH, Huo JZ, Bi C. Towards cost-effective and elastic cloud database deployment via memory disaggregation. Proceedings of the VLDB Endowment, 2021, 14(10): 1900-1912. [doi: 10.14778/3467861.3467877]
    [27] Sheshadri Ranganath, Ravi Murthy. Cloud Blog Databases. AlloyDB for PostgreSQL under the hood: Columnar engine. 2022. https://cloud.google.com/blog/products/databases/alloydb-for-postgresql-columnar-engine
    [28] Huang DX, Liu Q, Cui Q, Fang ZH, Ma XY, Xu F, Shen L, Tang L, Zhou YX, Huang ML, Wei W, Liu C, Zhang J, Li JJ, Wu XL, Song LY, Sun RX, Yu SP, Zhao L, Cameron N, Pei LQ, Tang X. TiDB: A raft-based HTAP database. Proceedings of the VLDB Endowment, 2020, 13(12): 3072-3084. [doi: 10.14778/3415478.3415535]
    [29] Prout A, Wang SP, Victor J, Sun Z, Li YZ, Chen J, Bergeron E, Hanson E, Walzer R, Gomes R, Shamgunov N. Cloud-native transactions and analytics in SingleStore. In: Proc. of the 2022 Int’l Conf. on Management of Data. Philadelphia: ACM, 2022. 2340-2352.
    [30] Skeen D. A Quorum-based Commit Protocol. Ithaca: Cornell University, 1982.
    [31] Ongaro D, Ousterhout J. In search of an understandable consensus algorithm. In: Proc. of the 2014 USENIX Annual Technical Conf. Philadelphia: USENIX Association, 2014. 305-320.
    [32] Lamport L. Paxos made simple. ACM SIGACT News (Distributed Computing Column), 2001, 12: 51-58.
    [33] Lamport L. Fast paxos. Distributed Computing, 2006, 19(2): 79-103. [doi: 10.1007/s00446-006-0005-x]
    [34] Chandra TD, Griesemer R, Redstone J. Paxos made live: An engineering perspective. In: Proc. of the 26th Annual ACM Symp. on Principles of Distributed Computing. Portland: ACM, 2007. 398-407.
    [35] Chen J, Jindel S, Walzer R, Sen R, Jimsheleishvilli N, Andrews M. The MemSQL query optimizer: A modern optimizer for real-time analytics in a distributed database. Proceedings of the VLDB Endowment, 2016, 9(13): 1401-1412. [doi: 10.14778/3007263.3007277]
    [36] Mohan C, Haderle D, Lindsay B, Pirahesh H, Schwarz P. ARIES: A transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM Transactions on Database Systems, 1992, 17(1): 94-162. [doi: 10.1145/128765.128770]
    [37] Gu JC, Lee Y, Zhang YW, Chowdhury M, Shin KG. Efficient memory disaggregation with INFINISWAP. In: Proc. of the 14th USENIX Conf. on Networked Systems Design and Implementation. Boston: USENIX Association, 2017. 649-667.
    [38] Shan YZ, Huang YT, Chen YL, Zhang YY. LegoOS: A disseminated, distributed OS for hardware resource disaggregation. In: Proc. of the 13th USENIX Conf. on Operating Systems Design and Implementation. Carlsbad: USENIX Association, 2018. 69-87.
    [39] Vuppalapati M, Miron J, Agarwal R, Truong D, Motivala A, Cruanes T. Building an elastic query engine on disaggregated storage. In: Proc. of the 17th USENIX Conf. on Networked Systems Design and Implementation. Santa Clara: USENIX Association, 2020. 449-462.
    [40] Armenatzoglou N, Basu S, Bhanoori N, Cai MC, Chainani N, Chinta K, Govindaraju V, Green TJ, Gupta M, Hillig S, Hotinger E, Leshinksy Y, Liang JT, McCreedy M, Nagel F, Pandis I, Parchas P, Pathak R, Polychroniou O, Rahman F, Saxena G, Soundararajan G, Subramanian S, Terry D. Amazon redshift re-invented. In: Proc. of the 2022 Int’l Conf. on Management of Data. Philadelphia: ACM, 2022. 2205-2217.
    [41] Melnik S, Gubarev A, Long JJ, Romer G, Shivakumar S, Tolton M, Vassilakis T. Dremel: Interactive analysis of Web-scale datasets. Proc. of the VLDB Endowment, 2010, 3(1-2): 330-339.
    [42] Melnik S, Gubarev A, Long JJ, Romer G, Shivakumar S, Tolton M, Vassilakis T, Ahmadi H, Delorey D, Min S, Pasumansky M, Shute J. Dremel: A decade of interactive SQL analysis at web scale. Proceedings of the VLDB Endowment, 2020, 13(12): 3461-3472. [doi: 10.14778/3415478.3415568]
    [43] Yu XY, Youill M, Woicik M, Ghanem A, Serafini M, Aboulnaga A, Stonebraker M. PushdownDB: Accelerating a DBMS using S3 computation. In: Proc. of the 36th IEEE Int’l Conf. on Data Engineering (ICDE). Dallas: IEEE, 2020. 1802-1805.
    [44] Yang YF, Youill M, Woicik M, Liu YZ, Yu XY, Serafini M, Aboulnaga A, Stonebraker M. FlexPushdownDB: Hybrid pushdown and caching in a cloud DBMS. Proceedings of the VLDB Endowment, 2021, 14(11): 2101-2113. [doi: 10.14778/3476249.3476265]
    [45] Amazon Athena. 2022. https://aws.amazon.com/cn/athena/
    [46] Perron M, Fernandez RC, DeWitt D, Madden S. Starling: A scalable query engine on cloud functions. In: Proc. of the 2020 ACM SIGMOD Int’l Conf. on Management of Data. Portland: ACM, 2020. 131-141.
    [47] Poppe O, Guo Q, Lang W, Arora P, Oslake M, Xu SZ, Kalhan A. Moneyball: Proactive auto-scaling in Microsoft Azure SQL database serverless. Proceedings of the VLDB Endowment, 2022, 15(6): 1279-1287. [doi: 10.14778/3514061.3514073]
    [48] Das P, Ivkin N, Bansal T, Rouesnel L, Gautier P, Karnin Z, Dirac L, Ramakrishnan L, Perunicic A, Shcherbatyi I, Wu W, Zolic A, Shen HB, Ahmed A, Winkelmolen F, Miladinovic M, Archembeau C, Tang A, Dutt B, Grao P, Venkateswar K. Amazon SageMaker Autopilot: A white box AutoML solution at scale. In: Proc. of the 4th Int’l Workshop on Data Management for End-to-end Machine Learning. Portland: ACM, 2020. 2.
    [49] Parchas P, Naamad Y, Van Bouwel P, Faloutsos C, Petropoulos M. Fast and effective distribution-key recommendation for amazon redshift. Proceedings of the VLDB Endowment, 2020, 13(12): 2411-2423. [doi: 10.14778/3407790.3407834]
    [50] Schleier-Smith J, Sreekanti V, Khandelwal A, Carreira J, Yadwadkar NJ, Popa RA, Gonzalez JE, Stoica I, Patterson DA. What serverless computing is and should become: The next phase of cloud computing. Communications of the ACM, 2021, 64(5): 76-84. [doi: 10.1145/3406011]
    [51] Zhan CQ, Su MM, Wei CX, Peng XQ, Lin L, Wang S, Chen Z, Li FF, Pan Y, Zheng F, Chai CL. AnalyticDB: Real-time OLAP database system at Alibaba cloud. Proceedings of the VLDB Endowment, 2019, 12(12): 2059-2070. [doi: 10.14778/3352063.3352124]
    [52] Sreekanti V, Wu CG, Lin XC, Schleier-Smith J, Gonzalez JE, Hellerstein JM, Tumanov A. Cloudburst: Stateful functions-as-a-service. Proceedings of the VLDB Endowment, 2020, 13(12): 2438-2452. [doi: 10.14778/3407790.3407836]
    [53] Kraft P, Li Q, Kaffes K, Skiadopoulos A, Kumar D, Cho D, Li J, Redmond R, Weckwerth N, Xia B, Bailis P, Cafarella M, Graefe G, Kepner J, Kozyrakis C, Stonebraker M, Suresh L, Yu XY, Zaharia M. Apiary: A DBMS-backed transactional function-as-a-service framework. arXiv:2208.13068, 2022.
    [54] Antonopoulos P, Arasu A, Singh KD, Eguro K, Gupta N, Jain R, Kaushik R, Kodavalla H, Kossmann D, Ogg N, Ramamurthy R, Szymaszek J, Trimmer J, Vaswani K, Venkatesan R, Zwilling M. Azure SQL database always encrypted. In: Proc. of the 2020 ACM SIGMOD Int’l Conf. on Management of Data. Portland: ACM, 2020. 1511-1525.
    [55] 李国良, 周煊赫. 面向AI的数据管理技术综述. 软件学报, 2021, 32(1): 21-40. http://www.jos.org.cn/1000-9825/6121.htm
    Li GL, Zhou XH. Survey of data management techniques for artificial intelligence. Ruan Jian Xue Bao/Journal of Software, 2021, 32(1): 21-40 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6121.htm
    [56] Hilprecht B, Binnig C, Röhm U. Learning a partitioning advisor for cloud databases. In: Proc. of the 2020 ACM SIGMOD Int’l Conf. on Management of Data. Portland: ACM, 2020. 143-157.
    [57] Wu WT, Wang C, Siddiqui T, Wang JX, Narasayya V, Chaudhuri S, Bernstein PA. Budget-aware index tuning with reinforcement learning. In: Proc. of the 2022 Int’l Conf. on Management of Data. Philadelphia: ACM, 2022. 1528-1541.
    [58] Jonas E, Schleier-Smith J, Sreekanti V, Tsai CC, Khandelwal A, Pu QF, Shankar V, Carreira J, Krauth K, Yadwadkar N, Gonzalez JE, Ada Popa R, Stoica I, Patterson DA. Cloud programming simplified: A Berkeley view on serverless computing. arXiv:1902.03383, 2019.
    [59] 张超, 李国良, 冯建华, 张金涛. HTAP数据库关键技术综述. 软件学报, 2023, 34(2): 761-785. http://www.jos.org.cn/1000-9825/6713.htm
    Zhang C, Li GL, Feng JH, Zhang JT. A survey of key techniques of HTAP databases. Ruan Jian Xue Bao/Journal of Software, 2023, 34(2): 761-785 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6713.htm
    [60] Li GL, Zhang C. HTAP databases: What is new and what is next. In: Proc. of the 2022 Int’l Conf. on Management of Data. Philadelphia: ACM, 2022. 2483-2488.
    [61] Narasayya V, Chaudhuri S. Multi-tenant cloud data services: State-of-the-art, challenges and opportunities. In: Proc. of the 2022 Int’l Conf. on Management of Data. 2022. 2465-2473.
    [62] Stoica I, Shenker S. From cloud computing to sky computing. In: Proc. of the 2021 Workshop on Hot Topics in Operating Systems. Ann Arbor: ACM, 2021. 26-32.
    引证文献
    网友评论
    网友评论
    分享到微博
    发 布
引用本文

董昊文,张超,李国良,冯建华.云原生数据库综述.软件学报,2024,35(2):899-926

复制
分享
文章指标
  • 点击次数:1959
  • 下载次数: 4874
  • HTML阅读次数: 2834
  • 引用次数: 0
历史
  • 收稿日期:2022-10-25
  • 最后修改日期:2022-12-11
  • 在线发布日期: 2023-08-23
  • 出版日期: 2024-02-06
文章二维码
友情链接:中国科学院软件研究所中国计算机学会中国科学院国家自然科学基金委员会CCF数字图书馆Int'l J of Softw Info计算机系统应用
您是第20376099位访问者
版权所有：中国科学院软件研究所 京ICP备05046678号-3
地址：北京市海淀区中关村南四街4号,邮政编码：100190
电话：010-62562563 传真：010-62562533 Email：jos@iscas.ac.cn
技术支持：北京勤云科技发展有限公司

京公网安备 11040202500063号