Abstract:The requirement OLAP engine for the updated data from OLTP engine has promoted the development of Hybrid Trans- actional/Analytical Processing (HTAP) database systems. In order to promote the comparison and development of HTAP database systems, it is crucial to define and implement a benchmark for evaluating the new features of HTAP database systems. In this paper, we analyze the key features of HTAP database systems, and review the key technologies of their implementations. Then, we explore the difficulties of designing HTAP database systems and the challenges of building HTAP benchmarks. Based on these difficulties and challenges, we summarize the key design dimensions of HTAP benchmarks, including data generation, workload generation, evaluation metric and architecture supportability. Then, we compare the differences of existing classic HTAP benchmarks according to the design, and thus analyze their advantages and disadvantages. In addition, we demonstrate running performance of the selected popular benchmarks, and expose their benchmarking abilities. Finally, we summarize the requirements for HTAP benchmarks as well as some future research directions, i.e. semantically consistent workload control and metric for evaluating freshness data access.