Computer vision has been widely used in various real-world scenarios due to its powerful learning ability. With the development of databases, there is a growing trend in research to exploit mature data management techniques in databases for vision analytics applications. The integration and processing of multimodal data, including images, video and text, promotes diversity and improves accuracy in vision analytics applications. In recent years, due to the popularization of deep learning, there has been a growing interest in vision analytics applications that support deep learning. Nevertheless, traditional database management techniques in deep learning scenarios suffer from the issues such as lack of semantics for vision analytics and inefficiency in application execution. Hence, vision database management systems that support deep learning have been widely studied. This study reviews the progress of vision database management systems. First, this study summarizes the challenges faced by vision database management systems in different dimensions, including programming interface, query optimization, execution scheduling, and data storage. Second, this study discusses the technologies in each of these four dimensions. Finally, the study investigates the future research directions of vision database management systems.