With the development of mobile internet and widespread use of mobile phones, a large amount of data that contains user' time and space attributes has been generated and collected. Investigating the semantic information of the collective data plays an important role in understanding the needs, analyzing preference of the user, even recommending and predicting space and time. Recently, many researchers all over the world have turned their focus on understanding the spatio-temporal semantic data. This paper summarizes the related works regarding the spatio-temporal semantic data. Firstly, according to the tasks, the basic concepts and research frameworks are introduced; then, the works of location semantic understanding, user behavior semantic understanding and event semantic understanding are summarized. Additionally, the application scenarios of recommending and predicting space and time field are described. Finally, the future research directions of spatio-temporal data semantic understanding are discussed.