Bag of visual words model is widely used in image classification and image retrieval. In traditional bag of words model, the statistical method of visual words ignores the spatial information and object shape information, resulting lack of ability to distinguish between image features. In this paper, an improved bag of words method is proposed to combine with salient region extraction and visual words topological structure so that it is can not only produce more representative visual words to certain extent, but also avoid the disturbance of complex background information and position change. First of all, the significant areas of training image are extracted and the bag of visual words model is built on the significant area. Secondly, in order to describe the characteristics of the image more accurately and resist the changing location and the influence of background information, the strategies of visual words topological structure and Delaunay triangulation method are utilized and integrated into the global information and local information. Simulation experiments are performed to compare with the traditional bag of words and other models, the results demonstrate that the proposed method obtained a higher classification accuracy.