Automatic image annotation is a challenging research problem involving lots of tags and various features. Aiming at the problem that the image annotation based on the traditional shallow machine learning algorithm has low efficiency and is difficult to apply to complex classification task, this paper proposes an automatic image annotation algorithm based on stacked auto-encoder (SAE) to improve both efficiency and effectiveness of annotation. In this paper, two types of strategies are proposed to solve the main problem of unbalanced data in image annotation. For the annotation model itself, to improve the annotation effect of low frequency tags, a balanced and stacked auto-encoder (B-SAE) that can enhance training for low frequency tags is proposed. Based on this model, a robust balanced and stacked auto-encoder algorithm (RB-SAE) is proposed to increase the annotation stability through enhanced training by group in sub B-SAE model. This strategy ensures that the model itself has a strong ability to deal with the unbalanced data. For the annotation process, taking the unknown image as the starting point, the local equilibrium dataset of the unknown image is constructed, and the high and low frequency attribute of the image is discriminated to determine the different annotation process. The local semantic propagation algorithm (SP) annotates the low frequency images and the RB-SAE algorithm annotates the high frequency images. The framework of attribute discrimination annotation (ADA) is formed to improve the overall image annotation effect. This strategy ensures that the labeling process has a strong ability to deal with unbalanced data. Experimental results generated from three public data sets show that many indicators in the presented model are all improved comparing with the previous models.