Many machine learning methods such as generative model and discriminative model have been applied to image semantic automatic image annotation. However, due to the “semantic gap”, the imbalanced training data, and the multi-label characteristic of image annotation, the annotation performance still calls for improvement. In this paper, an image annotation method is proposed which augments the classical generative model with the proposed discriminative hyperplane tree. Based on the high visual generative probability training images (neighborhood) of the unlabeled image, the local hyperplane classification tree is adaptively established. The semantic relevant training image set is obtained through top-down hierarchical classification procedure by exploiting the discriminative information at each level. The joint probability between the unlabeled image and the semantic words is estimated based on the obtained semantic relevant local training set under the proposed framework. This method combines the advantages of both generative model and the discriminative models. From the aspect of generative model: by exploiting the discriminative information of the semantic cluster in the discriminative hyperplane tree, a local generative set is progressively refined, and therefore, improves accuracy. From the aspect of discriminative model: the multiple label assignment can be naturally implement by estimating the joint probability, which reduces the limitation of discriminative model induced by the imbalanced and overlapping training set. The experiments on the ECCV2002 benchmark show that the method outperforms state-of-the-art generative model-based annotation method MBRM and discriminative model based ASVM-MIL with F1 measure improving by 14% and 13% respectively.