This paper proposes a novel adaptive model for Web image semantic automatic annotation. First, the model automatically collects training image data by exploring the associated textual data and the social tagging data of Web images, such as the Flickr’s Related Tags. Then, using a newly constrained piecewise penalty weighted regression to combine the adaptive estimation of the weight distribution of associated texts and the prior knowledge constrain together and implement the Web image semantic annotation. The proposed training data auto-generation methods and Web image annotation approaches are tested on a real-world Web image data set and promising results are achieved.