Automatic image annotation has become an important issue, due to the existence of a semantic gap. Based on probabilistic latent semantic analysis (PLSA), this paper presents an approach to annotate and retrieve images by fusing semantic topics. First, in order to precisely model training data, each image is represented as a bag of visual words. Then, a probabilistic model is designed to capture latent semantic topics from visual and textual modalities, respectively. Furthermore, an adaptive asymmetric learning approach is proposed to fuse these semantic topics. For each image document, the topic distribution of each modality is fused by multiplying different weights, which is determined by the entropy of the distribution of visual words. Consequently, the probabilistic model can predict semantic annotations for an unseen image because it associates visual and textual modalities properly. This approach is compared with several other state-of-the-art approaches on a standard Corel dataset. The experimental results show that this approach performs more effectively and accurately.