The popularity of smart electronic devices and the Internet makes the image data explode. In order to effectively manage the complex image resources, this study proposed an image hierarchical classification method based on a weighted semantic neighborhood set and formal concept partial order structure. Firstly, a weighted coefficient on different semantics is adaptively designed by the image semantic correlation scores, and a weighted semantic neighborhood (WSN) is constructed from the training sets. The semantic labels of the images are automatically generated by judging the word frequency distribution of the images in the semantic neighborhood set. Then, the context is built by taking the images as the objects and the semantic labels as the attributes. This study also proposed an efficient hierarchical classification method for complex image dataset based on the partial order structure. The hierarchical classification method can get the explicit structure relation and the subordinate relationship between the image categories, which provides an effective idea for the hierarchical classification management of the complex images of large data. Three datasets Corel5k, EspGame, and Iaprtc12 were labeled by the WSN method. The label result proved the integrity of the image semantics and the accuracy of the main semantics. Further, the Corel5k dataset was performed by the hierarchical classification method. The experimental results showed the significant performance of the hierarchical classification.