Determination of the hierarchical relationship and the objective patterns of sub-classifiers is a primary problem in the construction of a hierarchical classifier. In this paper, a method focusing on the similarities between patterns is proposed to generate a hierarchical structure automatically. Firstly, a similarity measurement utilizing the confusion matrix is advanced to avoid the drawbacks of the traditional measurements, such as high computation costs and invalidity of preliminary conditions. Then abiding by Fisher’s Principle, a Patterns’ Similarity Relationship Analyzing Machine (PSRAM), which is integrated with the supervised and unsupervised pattern recombination methods, is designed to adaptively construct the structure of a hierarchical classifier. Various tests are testified that the proposed method is effective and practical, and it can prominently improve the performance and robustness of the hierarchical classifier.