In this paper, a semi-supervised canonical correlation analysis algorithm called Semi-CCA is developed, which uses supervision information in the form of pair-wise constraints in canonical correlation analysis (CCA). In this setting, besides abundant unlabeled data examples, the domain knowledge in the form of pair-wise constraints which specify whether a pair of data examples belongs to the same class (must-link constraints) or not (cannot-link constraints) is also available. Meanwhile, the relative importance of must-link constraints and cannot-link constraints is validated. Experimental results on the artificial dataset, multiple feature database and facial database including Yale and AR show that the proposed Semi-CCA can effectively enhance the classifier performance by using only a small amount of supervision information.