According to support vector machines (SVMs), for those geometric approach based classification methods, examples close to the class boundary usually are more informative than others. Taking face detection as an example, this paper addresses the problem of enhancing given training set and presents a nonlinear method to tackle the problem effectively. Based on SVM and improved reduced set algorithm (IRS), the method generates new examples lying close to the face/non-face class boundary to enlarge the original dataset and hence improve its sample distribution. The new IRS algorithm has greatly improved the approximation performance of the original reduced set (RS) method by embedding a new distance metric called image Euclidean distance (IMED) into the kernel function. To verify the generalization capability of the proposed method, the enhanced dataset is used to train an AdaBoost-based face detector and test it on the MIT+CMU frontal face test set. The experimental results show that the original collected database can be enhanced effectively by the proposed method to learn a face detector with improved generalization performance.