This paper introduces an effective algorithm for color edge features extraction and proposes a novel edge orientation encoding, biaxial symmetry orientation encoding. The average performance of human face detection system, which is based on the support vector classifier using the histograms of color and color edge features, is evaluated with ROC in multi_fold cross validation. Experimental results show that color edge features outperform gray edge features evidently; the classification accuracy of the novel edge orientation coding outperforms the traditional edge orientation coding when they are linearly combined with color histogram respectively; the face detection accuracy can be significantly improved when both color and color edge histograms are used, non-deep rotated human face can be correctly detected in color image under different illuminations, with different expressions and partial occlusions.