Specular detection and removal are always hot problems in computer vision. Advanced results have a great impact on computer vision algorithms. In this paper, a specular detection and removal algorithm is proposed. First, through the comparison between highlight and diffuse chromaticity, a user interactive detection method for a single colored surface is developed. Second, a removal algorithm is proposed by introducing an image inpainting algorithm to this field and adding an illumination constraint. Different from traditional inpainting algorithms, the method described in this paper utilizes more cues which are embedded in the specular region. By integrating the embedded information such as the observed pixel color and illumination chromaticity, this method can overcome the shortcomings of the traditional inpainting methods, which can not keep shading variance in the highlight region. Experimental results show that this method can give a better illumination chromaticity estimation and more convincing results.