The problem of gathering colors from image of color maps with impure colors is studied in this paper.This is a key problem in automatic recognition of map image. The phenomena of color scattering in map images are analyzed,and an algorithm to gather colors from image of color maps is presented. The algorithm has been implemented in the environment of computer of the type of IBM486 and it has been shown that the algorithm was efficient.