To handle the slanted or curved surfaces in stereo matching, the adjacent segment geometric constraint and the statistical information of disparity are adopted in a proposed algorithm. The segment geometric constraint is incorporated in a new global energy function to acquire the optimal plane of each segment. The statistical information of disparity is adopted to find the reliable disparity pixel and a reliable segment. The constraints between the segments and pixels are used to estimate the disparity of the unreliable pixels. Experiments are performed on the classical images with a larger disparity range, more slanted surfaces, and surface with less texture and show the effectiveness of the proposed algorithm.