In this paper, an algorithm to segment foreground object from a video clip is proposed. The reliability model, which calculated from the local color pattern information, is presented. Firstly, pre-process all the frames of the video clip with the watershed algorithm; then, apply graph-cut on the key frames. Secondly, the reliability model is been computed by bi-directional procedure. Through the positive procedure, the reliability is set. Next, a small portion of the reliabilities is corrected through a reverse procedure assisted by a certain optical flow algorithm.At last, segment the video on the authority of the reliability of each frame. When dealing with the video which has similar foreground and background color, the reliability model shows a good segment result. And bi-directional procedure provides a way to improve segmentation of the occluded object in the video.