Action and behavior analysis of players is a direct method of high-level semantic analysis or highlight annotation in sports video. Accurate detection and segmentation of players is the key technology of this method. Employing domain knowledge and characteristics of mid-level feature patch in sports video, a semi-supervised algorithm is proposed to discover the mid-level feature patch and train the player detector for different types of video shots. The detection result is used to label the superpixel, and then player segmentation is accomplished by Grab Cut segmentation algorithm. Experimental results show that the mid-level feature patch based player detector is convenient to train and achieves high detection accuracy. The detected player regions can be used to segment the players effectively, and hence the computation procedure of player segmentation is simplified.