Abstract: Dealing with factors such as overlap, blurs from quickly moving and severe deformation, accurate and stable object tracking has become a critical challenge in compute vision field. First, in this paper, superpixels are used as middle level visual clue to describe the components of object/background with the color histograms of components as their features. The initial appearance model is proposed by clustering the features of a component library. The locality and flexibility of components representations allow the appearance model to describe object/background much more accurately. Then, the Bayesian filter model is used to compute the initial state of target region, and an algorithm is proposed to check and deal with the disturbance introduced by similar objects to avoid drift and obtain more robust tracking result. Finally, to reduce the influences of deformation, overlap and blurs to better preserve the features of object, an online appearance model update algorithm is developed based on the complementary set of the features of components library to enable the appearance model to reflect the real-time variation of object/background by the changes of components. Many experiments on video sequences with different tracking challenges (totally about 12 sequences) show that, compared with the existing object tracking methods, the proposed tracking algorithm results in less error of center position and more successful frame, and therefore can track an object more accurately, stably and effectively.