An approach to object tracking based on vision quantum is proposed in this paper in order to solve the high loss-tracking rate in variable structure object tracking. First, the gray information is detected in an image from top to bottom with vision quantum, and the distribution area and gray levels of larger probability density are counted in the vision quantum. Then all the energy frequencies of the visual quantum are calculated such that the weaker energy frequency gradient is removed by filtration and the stronger frequency gradient of vision quantum that the distribution of high frequency information account for half quantum area is reserved. The quantum cluster is composed of vision quantum with the same frequency variation. Finally, taking quantum cluster as candidate object information, the state of moving object is predicted with maximum likelihood estimation and the forecast results are served as moving reference position of vision quantum in the next frame. Further verification of the visual quantum balance state is made to ensure the effectiveness of object tracking. This method catches the point that the variable structure moving object has the feature of the energy frequency step invariance at the juncture pixels of the foreground and background. It can effectively overcome the changes in shape, scale and other factors that influence the moving object tracking, achieving lower loss-tracking rate and lower computational complexity by using independent and continuous visual quantum to describe the step invariant feature. Experimental results show that the proposed approach has good adaptability to variable structure tracking with real-time and robust tracking performance.