Using Kinect camera combined with augmented reality and hand gesture recognition method, this paper designs and implements a virtual playing system for bowstring musical instruments (e.g. erhu). The fusion rendering of real scene captured by Kinect and virtual instrument forms the augmented reality scene. Through the depth data and a Bayesian skin model, the system segments the user's left hand region, then draws the hand image on the enhanced image, thus solving the problem of occlusion between virtual and real worlds in augmented reality system. Based on inverse kinematic methods and Markov model the presented design also provides a 3D virtual gesture fitting method to recognize left hand gestures, then completes the virtual instruments playing combined with right hand movement status.