利用Kinect相机结合增强现实技术和手势识别方法设计并实现了一个弓弦乐器虚拟演奏系统——以二胡为例.将Kinect获取的现实场景和虚拟乐器融合在一起绘制成增强现实场景.通过Kinect得到的深度数据和贝叶斯肤色模型将用户的左手分割出来,并再次绘制在增强图像上形成新的图像,从而解决虚拟演奏场景中的虚实遮挡问题.利用基于反向动力学和马尔可夫模型的三维虚拟手势拟合方法,对演奏过程中的左手手势进行识别,并结合右手的运动状态完成乐器的虚拟演奏.

Using Kinect camera combined with augmented reality and hand gesture recognition method, this paper designs and implements a virtual playing system for bowstring musical instruments (e.g. erhu). The fusion rendering of real scene captured by Kinect and virtual instrument forms the augmented reality scene. Through the depth data and a Bayesian skin model, the system segments the user's left hand region, then draws the hand image on the enhanced image, thus solving the problem of occlusion between virtual and real worlds in augmented reality system. Based on inverse kinematic methods and Markov model the presented design also provides a 3D virtual gesture fitting method to recognize left hand gestures, then completes the virtual instruments playing combined with right hand movement status.

国家社会科学基金(12AZD120);国家自然科学基金(61332017, 61173124);浙江省自然科学基金(LY14F020037);国家科技支撑计划(2015BAK04B00)