Recognition of complex dynamic gesture is a key issue for visual gesture-based human-computer interaction. In this paper, an HMM-FNN model is proposed for gesture recognition, which combines ability of HMM model for temporal data modeling with that of fuzzy neural network for fuzzy rule modeling and fuzzy inference. Complex dynamic gesture has two important properties: Its motion can be decomposed and usually being defined in a fuzzy way. By HMM-FNN, complex gesture is firstly decomposed into three components: Posture changing, movement in 2D plane and movement in Z-axis direction, each of which is modeled by HMM. The likelihood of each HMM to observation sequence is considered as membership value of FNN, and gesture is classified through fuzzy inference of FNN. In this proposed method, high-dimensional gesture feature is transformed into several low-dimensional features, as a result, computational complexity is reduced. Furthermore, human's experience or prior knowledge can be used to build and optimize model structure. Experimental results show that the proposed method is an effective method for recognition of complex dynamic gesture, and is superior to conventional HMM method.