A simple and efficient method based on semi-supervised learning technique is proposed for behavior modeling and abnormality detection. The method is composed of the following steps: (1) Dynamic time warping (DTW) based spectral clustering method is used to obtain a small set of samples to initialize the hidden Markov models (HMMs) of normal behaviors; (2) The HMMs’ parameters are further trained by the method of iterative learning from a large data set; (3) Maximum a posteriori (MAP) adaptation technique is used to estimate the HMMs’ parameters of abnormal behaviors from those of normal behaviors; (4) The topological structure of HMM is finally constructed to detect abnormal behaviors. The main characteristic of the proposed method is that it can automatically select the number of normal behavior patterns and samples from the training dataset to build normal behavior models and can effectively avoid the running risk of over-fitting when the HMMs of abnormal behaviors are learned from sparse data. Experimental results demonstrate the effectiveness of the proposed method in comparison with other related works in the literature.