In this paper, a facial feature point tracking scheme is proposed by integrating Lucas-Kanade optical flow tracking algorithm and the face alignment statistical model, DAM (direct appearance model), together in a Bayesian framework. The prediction of feature positions from Lucas-Kanade algorithm exploits the inter-frame correlations and accelerates the tracking speed. The texture-shape constraint under DAM improves the localization accuracy and robustness. Experiments show that this method adapts well to the various face movements. It can be used in face recognition or 3D face modeling.