Travel time prediction is critical for route planning and traffic monitoring. Due to complex relationships among road segments, spatial-temporal dependency, and other factors, it is challenging to perform modeling upon trajectory dataset. Without incorporating external factors into modeling, existing methods may import incorrect information and ignore road segment dependence, which results in poor prediction accuracy. A two-phase travel time prediction framework is proposed to solve the mentioned issues. During the first stage, trajectory data are mapped to a sequence of segments to generate a low-dimensional vector, which avoids introducing incorrect information while preserving the road segment dependence. During the second phase, after integrating road segment encoding and external factors such as weather and date, a travel time prediction model based on deep neural network is designed. The detailed experimental results on a real-world taxi trajectory dataset show that the proposed method is more accurate than existing methods.