It is the important precondition for browsing, retrieval and indexing videos that effectively and rationally organizing the video structure. In this paper, for hierarchically organizing the video structure, a method for extracting story units is proposed. k-nearest neighbor hyper-graph is used to represent the content relations among shots, and shots are clustered based on hyper-graph model. By analyzing time projection relations among shot clusters, story units are extracted, and represented by the 1D strings. A frame using specific domain knowledge for identifying the type of story unit is also proposed, and is applied to identify the dialogs in the videos. The new algorithm is applied to multiple test videos, and the experiments results are satisfying.