With the popularity of security surveillance systems, more and more surveillance cameras have been installed at various traffic roads and public places. A lot of surveillance videos are produced every day. Currently, surveillance video analysis is being done by manual monitoring, which is very inefficient. Most researches on video analysis focus on abnormal behavior detection and tracking of the target individual and lack of analysis of associations between objects/scenes, and there have no effectively representation and analytical methods for the association between objects and scenes. This paper presents a three-dimensional trajectory of moving objects associated video visual analysis to assist manual analysis of videos. First, video data preprocessed to get trajectory information of target object. Because two-dimensional trajectory is unable to deal with the recurring motion, self-intersecting motion and pause, and also lacks of time information, it is difficult be used to analyze correlation between multi-object trajectories in same space. Therefore, time dimension is added to generate three-dimensional trajectory to unite scenes and objects to calculate correlation between trajectories and build the association model. This approach supports sketch interaction so that users can add sketch annotation to assistant analyzing. By utilizing the absence information of objects in certain scene and some predefined rules, the presented approach can be used to alert the abnormal action and assistant user to make corresponding decisions.