In this paper, a systematic approach is presented to automatically construct the 3D natural scene from video sequences captured by a vibrated camera with only approximately known motion. The proposed approach performs accurate orientation estimation of temporal spatial texture without need of feature extraction, location of depth boundary occluding area recovery. Furthemore, this approach extends and integrates epipolar plan image algorithm and panorama image algorithm. As the result, it not only bypasses the local minimum problem during the iterative procedure in the conventional motion based lager segmentation method,but also has the advantage of high computation and storage efficiency,adaptability,and robustness.The results could be used for landmark selection of robot navigation and image synthesis of virtualized reality.