This paper presents a generic framework for manipulating the images based on adaptive mesh deformation, which allows users to move, scale, rotate, and deform the salient objects in the image and retarget the image into other regions with arbitrary boundary shapes. The image is embedded into a triangular mesh and the manipulation is formulized into a quadratic energy minimization problem. The triangles corresponding to the salient objects are constrained by specific transformations with respect to the deformation types of the objects such as translation, scaling, or rotation etc. The solution to the energy optimization can be obtained by solving one or several sparse linear systems. Experimental results show that the algorithm is effective, robust, and efficient and can be integrated into other image processing tools.