In recent years, due to the powerful learning ability, convolutional neural networks (CNN) have achieved more satisfactory results than conventional learning methods in image restoration tasks. However, these CNN-based methods generally have the disadvantage of producing over-smoothed restored image due to the fact that losing important textural details. In order to solve this problem, this study proposes an image restoration method based on cascaded dense CNN (CDCNN) in contourlet transform, which can be used for three classical image restoration tasks, namely, single image denoising, super resolution, and JPEG decompression. First, this study constructs a compact cascading dense network structure, which not only fully exploits and utilizes the different hierarchical features of images, but also solves the problem of the long-term dependency problem as growing the network depth. Next, this study introduces the contourlet transform into CDCNN, which can sparsely represent the important image features. Here, the contourlet subbands of low-quality image and corresponding restored image are used as the input and output of the network respectively, which can recover realistic structure and texture details more effectively. Comprehensive experiments on the standard benchmarks show that the unanimous superiority of the proposed method on all three tasks over the state-of-the-art methods. The proposed method not only obtains higher peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), but also contains more realistic textural details in the subjective reconstruct images.