Rain streaks can severely degrade the quality of captured images and affect outdoor vision. However, due to non-uniform in shape, direction, and density of rain in different images, it is a difficult task to remove rain from a single image. This study proposes a single image de-raining using an ensemble recurrent dual-attention-residual network, called RDARENet. In the network, as contextual information is very important for the process of rain removal, a multi-scale dilated convolution network is firstly adopted to acquire large receptive field. Rain streaks can be regarded as the accumulation of multiple rain streaks layers, the residual of the channel attention and spatial attention mechanisms are used to extract the features of the rain streaks and restore the background layer information. The channel attention can assign different weights to rain streaks layers, and the spatial attention enhances the representation of the area through the relationship between adjacent spatial features. With the deepening of the network, to prevent the loss of low-level information, a cascaded residual network and a long-term memory network are used to transfer low-level feature information to the high-level and remove rain streaks stage by stage. In the output of the network, ensemble learning method is adopted to weight the output of each stage through the gated network, and add to get the clean image. Extensive experiments demonstrate that the effect of removing rain and restoring texture details is greatly improved.