Firstly, in this paper, the key features of memory neural networks in the strongly supervised model and the weakly supervised model and introduced. Then the corresponding application scenarios and processing methods, as well as the advantages and disadvantages of the two models are summarized. Next, a brief survey on the development and application of the two models (including the innovation on the model and the innovation in application) is provided, and the key roles of individual innovative models in the natural language processing are summarized. Finally, the complex challenges of memory neural networks in the natural language processing and the future development of memory neural networks are also discussed.