In the age of big data, learning from multi-source data plays an important role in many real applications. To date, plenty of multi-source data learning algorithms have been proposed, however, they pay little attention to the fundamental theoretic laws. Meanwhile, it is hard for the classical machine learning theories to govern all learning systems, and to further provide a theoretical support for multi-source learning algorithms. From the perspective of knowledge acquisition through learning, a survey is given on the research progress of three key problems:the human cognitive mechanism, three classical machine learning theories (such as computational learning theory, statistical learning theory, and probabilistic graphical model), and the design of multi-source learning algorithms. Future theoretical research issues of multi-source data learning also presented and investigated.