The mixed cooperative-competitive multi-agent system consists of target agents that are learning-based and therefore controllable agents and external agents that are not controllable by the system. The target agents cooperate with each other, compete with external agents, deal with the dynamics of the environment and the external agents, and complete the tasks. To deal with the problem of how to train the target agents so that they can learn the optimal policy for completing the tasks, the existing work proposes two kinds of solutions: (1)focusing on the cooperation between target agents, which views the external agents as a part of the environment, and leverages the multi-agent-reinforcement learning to train the target agents; but these approaches cannot handle the uncertainty or dynamics of the external agents’ policies; (2) focusing on the competition between target agents and external agents, which models the competition as two-player games, and uses self-play to train the target agents; these approaches only solve the problem where there is one target agent and one external agent, but it is difficult to extend these approaches to a multi-agent system consisting of multiple target agents and multiple external agents. This paper combines the two kinds of solutions, and proposes a counterfactual regret advantage-based self-play approach. Specifically, first, based on the counterfactual regret minimization and counterfactual multi-agent policy gradient, we design a counterfactual regret advantage policy gradient for improving the accuracy of the policy evaluation. Second, in order to deal with the dynamics of external agents’ policy during the self-play process, we leverage the imitation learning, which takes the external agents’ history trajectors as training data and imitates the external agents’ policy, to explicitly model the external agents’ behaviors. Third, based on the counterfactual regret advantage policy gradient and the imitation of external agents, we design a self-play approach for learning an optimal joint policy for target agents under the uncertainty or dynamics of external agents’ policy. We conduct a set of experiments on the joint electromagnetic countermeasure, including three typical mixed cooperative-competitive tasks. The experimental results demonstrate that compared with other baseline approaches, the proposed approach has at least 78% performance improvement.