With the wider application of deep learning in the field of computer vision, face authentication, license plate recognition, and road sign recognition have also presented commercial application trends. Therefore, research on the security of deep learning models is of great importance. Previous studies have found that deep learning models are vulnerable to carefully crafted adversarial examples that contains small perturbations, leading completely incorrect recognition results. Adversarial attacks against deep learning models are fatal, but they can also help researchers find vulnerabilities of models and make further improvements. Motivated by that, this study proposes a black box physical attack method based on particle swarm optimization (BPA-PSO) for deep learning road sign recognition model in scenario of autonomous vehicles. Under the premise of unknown model structure, BPA-PSO can not only realize the black box attack on deep learning models, but also invalidate the road sign recognition models in the physical scenario. The attack effectiveness of BPA-PSO algorithm is verified through a large number of experiments in the digital images of electronic space, laboratory environment, and outdoor road conditions. Besides, the abilities of discovering models' vulnerabilities and further improving the application security of deep learning are also demonstrated. Finally, the problems existing in the BPA-PSO algorithm are analyzed and possible challenges of future research are proposed.