Considering the fact that the determination of the executable file maliciousness is hard to achieve, an approach based on the evidence theory is presented in this paper. First, a model for determining the maliciousness is established. Then characters compromising security are extracted to construct the set of program behaviors through decompiling the program. The model is trained using the BP neural network to gain the basic probability assignment functions (BPAF) of each behavior, and the weighted sum method is applied to combine the program behaviors, determining the executable file maliciousness. Experimental results demonstrate the validity of the approach which uses the evidence theory to determine the maliciousness of program.