The observation of tumor location and growth is an important link in the formulation of tumor treatment plans. Intervention methods based on medical images can be employed to visually observe the status of the tumor in the patient in a non-invasive way, predict the growth of the tumor, and ultimately help physicians develop a treatment plan specific to the patient. This study proposes a new deep network model, namely the conditional adversarial spatiotemporal encoder model, to predict tumor growth. This model mainly consists of three parts: the tumor prediction generator, the similarity score discriminator, and conditions composed of the patient’s personal situations. The tumor prediction generator predicts the tumor in the next period according to the tumor images of two periods. The similarity score discriminator is used to calculate the similarity between the predicted tumor and the real one. In addition, this study adds the patient’s personal situations as conditions to the tumor growth prediction process. The proposed model is experimentally verified on two collected medical datasets. The experimental results achieve a recall rate of 76.10%, an accuracy rate of 91.70%, and a Dice coefficient of 82.4%, indicating that the proposed model can accurately predict the tumor images of the next period.