A method is proposed to recover the reflectance properties of all objects in an environment. The input of the algorithm is a 3D geometric model of the scene, a panorama of it, and the information of light sources. The result is a full model of the reflectance properties of the scene. Recovery is done in a progressive refinement manner. At the very beginning, supposes all surfaces are diffuse and generates a new panorama. Then the new rendered panorama and the original one are compared iteratively. If the differences of some objects are greater than a threshold, more complex reflectance models are chosen for them. Finally, each object in the scene has a proper reflectance model, the scene can be rendered under novel lighting or viewing conditions, old objects can be removed from the scene, and new objects can be augmented into the scene. Special efforts have been done on recovering textures of textured diffuse, isotropic, and anisotropic surfaces; shadows and highlights are eliminated almost clearly.