Traditional cloth and garment simulation is normally based on the mesh model where cloth and garment are generally rendered according to the lighting, texture and color setting, which cannot fully reflect their photorealistic appearance. This paper proposes a novel fiber-based cloth and garment simulation algorithm that facilitates photorealistic appearance. The algorithm first uses the sphere set constraining the particle perturbation to express the contour of fiber, and then simulates the fiber-based yarn reconstruction helped by the dynamics and kinematics principle of yarn twisting process. Next, it combines the organization structure of the yarn warp knitting with the mass spring model to construct the cloth model based on the fiber blocks. Finally, it builds up the hybrid bounding volume hierarchy of the human body model to speed up the collision detection between the body model and the garment composed of the fiber based cloth, which helps achieving satisfactory dynamic simulation. Experimental results show that the presented method is valid and effective, as it can reflect the detail in the fiber level of the cloth models and express the geometric texture and the photorealistic appearance of garments.