Soil–rock mixtures are widely encountered in geotechnical engineering projects. The instability and failure mechanism of grap-graded soil–rock mixtures under rainfall conditions has always been the focus of geological disaster research. To deeply explore the mechanism of seepage deformation of soil–rock mixtures, an indoor physical permeability test that considers soil–rock mixtures with different fine contents was conducted, and a particle-scale numerical simulation test of the permeability evolution was carried out using the coupling model of PFC3D and ABAQUS. The test results showed that the spatial distribution of fine particle loss along the height direction could be divided into three areas: top loss, middle uniform, and bottom loss area. The “island” effect of coarse particles, which is caused by excessive fine content and makes the fine particles bear more load, was eliminated with the loss of fine particles. In this preset working condition of coarse and fine particle diameters, setting FC to 35% may be the best way to fill the voids between the coarse particles. Particle migration leads to a change in the load-bearing skeleton structure, thereby causing seepage deformation. Therefore, the particle-scale numerical test method can better reproduce the seepage deformation process of grap-graded soil–rock mixtures.