Three types of NdFeB magnets of the same composition and different grain sizes were prepared, and then the grain boundary diffusion was carried out using metal Tb under the same technical parameters. The effect of grain size on the grain boundary diffusion process and properties of sintered NdFeB magnets was investigated. The diffusion course was assessed using X-ray diffraction (XRD), field emission scanning electron microscope (FESEM) and electron probe microanalyzer (EPMA). The magnetic properties of the magnet before and after diffusion were investigated. The results show that the grain refinement of the magnet leads to higher Tb utilization efficiency and results in higher coercivity at different temperatures, which can be attributed to the formation of a deeper and more complete core-shell structure, resulting in better magnetic isolation and higher anisotropy of the Nd2Fe14B grains. This work may shed light on developing high coercivity with low heavy rare earth through grain refinement.