The mechanical properties of the prior-β Zr layer in zirconium alloys in loss of coolant accident (LOCA) are of great significance to nuclear safety. Nb as an important alloying element in zirconium alloys, has an important influence on its mechanical properties. In this paper, the phase transformation behavior of Zr-xNb (x=0, 0.5, 1, 2.5; wt.%) alloy during LOCA quenching and tensile behavior after quenching were investigated by molecular dynamics method, and the Nb"s effect on mechanical properties of Zr-xNb was analyzed. The results show that the simulated phase transformation process of quenching leads to lamellar polycrystals similar to the structure of prior-β Zr , which are mainly composed of fcc and hcp structural atoms. The bcc→fcc phase transformation path follows the Brain phase relationship, while the bcc→hcp phase transformation path follows the P-S phase relationship. During the cooling process, the addition of Nb reduces the difference between the free energy of the bcc and hcp phases, thus making the temperature of β to α + β phase transition lower. In the Zr, Zr-0.5Nb and Zr-1Nb alloy models, Nb promotes the generation of bcc phase at the grain boundaries, which makes the deformation concentrated at the grain boundaries thus leads to the fracture of the grain boundaries. In Zr-2.5Nb, the content of Nb in the grain is also higher for the formation of bcc phase, which makes the deformation homogeneous and improves its plasticity. In addition, the clusters of Zr-2.5Nb alloy models are diffusely distributed, which makes the tensile strength of Zr-2.5Nb improved.