The microstructure evolution and mechanical properties of Mg-3.4Y-3.6Sm-2.6Zn-0.8Zr alloy after different heat treatment experiments were investigated by differential thermal analysis (DSC), optical microscopy (OM), X-ray diffraction (XRD), scanning electron microscopy (SEM), energy dispersive spectroscopy (EDS) and tensile test, and the optimum heat treatment systems composing of solid-solution treatment at 500°C for 15 hours and aging treatment at 225°C for 40 hours were proposed. After the solution treatment at 500°C for 15 hours, the layer long period stacking ordered (LPSO) structure disappears, whereas (Mg,Zn)3(Y,Sm) at the grain boundary dissolves into a granular state from the reticular phase, and a large number of long strip phases Mg12(Y,Sm)Zn are formed at the same time. After the aging treatment, a large number of dispersed β′ phases precipitate into the α-Mg grains and it is beneficial to improve the yield strength of the experimental alloy. The yield strength (YS), ultimate tensile strength (UTS) and elongation (EL) of the experimental alloy are 170.0 MPa, 260.8 MPa and 14.1%, respectively. The fracture mode changes from intergranular to transgranular fracture after the heat treatment process.