The rare earth (RE) doped phosphate glasses are attractive materials in optics due to their low transition temperature and a high capacity to dissolve rare earth ions compared to silicate glasses. In this work, we are interested in understanding the mechanisms leading to the structural modification of Zn polyphosphate and metaphosphate glasses under electron irradiation in order to try to control the RE doping ions environment linked to the luminescence properties. In particular, we present here some results on the impact of the electron energy (700 keV and 2.5 MeV) on the RE environment modification. The glass compositions deal with mixing different alkaline and alkali-earth ions (Na, Li, K and Mg) and different polymerization degree of phosphate glasses. After 2.5 MeV or 700 keV irradiation, the Raman spectra traduce a significant depolymerization of the polyphosphate vitreous network from 108 Gy dose opposite to metaphosphate glasses that remain more stable. The modification of the network seems not to strongly affect the local environment of Eu3+ as demonstrated by the significant increase of the asymmetry ratio (As) of Eu3+ ions in irradiated metaphosphate glasses compared to polyphosphate glasses. Under 700 keV irradiation, the Eu3+ As variation can be noticed in a larger volume than the penetration depth of the electrons. Moreover, the variation depends on the alkaline ion type showing a role of alkaline migration on the Eu3+ environment modification. The shift of the transition associated with a broadening indicates a modification of the Eu3+ crystal field and larger site distribution of Eu3+ ions in polyphosphate glasses. To better explain the composition influence, the local environment of Eu3+ ions will be studied by MD calculations. Finally, the reduction of Eu3+ into Eu2+ is more efficient under 700 keV irradiation: four to seven times compared to 2.5 MeV.