The elasto-optic tensor describes the relationship between strain and index of refraction, and thus is the key glass property describing the stress-optic coefficient and stimulated brillouin scattering. As such, understanding its structural origin is critical in glass design. We showed experimentally that the dispersion of the shear component of the elasto-optic tensor is qualitatively different for different additives, such that barium gives positive dispersion and lead gives negative. We also developed in separate work a bond-based model for the full elasto-optic tensor. Here we build on these studies by developing a bond-based model of dispersion for the elasto-optic tensor, and in this way providing an explanation for the qualitatively different behavior of additives such as barium and lead, based on the nature of the bonds they form with oxygen. These results should lead to chemical design rules for generating glass with desired elasto-optic response, at all wavelengths.