Different colours of light have the same velocity in vacuum, but different in different transparent mediums.
The wavelength of violet coloured light is minimum. So its velocity is minimum and while passing through the prism, it gets reflected the most and thus it forms the base of spectrum.
The wavelength of red coloured light is maximum and so its velocity is maximum. So, while passing through prism, it gets refracted the least and thus it forms the top of spectrum.