Let us draw circles around the Earth at various radii. In each sphere, we have air with different density and refractive index. If light enters from an oblique angle, then it gets refracted. Like in the morning and evening the refraction is high.
At noon, the light rays enter the above spheres at normal to the spherical surface. That is incidence angle = 0. So angle of refraction is 0. Light rays pass thru without deviation and dispersion. All the colors in light travel together. The distance traveled by light rays is the least. So there is less scattering possible.
Since all colors are together without dispersion, and little scattering, Sun appears white.