Irregular surfaces we see are quite irregular at the size that our eye can detect.
But for the size of rays of light, their wavelength is very small..in terms of 100s of nano meters perhaps..
So the roughness does not matter.. light is incident at one precise point, and it gets reflected according to the laws of reflection.. However, different rays get incident on surfaces which are not parallel to each other... So the reflected rays are all in various directions.. So the scattering of light happens. The surface does not appear so shiny.