There are two main aspects of experimental results which cannot be explained by wave theory. If light is wave then its energy increases as we increase the intensity of the wave, but this increases only the number of electrons emitted but does not increase the energy of the electron. Secondly if electrons absorbs the energy of the wave, the emission of the electron must be delayed with respect to the instance light falls on the substance, but it is immediate.
Lastly, photon theory of light is just not what you state, "And a "photon" is simply a measure of energy of an electromagnetic wave with a certain wavelength or frequency." the model is completely different.
In a a wave model electron has to oscillate as per the oscillating electric field and gain enough energy and then get out.
In a photon model which is not just another particle model but it is a quantum model emphasizing duality at the fundamental level. According to it light consists of particles each with energy "hf". [Note that energy is particle aspect and frequency is wave aspect. This particle meets the electron and itself gets vanished, giving all its energy in one shot process to the electron.