To me, what Wordsworth is doing in this poem is thinking about the nature of art and poetry.In this poem, he hears the girl singing and he thinks that the song is really beautiful.  However, he does not understand her song.  Probably this is because she is not singing in English (she's Scottish).  Even though he does not understand her, he is struck by how beautiful her song is and he says he heard it in his heart long after he couldn't actually hear it anymore.I think this represents his idea that art and poetry are things of pure emotion like that song, and not things for understanding with your brain.
The overall theme of the poem seems to be one that reveres the idea of nature and the individual's place within it. The ability of the speaker, presumably Wordsworth, to be able to hear the song of the solitary reaper and project the meaning of it in his own mind allows for an expansion of moral imagination that could have only been possible with the individual's assertion into the natural setting and the reverence of that setting.  The theme being advanced at the conclusion of the poem is that the "spontaneous overflow of emotions" that Wordsworth says defines poetry can also have application to the natural setting and we have to be mindful of this potential that exists and revere it.