Adverbs are modifiers. It is easy to spot them as modifiers are usually found near the word they modify. In adverbs case, you can find them near an adjective, verb, or another adverb. If you can't find adverbs near any of them, they're probably at the beginning or end of the sentence.
In the sentence you gave:
1. I visited the Muscat Festival when my grandparents had come over.
when my grandparents had come over= adverb phrase/ adverb of time
2. You are always late.
always- adverb of time