It’s a bit more complicated than that - it was given as part of the treaty of surrender, and one could certainly argue that there was duress involved.The whole thing is amazing, but this point just makes me think of some sort of parallel to the current dispute where 2020 Punjab has a group calling for the return of the Koh-i-Noor and getting a 'but you literally sold it to us' response.
But speaking of that treaty, there’s something else to do with that. The Maharaja of the Sikh Empire, Duleep Singh, was made a ward of the British state when he was a young kid as part of that treaty. He was separated from his mother, and he was Anglicized and converted to Christianity. When he grew up, he converted back to Sikhism and he attempted to get the British government to give him further recognition, threatening a Second Indian Mutiny if it did not. But this was refused, and the British government refused to even allow him to return to his home. After his death, he was buried in Britain with Christian rites. Returning his body to India and/or giving him Sikh rites ought to be a much more important issue than the fate of any shiny stone. But sadly not.