If you gave this money to your brother as a grant and he accepted it, knowing it was a grant, it is not permissible for you to take it back. The Prophet (peace be upon him) said,
"Anyone who takes back a gift is like a dog that eats its own vomit."
Consequently, this sum of money takes the same ruling as any money owned by your brother and you have to return it to his heirs. If you are one of his heirs, you will take only your share according to the laws of inheritance. May Allah grant us success. May peace and blessings be upon our Prophet Muhammad, his family, and Companions.