If the reality is as you mentioned, your brother has divorced his wife once even though he claims that he was lying or joking. This is because
divorce becomes effective once it is pronounced, whether this is done seriously or jokingly. However, he can take her back in marriage as long as she is in her `Iddah (woman's prescribed waiting period after divorce) and he did not divorce her twice before. May Allah grant us success. May peace and blessings be upon our Prophet Muhammad, his family, and Companions.