Many classical Islamic sources argue that it is not permissible for non-Muslims to reside in the Ḥijāz, especially Mecca and Medina. Such arguments are usually based on a famous Prophetic saying, “Two religions should not join/remain in the peninsula/land of the Arabs”, and on the reported action taken by the second caliph ʿUmar b. al-Khaṭṭāb to remove non-Muslims from settlements in western Arabia. In this article, it is argued that the contradictory nature of the evidence for this expulsion casts serious doubt on whether such a widespread action actually took place, certainly not in the decades immediately following Muḥammad's death. It concludes that the widely attested classical prohibition on non-Muslims residing in the Ḥijāz rather had much more to do with the gradually evolving need to draw up firmer communal boundaries, which could help distinguish Muslims from others, and the role played by sacred spaces in doing so.