In Canada, long-term care and retirement home residents have experienced high rates of COVID-19 infection and death. Early efforts to protect residents included restricting all visitors as well as movement inside homes. These restrictions, however, had significant implications for residents’ health and well-being. Engaging with those most affected by such restrictions can help us to better understand their experiences and address their needs. In this qualitative study, 43 residents of long-term care or retirement homes, family members and staff were interviewed and offered recommendations related to infection control, communication, social contact and connection, care needs, and policy and planning. The recommendations were examined using an ethical framework, providing potential relevance in policy development for public health crises. Our results highlight the harms of movement and visiting restrictions and call for effective, equitable, and transparent measures. The design of long-term care and retirement policies requires ongoing, meaningful engagement with those most affected.