Philosophical views of responsibility often identify responsible agency with capacities such as rationality and self-control. Yet in ordinary life, we frequently hold individuals responsible who are deficient in these capacities, such as children or people with mental illness. The existing literature that addresses these cases has suggested that we merely pretend to hold these agents responsible or that they are responsible to a diminished degree. In this paper, I demonstrate that neither of these approaches is satisfactory, and I offer an alternative focused on the role relationships play in determining whether it is appropriate to hold someone responsible. I argue that relationships are sources of normative expectations about how parties in that relationship ought to behave and that we can be responsible in virtue of being subject to these norms. This is so not only for those who are impaired or immature, but for all of us.