In this paper, we attempt to investigate the attack tolerance of human mobility networks where the mobility is restricted to some extent, for instance, in a hospital, one is not allowed to access all locations. Similar situations also arise in schools. In such a network, we will show that people need to rely upon some intermediate agents, popularly known as the brokers to disseminate information. In order to establish this fact, we have followed the approach of attack in a network which in turn helps to identify important nodes in the network in order to maintain the overall connectivity. In this direction, we have proposed, a new temporal metric, brokerage frequency which significantly outperforms all other state-of-the-art attack strategies reported in Trajanovski et al. (2012), Sur et al. (2015).