We explore the dynamics of affective partisanship and policy divergence in a behavioral voting model. Voters are adaptive and influenced by partisan affect, while political parties are rational and office motivated. We show that the affective partisanship of the electorate and the divergence of party platforms can be mutually reinforcing, thus providing an explanation for the observed co-movement of affective and elite polarization in recent decades. Whether the induced behavioral path exhibits low polarization or high polarization depends on the salience of group identity and the number of moderate voters. Thus, shocks to those factors, perhaps due to such events as economic crises or war, can lead to the polarization or depolarization of the electorate and of the elite.