In this paper, we give necessary and sufficient conditions for the rigidity of the perimeter inequality under Schwarz symmetrization. The term rigidity refers to the situation in which the equality cases are only obtained by translations of the symmetric set. In particular, we prove that the sufficient conditions for rigidity provided in M. Barchiesi, F. Cagnetti and N. Fusco [Stability of the Steiner symmetrization of convex sets. J. Eur. Math. Soc. 15 (2013), 1245-1278.] are also necessary.