We describe double Hurwitz numbers as intersection numbers on the moduli space of curves ${{\overline{M}}_{g,n}}$ Using a result on the polynomiality of intersection numbers of psi classes with the Double Ramification Cycle, our formula explains the polynomiality in chambers of double Hurwitz numbers and the wall-crossing phenomenon in terms of a variation of correction terms to the $\varphi$ classes. We interpret this as suggestive evidence for polynomiality of the Double Ramification Cycle (which is only known in genera 0 and 1).