We study the asymptotic behavior of the N-dimensional colored Jones polynomial of the figure-eight knot evaluated at $\exp \bigl ((u+2p\pi \sqrt {-1})/N\bigr )$, where u is a small real number and p is a positive integer. We show that it is asymptotically equivalent to the product of the p-dimensional colored Jones polynomial evaluated at $\exp \bigl (4N\pi ^2/(u+2p\pi \sqrt {-1})\bigr )$ and a term that grows exponentially with growth rate determined by the Chern–Simons invariant. This indicates a quantum modularity of the colored Jones polynomial.