Finite $W$-algebras are certain associative algebras arising in Lie theory. Each $W$-algebra is constructed from a pair of a semisimple Lie algebra ${\mathfrak{g}}$ (our base field is algebraically closed and of characteristic 0) and its nilpotent element $e$. In this paper we classify finite-dimensional irreducible modules with integral central character over $W$-algebras. In more detail, in a previous paper the first author proved that the component group $A(e)$ of the centralizer of the nilpotent element under consideration acts on the set of finite-dimensional irreducible modules over the $W$-algebra and the quotient set is naturally identified with the set of primitive ideals in $U({\mathfrak{g}})$ whose associated variety is the closure of the adjoint orbit of $e$. In this paper, for a given primitive ideal with integral central character, we compute the corresponding $A(e)$-orbit. The answer is that the stabilizer of that orbit is basically a subgroup of $A(e)$ introduced by G. Lusztig. In the proof we use a variety of different ingredients: the structure theory of primitive ideals and Harish-Chandra bimodules for semisimple Lie algebras, the representation theory of $W$-algebras, the structure theory of cells and Springer representations, and multi-fusion monoidal categories.