Let
$(G,\unicode[STIX]{x1D6EC})$ be a self-similar
$k$-graph with a possibly infinite vertex set
$\unicode[STIX]{x1D6EC}^{0}$. We associate a universal C*-algebra
${\mathcal{O}}_{G,\unicode[STIX]{x1D6EC}}$ to
$(G,\unicode[STIX]{x1D6EC})$. The main purpose of this paper is to investigate the ideal structures of
${\mathcal{O}}_{G,\unicode[STIX]{x1D6EC}}$. We prove that there exists a one-to-one correspondence between the set of all
$G$-hereditary and
$G$-saturated subsets of
$\unicode[STIX]{x1D6EC}^{0}$ and the set of all gauge-invariant and diagonal-invariant ideals of
${\mathcal{O}}_{G,\unicode[STIX]{x1D6EC}}$. Under some conditions, we characterize all primitive ideals of
${\mathcal{O}}_{G,\unicode[STIX]{x1D6EC}}$. Moreover, we describe the Jacobson topology of some concrete examples, which includes the C*-algebra of the product of odometers. On the way to our main results, we study self-similar
$P$-graph C*-algebras in depth.