We give a complete combinatorial classification of the parabolic Verma modules in the principal block of the parabolic category $\mathcal{O}$ associated with a minimal or a maximal parabolic subalgebra of the special linear Lie algebra for which the answer to Kostant’s problem is positive.