In this paper we consider linear Hamiltonian differential systems without the controllability (or normality) assumption. We prove the Rayleigh principle for these systems with Dirichlet boundary conditions, which provides a variational characterization of the finite eigenvalues of the associated self-adjoint eigenvalue problem. This result generalizes the traditional Rayleigh principle to possibly abnormal linear Hamiltonian systems. The main tools are the extended Picone formula, which is proven here for this general setting, results on piecewise constant kernels for conjoined bases of the Hamiltonian system, and the oscillation theorem relating the number of proper focal points of conjoined bases with the number of finite eigenvalues. As applications we obtain the expansion theorem in the space of admissible functions without controllability and a result on coercivity of the corresponding quadratic functional.