This paper proposes a novel mechanism for the initial alignment of low-cost INS aided by GPS. For low-cost INS, the initial alignment is still a challenging issue because of the high noises from low-cost inertial sensors. In this paper, a two-stage Kalman Filtering mechanism is proposed for the initial alignment of low-cost INS. The first stage is designed for the coarse alignment. To solve the problems encountered by the general coarse alignment approach, an INS error dynamic accounting for unknown initial heading error is developed, and the corresponding observation equation, taking into account the unknown heading error, is also developed. The second stage is designed for the fine alignment, where the classical INS error dynamics based on small attitude error is used. Experimental results indicate that the proposed alignment approach can complete the initial alignment more quickly and more accurately compared with the conventional approach.