A high-precision line-of-sight extraction technique is essential for autonomous optical navigation during the Mars approach phase. To support future Mars exploration missions, an optical image simulation system is a necessary ground verification facility for Mars image generation and line-of-sight extraction algorithm tests. In this paper, an optical image generation procedure is first developed according to projection relationships, reference flight profiles and camera parameters. Next, a hybrid image processing and subpixel-level line-of-sight extraction algorithm is proposed through modification of moment-based sub-pixel edge detection and improvement of direct least-square fitting approaches. Finally, an optical image simulation system is established, and the experimental results show that the proposed procedure can effectively simulate the optical image in the field-of-view of a Mars spacecraft, and the hybrid extraction algorithm can obtain high-precision Mars centroid information.