As Earth-based radio tracking navigation is severely limited because of communications constraints and low relative navigation accuracy, autonomous optical navigation capabilities are essential for both robotic and manned deep-space exploration missions. Image processing is considered one of the key technologies for autonomous optical navigation to extract high-precision navigation observables from a raw image. New image processing algorithms for deep-space autonomous optical navigation are developed in this paper. First, multiple image pre-processing and the Canny edge detection algorithm are adopted to identify the edges of target celestial bodies and simultaneously remove the potential false edges. Secondly, two new limb profile fitting algorithms are proposed based on the Least Squares method and the Levenberg-Marquardt algorithm, respectively, with the assumption that the perspective projection of a target celestial body on the image plane will form an ellipse. Next, the line-of-sight (LOS) vector from the spacecraft to the centroid of the observed object is obtained. This is taken as the navigation measurement observable and input to the navigation filter algorithm. Finally, the image processing algorithms developed in this paper are validated using both synthetic simulated images and real flight images from the MESSENGER mission.