To solve the constant contact force control problem between the end tool of a 5 degrees of freedom hybrid optical mirror processing robot and a workpiece, an adaptive impedance control method for the pneumatic servo-polishing system of the robot is designed. Firstly, the pneumatic servo-polishing control system at the end of the robot is set up. Secondly, the impedance control method for contact force is investigated based on the mathematical model of the pneumatic servo-polishing control system. Additionally, the causes of steady-state error of impedance control are analyzed theoretically, and the calculation method for steady-state error of impedance control is deduced. Finally, an indirect adaptive impedance controller based on Lyapunov Stability Principle is developed to estimate the environmental stiffness and position online, so as to reduce steady-state error and realize the tracking of polishing contact force. The simulation and experimental results suggest that the adaptive impedance control method not only recognizes that the contact force of the robot is relatively constant during the polishing process but also has high control accuracy for the force, fast-tracking response for the abrupt force, and considerable adaptability to the variable environmental stiffness.