In the manufacturing process of sophisticated and individualized large components, classical solutions to build large machine tools cannot meet the demand. A hybrid robot, which is made up of a 3 degree-of-freedom (3-DOF) parallel manipulator and a 2-DOF serial manipulator, has been developed as a plug-and-play robotized module that can be rapidly located in multi-stations where machining operations can be performed in situ. However, processing towards high absolute accuracy has become a huge challenge due to the movement of robot platform. In this paper, a human-guided vision system is proposed and integrated in the robot system to improve the accuracy of the end-effector of a robot. A handheld manipulator is utilized as a tool for human–robot interaction in the large-scale unstructured circumstances without intelligence. With 6-DOF, humans are able to manipulate the robot (end-effector) so as to guide the camera to see target markers mounted on the machining datum. Simulation is operated on the virtual control platform V-Rep, showing a high robust and real-time performance on mapping human manipulation to the end-effector of robot. And then, a vision-based pose estimation method on a target marker is proposed to define the position and orientation of machining datum, and a compensation method is applied to reduce pose errors on the entire machining trajectory. The algorithms are tested on V-Rep, and the results show that the absolute pose error reduces greatly with the proposed methods, and the system is immune to the motion deviation of the robot platform.