This paper proposes a virtual reality-based dual-mode teleoperation architecture to assist human operators in remotely operating robotic manipulation systems in a safe and flexible way. The architecture, implemented via a finite state machine, enables the operator to switch between two operational modes: the Approach mode, where the operator indirectly controls the robotic system by specifying its target configuration via the immersive virtual reality (VR) interface, and the Telemanip mode, where the operator directly controls the robot end-effector motion via input devices. The two independent control modes have been tested along the task of reaching a glass on a table by a sample population of 18 participants. Two working groups have been considered to distinguish users with previous experience with VR technologies from the novices. The results of the user study presented in this work show the potential of the proposed architecture in terms of usability, both physical and mental workload, and user satisfaction. Finally, a statistical analysis showed no significant differences along these three metrics between the two considered groups demonstrating ease of use of the proposed architecture by both people with and with no previous experience in VR.

In this article, we review the main results achieved by the research activities carried out at PRISMA Lab of the University of Naples Federico II where, for 35 years, an interdisciplinary team of experts developed robots that are ultimately useful to humans. We summarize the key contributions made in the last decade in the six research areas of dynamic manipulation and locomotion, aerial robotics, human-robot interaction, artificial intelligence and cognitive robotics, industrial robotics, and medical robotics. After a brief overview of each research field, the most significant methodologies and results are reported and discussed, highlighting their cross-disciplinary and translational aspects. Finally, the potential future research directions identified are discussed.

As the proportion of the elderly population in the USA expands, so will the demand for rehabilitation and social care, which play an important role in maintaining function and mediating motor and cognitive decline in older adults. The use of social robotics and telemedicine are each potential solutions but each have limitations. To address challenges with classical telemedicine for rehabilitation, we propose to use a social robot-augmented telepresence (SRAT), Flo, which was deployed for long-term use in a community-based rehabilitation facility catering to older adults. Our goals were to explore how clinicians and patients would use and respond to the robot during rehab interactions. In this pilot study, three clinicians were recruited and asked to rate usability after receiving training for operating the robot and two of them conducted multiple rehab interactions with their patients using the robot (eleven patients with cognitive impairment and/or motor impairment and 23 rehab sessions delivered via SRAT in total). We report on the experience of both therapists and patients after the interactions.

The increase of human presence in the subsea and seabed environments necessitates the development of more capable and highly dexterous, innovative underwater manipulators. Biomimetic soft-robot arms represent a promising candidate for such manipulation systems. However, the well-known modeling techniques and control theories of traditional rigid robots do not apply to soft robots. The challenges of kinematic and dynamic modeling of soft robots with infinite degrees of freedom require the development of dedicated modeling methods. A novel procedure for representing soft-robotic arms and their motion in a rigid-body simulation environment is proposed in this paper. The proposed procedure relies on the piecewise constant curvature approach to simplify the very complex model of hyper-redundant soft-robotic arms, making it suitable for real-time applications. The proposed method is implemented and verified to be used in model-mediated teleoperation of the soft arms of a biomimetic robotic squid designed for underwater manipulation as a case study.

This article is concerned with remote monitoring and control of the 2-degrees of freedom (DoF) robotic manipulators, which have nonlinear dynamics over the packet erasure channel, which is an abstract model for communication over the Internet, WiFi, or Zigbee modules. This type of communication is subject to imperfections, such as random packet dropout and rate distortion. These imperfections cause a significant challenge for monitoring and control of robotic manipulators in the industrial environments because sensitive data, such as sensor data and control commands may not ever reach to their destination resulting in significant performance degradation. Therefore, the effects of these imperfections must be compensated. In this article, we apply two coding and control techniques previously developed for the telepresence ad teleoperation of autonomous vehicles to compensate the effects of the above communication imperfections for remote monitoring and control of the 2-DoF robotic manipulators controlled over the packet erasure channel. To achieve this goal, we design a new linear controller and a new nonlinear controller for the 2-DoF robotic manipulators over the packet erasure channel. The first technique is based on the linearization method and the second technique uses a nonlinear controller. The performances of these two techniques for remote monitoring and control of robotic manipulators are evaluated and compared with each other in this paper. We illustrate their satisfactory performances in the presence of severe communication imperfections.

Although robotic or robot-assisted surgery has been increasingly used by many surgical disciplines, its application in cranial or skull base surgery is still in its infancy. Master-slave teleoperation setting of these robotic systems enables these surgical procedures to be replicated in a virtual reality environment for surgeon training purposes. A variety of teleoperation modes were previously determined with respect to the motion capability of the surgeon’s hand that wears the ring as the surgeon handles a surgical tool inside the surgical workspace. In this surgery training simulator developed for a robot-assisted endoscopic skull base surgery, a new strategy is developed to identify the preferred motion axes of the surgeon. This simulator is designed specifically for tuning the teleoperation system for each surgeon via the identification. This tuning capability brings flexibility to adjust the system operation with respect to the motion characteristics of the surgeon.

The capabilities of teleoperated robots can be enhanced with the ability to recognise and reproduce human-like behaviour. The proposed framework presents motion recognition for a Kinect-based NAO teleoperation. It allows the NAO robot to recognise the human motions and act as a human motion imitator. A stable whole-body imitation is still a challenging issue because of the difficulty in dynamic balancing of centre of mass (CoM). In this paper, a novel adaptive balancing technique for NAO (ABTN) is proposed to control the whole body in single as well as double supporting phases. It targets dynamic balancing of the humanoid robot by solving forward kinematics and applying a weighted average of mass with the CoMs of individual links with respect to the previous joint frames, which provides us with the dynamic CoM of the whole body. Our novel approach uses this dynamic CoM and calculates joint angles using proposed pitch and roll control algorithm to keep the dynamic CoM inside the stable region. Additionally, the NAO robot is capable of recognising human motions using the proposed 7-layer one-dimensional convolutional neural network (1D-CNN). To solve the problem of variable length of time sequences, Zero padding is introduced with 1D-CNN. It attains a recognition accuracy of 95% as compared to the hidden Markov model and neural network. The experimental results demonstrate that the developed teleoperation framework is robust and serves as potential support for the development and application of teleoperated robots.

The enhanced dexterity and manipulability offered by master–slave teleoperated surgical systems have significantly improved the performance and safety of minimally invasive surgeries. However, effective manipulation of surgical robots is sometimes limited due to the mismatch between the slave and master robots’ kinematics and workspace. The purpose of this paper is first to formulate a quantifiable measure of the combined master–slave system manipulability. Next, we develop a null-space controller for the redundant master robot that employs the proposed manipulability index to enhance the performance of teleoperation tasks by matching the kinematics of the redundant master robot with the kinematics of the slave robot. The null-space controller modulates the redundant degrees of freedom of the master robot to reshape its manipulability ellipsoid (ME) towards the ME of the slave robot. The ME is the geometric interpretation of the kinematics of a robot. By reshaping the master robot’s manipulability, we match the master and slave robots’ kinematics. We demonstrate that by using a redundant master robot, we are able to enhance the master–slave system manipulability and more intuitively transfer the slave robot’s dexterity to the user. Simulation and experimental studies are performed to validate the performance of the proposed control strategy. Results demonstrate that by employing the proposed manipulability index, we can enhance the user’s control over the force/velocity of a surgical robot and minimize the user’s control effort for a teleoperated task.

Stability is a major problem with bilateral teleoperator when there is a transmission time delay. By adding adequate damping gain into position servo, a stable bilateral control can be achieved under the existence of time delay. However, large damping gain is required to stabilize the system and the performance is degraded. In this paper, the performance improvement by introducing a high-pass filter into a proportional derivative-based teleoperator with time delay has been studied. We proposed new control law and derived the stability condition. We demonstrated the performance improvement using a hybrid matrix, 1-degree of freedom (DOF), 2-DOF simulations, and 2-DOF peg-in-hole experiments.

This paper proposes a novel control framework for a single-master/multi-slave teleoperation system to grasp and handle an object, considering nonlinearity and uncertainty in the dynamics of the slaves and time-varying delay in the communication channel. Based on passive decomposition approach, the dynamics of the slaves are decomposed into two decoupled systems, and then two higher order sliding mode controllers are designed to control them. Also, a synchronization control methodology for the master is developed. Stability is fully studied using the passivity property and Lyapunov theorem. Finally, simulation and practical results confirm that the control system works well against the conditions.

Humans are experts in manipulation and grasping tasks. However, several industrial tasks represent a risk to human operators, for instance, handling radioactive material or transporting heavy objects. Teleoperation robotic schemes extend human capabilities, but they are highly nonlinear systems. In this paper, we address the problem of dexterous remote manipulation by means of a unilateral heterogenous teleoperation scheme composed by a single-master and multiple-slave manipulators handling a rigid object. In order to achieve a stable grasp, a decentralized force/position controller with continuous and bounded input torques based on the Orthogonalization Principle and a second-order sliding mode control is proposed for the slave robots. In addition, a trajectory planning method based on holonomic constraints is proposed to control multiple-slave manipulators with a single-master device. Experimental results are presented to evaluate the performance of the proposed approach.

This paper presents a novel approach to implement bilateral control loops between local haptic devices and remote industrial manipulators using a layer of simulation and virtual reality. The remote scene of manipulation has been visualized in an open-source software environment, where forward and inverse kinematics of the manipulators can be computed. Therefore, the explicit knowledge of mathematical models of the robots is not required for the implementation of the proposed bilateral control schemes. A haptic coupling has been designed between the human operator and the task in the remote environment. Virtually introduced force feedback has contributed to the performance of the proposed bilateral loop by facilitating the adaptation of unexperienced human operators. Teleoperation of one remote manipulator has been experimentally demonstrated with the proposed controllers. Structural modularity of the bilateral haptic control schemes makes them directly extendable for the teleoperation of multiple collaborative robots. Stability and transparency of the proposed bilateral haptic controllers have been theoretically and experimentally investigated.

This paper proposes a bilateral control structure with a realization of the force derivative in the control loop. Due to the inherent noisy nature of the force signal, most teleoperation schemes can make use of only a proportional (P) control structure in the force channel of the bilateral controllers. In the proposed scheme, an α–β–γ filter is designed to smoothly differentiate the force signal obtained from a reaction force observer integrated to both of the master and slave plants. The differentiated force signal is then used in a proportional-derivative (PD) force controller working together with a disturbance observer. In order to design the overall bilateral controller, an environment model based on pure spring structure is assumed. The controller is designed to enforce an exponentially decaying tracking error for both position and force signals. With the presented controller design approach, one can independently tune the controller gains of the force and the position control channels. The proposed approach is experimentally tested in a platform consisted of direct drive linear motors. As illustrated by the experiment results, the contribution of the PD control in the force channel improves the teleoperation performance especially under hard-contact motion scenarios by attenuating the oscillations, hence, improving the transparency when compared to the structures using only a P force control.

It is a challenging task for a human operator to manipulate a robot from a remote distance, especially in an unknown environment. Excellent teleoperation provides the human operator with a sense of telepresence, mainly including real-world vision, haptic perception, etc. This paper presents a novel virtual environment building method using the red–green–blue (RGB) colour information, the surface normal feature-based 3D-point-cloud registration method and the weighted sliding-average least-square-method-based real-world dynamic modelling for teleoperation. The experiments prove the method to be an accurate and effective means of teleoperation.

This work presents a novel command strategy developed to improve operator performance and minimize difficulties in teleoperation tasks for mobile-manipulator systems with a holonomic base. Aimed specifically at novice operators, virtual fixtures are introduced as a means to minimize collisions and assist in navigation. Using the 6-degree-of-freedom (DOF) Omnibot mobile-manipulator system (MMS), a command strategy is implemented such that the operator need only control a 3-DOF haptic joystick to achieve full control of the Omnibot MMS. The command strategy is used to coordinate control between the arm and the base of the system, prevent collisions with known obstacles, and alert the operator of proximity to those obstacles with haptic forces. Through experimental testing it is shown that operator performance improved with the use of virtual fixtures.

In this paper, we present a novel stochastic framework for network-based bilateral teleoperation systems. A Bayesian approach, which provides robust tracking performance in real-world applications, is proposed to estimate and predict the stochastic variables and compensate for the unreliable network conditions. Combining with a practical approach in transport and application layers of the Internet, this paper demonstrates a high performance and efficient prediction and estimation method for bilateral teleoperation system. Experimental results show that the proposed Bayesian approach estimates and predicts true position and force data over unreliable network conditions, and therefore, improves the performance of overall teleoperation systems.

Due to the special working environment of teleoperation, there are usually uncertainties existing in the dynamics of teleoperating robots. In this paper, an adaptive bilateral control scheme is proposed for the nonlinear teleoperation with parameterized dynamic uncertainties and time delays. Compared to the existing time-delay adaptive bilateral controllers, the proposed scheme has the advantage of faster and more accurate parameter adaptation. In this way, the transient performance of teleoperators can be improved without increasing the control gains, which is helpful for stability of teleoperation. The adaptive bilateral controller is designed to be applicable both in free motion and in constrained motion. The synchronization errors in the unconstrained subspace are asymptotically convergent under time delays. In the constrained subspace, the contact force remains bounded with the proposed controller. The stability of teleoperation is proved using Lyapunov methods and the passivity theory. Simulation results demonstrate the effectiveness of the proposed method.

This paper describes and evaluates the use of a head-mounted display (HMD) for the teleoperation of a field robot. The HMD presents a pair of video streams to the operator (one to each eye) originating from a pair of stereo cameras located on the front of the robot, thus providing him/her with a sense of depth (stereopsis). A tracker on the HMD captures 3-DOF head orientation data which is then used for adjusting the camera orientation by moving the robot and/or the camera position accordingly, and rotating the displayed images to compensate for the operator's head rotation. This approach was implemented in a search and rescue robot (RAPOSA), and it was empirically validated in a series of short user studies. This evaluation involved four experiments covering two-dimensional perception, depth perception, scene perception, and performing a search and rescue task in a controlled scenario. The stereoscopic display and head tracking are shown to afford a number of performance benefits. However, one experiment also revealed that controlling robot orientation with yaw input from the head tracker negatively influenced task completion time. A possible explanation is a mismatch between the abilities of the robot and the human operator. This aside, the studies indicated that the use of an HMD to create a stereoscopic visualization of the camera feeds from a mobile robot enhanced the perception of cues in a static three-dimensional environment and also that such benefits transferred to simulated field scenarios in the form of enhanced task completion times.

Teleoperated grasping requires the abilities to follow the intended trajectory from the user and autonomously search for a suitable pre-grasp pose relative to the object of interest. Challenges include dealing with uncertainty due to the noise of the teleoperator, human elements and calibration errors in the sensors. To address these challenges, an effective and robust algorithm is introduced to assist grasping during teleoperation. Although without premature object contact or regrasping strategies, the algorithm enables the robot to perform online adjustments to reach a pre-grasp pose before final grasping. We use three infrared (IR) sensors that are mounted on the robot hand, and design an algorithm that controls the robot hand to grasp objects using the information from the sensors' readings and the interface component. Finally, a series of experiments demonstrate that the system is robust when grasping a wide range of objects and tracking slow-moving mobile objects. Empirical data from a five-subject user study allows us to tune the relative contributions from the IR sensors and the interface component so as to achieve a balance of grasp assistance and teleoperation.

A novel nonlinear teleoperation algorithm for simultaneous inertial parameters and force estimation at the master and slave sides of the teleoperation system is proposed. The scheme, called Extended Active Observer (EAOB), is an extension of the existing active observer. It provides effective force tracking at the master side with accurate position tracking at the slave side in the presence of inertial parameter variation and measurement noise. The proposed method only requires the measurement of robot position, and as a result significantly reduces the difficulty and cost of implementing bilateral teleoperation systems. The approach is described and its stability is analytically verified. The performance of the method is validated through computer simulation and compared with the Nicosia observer-based controller. According to the results, EAOB outperforms the Nicosia observer method and effectively rejects noise.