The language-guided visual robotic grasping task focuses on enabling robots to grasp objects based on human language instructions. However, real-world human-robot collaboration tasks often involve situations with ambiguous language instructions and complex scenarios. These challenges arise in the understanding of linguistic queries, discrimination of key concepts in visual and language information, and generation of executable grasping configurations for the robot’s end-effector. To overcome these challenges, we propose a novel multi-modal transformer-based framework in this study, which assists robots in localizing spatial interactions of objects using text queries and visual sensing. This framework facilitates object grasping in accordance with human instructions. Our developed framework consists of two main components. First, a visual-linguistic transformer encoder is employed to model multi-modal interactions for objects referred to in the text. Second, the framework performs joint spatial localization and grasping. Extensive ablation studies have been conducted on multiple datasets to evaluate the advantages of each component in our model. Additionally, physical experiments have been performed with natural language-driven human-robot interactions on a physical robot to validate the practicality of our approach.

This paper presents a new approach for geometrically constrained path planning applied to the field of robotic grasping. The method proposed in this paper is based on the Fast Marching Square (FM $\, ^2$ ) and a path calculation approach based on an optimization evolutionary filter named Differential Evolution (DE). The geometric restrictions caused by the link lengths of the kinematic chain composed by the robot arm and hand are introduced in the path calculation phase. This phase uses both the funnel potential of the surroundings created with FM $\, ^2$ and the kinematic constraints of the robot as cost functions to be minimized by the evolutionary filter. The use of an optimization filter allows for a near-optimal solution that satisfies the kinematic restrictions, while preserving the characteristics of a path computed with FM $\, ^2$ . The proposed method is tested in a simulation using a robot composed by a mobile base with two arms.

The paper addresses a problem of “in-hand” locating parts of different shapes in robotic grasping. The goal of the process is to locate a part of an arbitrary shape from an imprecisely determined initial position within a gripper to a final prescribed one. Two possible approaches to solve the problem are considered: non-adaptive, using ordinary rigid jaws of gripper and, adaptive, using an adaptive jaw which improves the performance of the locating process. The latter approach is proposed to be solved by a new type of grasping mechanism. Its theoretical analysis enables to obtain formal conditions for part behavior during the successive steps of the locating process. This process was simulated and then experimentally investigated on an actual gripper model. The proposed new class of mechanisms opens a promising avenue to the creation of a practical class of universal robotic grippers for industry.

We consider an optimization formulation that captures the quality of a grasp. In this model the object geometry, finger contact points and the magnitude of the external load are given. Perturbation of these parameters from their nominal values might lead to significant variations on the predicted grasping quality measures. In this paper we study the sensitivity of a class of grasping quality functions subject to these perturbations and introduce a global sensitivity measure together with a computational procedure to evaluate it. Our analysis is developed within the framework of sensitivity theory and dual methods of nonlinear programming. Numerical simulations supporting the theoretical analysis are presented.