Joint angle synergy-based humanoid robot motion generation with fascia-inspired nonlinear constraints

Abstract

When generating simultaneous joint movements of a humanoid with multiple degrees of freedom to replicate human-like movements, the approach of joint synergy can facilitate the generation of whole-body robotic movement with a reduced number of control inputs. However, the trade-off of minimizing control inputs and keeping characteristics of movements makes it difficult to improve movement performance in a simple control manner. In this paper, we introduce an approach by connecting and constraining these joints. It is inspired by the fascia network of the human body, which constrains the whole-body movements of a human. Compared to when only joint synergy is used, the effectiveness of the proposed method is verified by calculating the errors of joint positions of generated movements and human movements. The paper provides a detailed exploration of the proposed method, presenting simulation-experimental results that affirm its effectiveness in generated movements that closely resemble human movements. Furthermore, we provide one possible method on how these concepts can be implemented in actual robotic hardware, offering a pathway to improve movement control in humanoid robots within their mechanical limitations.

1. Introduction

The design of humanoid robots [1, 2] often involves a balance between complexity and functionality. Humanoid robots are designed with sufficient degrees of freedom (DOFs) to perform their intended functions. With these DOFs, determining their optimal movements becomes an increasingly intricate challenge when achieving human-like movements: It arises not only from the number of joints and actuators but also from the interaction between them to achieve the proposed movements. Especially when considering such humanoid robots [3, 4, 5], to function in complex and varied human-centric environments, they need to perform a wide range of movements that mimic human capabilities for interactions. This challenge drives the need for innovative control methodologies that allow these robots to perform human-like movements in low control inputs, that is, to move the joints in groups coordinately. In this case, a humanoid robot can be controlled with a relatively low computational cost for better real-life implementation.

Figure 1. The concept of the method. Extracting synergy from the whole-body motion of a human and adding nonlinear constraints based on the extracted synergy. This is realized by conditionally enabling a joint angle to affect other joints with a selected connection pattern. By using this method, compared to linear synergy joint space, the nonlinear synergy joint space will perform movements closer to the original human movement joint space.

In motion generation of humanoid robots with multiple DOFs [6, 7, 8, 9], The concept of “synergy” [10] emerges as a potential solution to the control of achieving human-like movements with minimal DOFs. It suggests that complex movements can be decomposed into simpler coordinated patterns, in which muscle groups work in unison to produce natural movements, and the control efforts are reduced by adapting the simplified patterns with fewer control inputs as fewer control inputs are necessary to direct a wide range of movements. However, there is a trade-off to consider: in the process of minimizing control inputs, essential characteristics of movement might be lost, compromising the realization of natural movements [11]. Tripathi et al. extend existing research by applying a “critical joint” strategy and segmenting body parts for enhanced synergy analysis in humanoid robots, focusing on software solutions while retaining original hardware configurations. It introduces an improved method for analyzing and performing complex movements [12]. Patel et al. adopt nonlinear techniques to analyze hand synergies, focusing solely on movements within the hand [13].

Actually, in addition to the constraint of synergy, there are restrictions by body structure that limit the ability to perform postures beyond a certain range. Therefore, in this study, constraints are added to prevent performing movements that humans cannot achieve, inspired by the human body’s intrinsic mechanisms, particularly the fascia network [14]. When realized in hardware, this constraint can reduce the load on the control. Fascia is an extensive organ of fibrous tissue that permeates the entire body and provides a framework that helps support and protect individual muscle groups, organs, and the entire body as a unit [15]. In our approach, we integrate nonlinear constraints into joint angle synergy modeling. This integration is not a direct replication of the physical properties of the fascia but rather an abstraction of its guiding methods. Our propose draws insights from its methods to craft constraints that guide robot movement. The interconnectivity within the fascia network, where the constrained physical conditions in one fascia line can influence and propagate to connected muscles, serves as a paradigm in crafting these constraints, thereby ensuring a more diverse robot movement.

In this paper, we delve into the detailed exploration of our proposed method, designed for humanoid robots with multiple DOFs. The concept is shown in Fig. 1. The study centers on the development and optimization of the constraints inspired by human fascial connections. By integrating it into the method of joint synergy, through the experiment and evaluations, we focus on how our approach contributes to generating robot movements within the existing control strategy by introducing nonlinear constraints.

When considering the implementation of the method in actual robots, we ensure that each joint is compensated for gravity separately, such as using the mechanism of [16]. As gravity compensation is included, the constraint model imposed by the joint angles can be simplified from kinematics and dynamics consideration to considering the kinematics of the robot posture only. We assume that when the joints are connected, the connected joints rotate according to the amount of rotation of one joint (assuming that there is no loss). The conditions models and notation assumed in the simulation of this study are shown in Fig. 2.

Figure 2. The conditions, models, and notations assumed in the simulation of this study. Consider a situation where 3 joints are connected in series, with the $J_1$ being fixed to the floor. In the case of this figure, we assumed a specific fascia line $L$ , the change of segment fascia tension $\lambda _i$ , which is influenced by the joint angle difference of $q$ (joint angle) and $q_{i(0)}$ (the initial angle of $J_i$ ), will influence the $\kappa$ , the fascia tension. The calculation of $\lambda _i$ and $\kappa$ will be introduced in section 3.4.1. In the case of the figure, $\lambda _1$ and $\lambda _2$ contribute to the calculation of $\kappa$ because, in the modeled configuration in section 3, they can extract the fascia line $L$ . $\lambda _3$ does not contribute to the calculation of $\kappa$ . Subsequently, if the value of $\kappa$ is less than the threshold preset, it will activate the condition of $S_k$ . This mechanism will be introduced in section 3.4.2. Each $S_k$ relates with a corresponding connection pattern $T_k$ . For the joint angle change $\delta q$ in one control cycle, the $w$ in $T_k$ stands for the joint displacement relationship between two angles. The $w$ column index is the joint that starts displacement, and the $w$ row index is the joint that is displaced. In this figure, the joint angle displacement of one control cycle $\delta q$ of $J_3$ will be synchronized with the $\delta q_1$ of $J_1$ , as $w \delta q_1$ . The notations will be further introduced in subsequent sections.

The contributions of this study are threefold:

• • Introduction of fascia-inspired nonlinear constraints: We integrated human fascia network-inspired nonlinear constraints into robotic joint angle synergy, facilitating the replication of human-like movements with multiple joints.

• • Demonstration of movement authenticity with applied constraints: By employing the proposed fascia-inspired nonlinear constraints, we validated that the generated robotic motion closely resembles original human movements compared with those generated by joint synergy.

• • One possible method for hardware implementation: We provided ideas and methods for applying our research approach to actual hardware.

The rest of the paper is organized as follows: Section 2 introduces the background that conceives us of the proposed method, including the meaning of fascia-inspired, and the actuator network system (ANS) that provides a realization method of hardware constraint to the proposed method. Section 3 describes our proposed method of the fascia-inspired nonlinear constraints in joint angle synergy. Section 4 describes the experiment and evaluation procedures and results by using a series of human motion datasets. Section 5 gives a discussion of the proposed method, including the potential hardware realization, and the limitations of the research. Section 6 provides the conclusion of the paper.

2. Background

To introduce the background that conceives us of the proposed method, in this section, we introduce the basic idea of fascia-inspired constraint, and ANS [17], the approach of realizing the constraint in the hardware aspect.

2.1. Fascia inspired constraint

The human fascia is a network of connective tissues that provides support, protection, and structure to muscles and organs, aiding in movement and shock absorption. It is divided into multiple layers and lines, connecting different muscle groups, and enabling coordinated movements. Myers [14] identified several lines in the body where the fascia connects different muscle groups across various joints. Figure 3 shows the typical fascia lines and their muscle components of the human fascia system.

Figure 3. To introduce the structure and function of human fascia lines, we provide the illustration of (a) the superficial back line (SBL) and (b) the superficial front line (SFL) of the human fascia system [14]. For orange-highlighted parts, (a) the lower SBL from toes to knees, (b) the lower SFL from toes to pelvis. Copyright statement: the figure is sketched referring to the illustration of the book: anatomy trains [14].

When a person moves beyond the natural range of motion, specific fascia lines tense to restrict these movements, causing discomfort or pain. This reaction varies when activating different fascia patterns based on different movements. The tension in fascia lines is influenced by the connected muscles’ activity and condition. The tension within the muscles consequently affects the tension of the associated fascia lines.

The constraint inspired by fascia can become a part of the mechanisms that realize human-like postures and movements, which may benefit in generating posture and movements closer to humans. By exploring the idea of implementing physical constraints inspired by fascia on humanoid robots, there is a possibility that human-like postures and movements could be achieved without the necessity for additional computational costs.

2.2. Actuator network system

To realize such physically connected constraints, we introduce our proposed ANS, a mechanism that forms a connection network by fluid-transmissive tubes composed of cylinders and valves [17]. ANS changes the movements of cylinders by switching the connection patterns among the actuators.

To perform the switch in ANS, a connection controller is used. When comparing the fast-changing control cycle on a standalone actuator with a feedback control, the connection controller controls actuator groups and does not need to change the control cycle that fast. As a result, the complexity of the control is reduced. Although this method does not allow for precise control of each actuator, the connection enabled simplified control in scenarios where functional synergetic motion is carried out by the interaction of groups of actuators. By physically establishing the constraints among the modular joints, different movement responses of a spinal robot to external forces can be generated without extra computational cost, by switching the connection patterns [18]. Switching the connection patterns of ANS also generated more suitable dynamics for the surrounding environment [19, 20].

Despite our main proposal in this paper being the fascia-inspired constraints, as exploiting ANS indicates the potential of realizing the main proposal, an implementation example of a physical constraint that can be realized by ANS is shown in Section 5.

3. Fascia-inspired nonlinear constraints in joint angle synergy

Conventionally in humanoid robots, each input angle corresponds to an actuator. Instead of controlling each joint independently, joint synergy reduces the dimension of the inputs by decomposing the inputs and reconstructing them while keeping maximized movement characteristics.

To introduce the fascia-inspired constraints in joint angle synergy, in this section, we set up a specific scenario to describe how nonlinear constraints can be implemented and optimized in the humanoid robot system. Following this, we delve into the details and explore the methodology applied for implementing and optimizing the nonlinear constraints.

3.1. Overview of the method

Instead of replicating the original movement with the synergy control only, we consider exploiting the inherent mechanism based on the prior knowledge of the fascia-inspired nonlinear constraints, introduced in Section 2.

Therefore, based on the simplicity of the synergy control method, we aim to introduce constraints that enable the joints to affect mutually, making the robot model generate motions in addition to linear combined motions. In this research, it refers to applying constraints on the multiple DOFs joints in the aspect of rotation angles.

This approach aims to reduce the errors in conventional joint synergy control and ultimately expects to generate movements more reminiscent of human motions. It is divided into 4 steps, as shown in Fig. 4:

Figure 4. The overall flowchart of the proposed method. To show the whole implementation scenario, the approach is divided into 4 steps: 1) Define the base model, 2) reconstruct the joint angles, 3) implement nonlinear movement constraint, 4) optimization.

1) Defining the human model as the base model

Referring to Section 3.2, the human model is defined for the problem statement, which describes the joint rigid structure with a representation of forward kinematics and joint coordinates. It provides a reference for simulating human movements and serves as a benchmark for subsequent modeling steps.

2) Reconstructing the joint angles by extracting and exploiting joint angle synergy

Referring to Section 3.3, the proposed method progresses as an extension to joint synergy; therefore, a synergy extraction method on the defined based model is introduced. We obtained motion sets of reduced dimension joint angles that keep the characteristics of captured human movements. The extraction result is used to reconstruct the joint angles of the human model, as the input of the nonlinear movement constraint implementation.

3) Implementing the nonlinear movement using the proposed model of fascia constraint activation

Referring to Section 3.4, we propose an approach by introducing a nonlinear movement constraint that mimics the mechanism of fascia constraint activation. The human model kinematics is operated through a form of joint angle increment $\boldsymbol{\delta }\textbf{e}$ , which is defined as the angle change that is influenced by the implemented constraints of every control cycle. For the reconstructed input joint angle of each control cycle, to estimate the joint angle increments, the movement of the human model is constrained by different connection patterns. The connection patterns are selected by different constraint activation conditions, according to the calculation result of the fascia constraint model and the threshold presets, as a general determination criterion.

4) Optimization on connection patterns

Referring to Section 3.5, the configuration of connection patterns among the joints is refined through an optimization process. This is achieved by formulating and minimizing error functions that quantify the deviation between the kinematics constrained by our model and those from naive joint synergy.

3.2. Human model

To explain the proposed method, we introduce a 6-joint human model as the base model, to show a whole-body human motion in a 2-D lateral right-hand coordinate frame.

Figure 5. The initial posture and the joint angle definition with the base model. Joint angles are derived from the arc-tangent of the ratio between the y and x coordinates differences of relevant points. For the sake of the figure demonstration, the $J_4$ and $J_5$ in (a) are separated, in fact in the defined human model they coincide, as shown in (b).

Figure 5 shows the definition of the model, including our defined initial posture for subsequent calculation of the fascia constraint model, notated as $\mathbf{q_{ini}}$ . The model consists of 6 active joints, 2 reference joints, and rigid structures.

The active joints fulfill the minimal requirements of performing a whole-body motion of a human. They refer to a joint actuation with the capability to execute controlled rotational movements. In this model, these joints are named as $J_1, J_2, J_3, J_4, J_5, J_6$ (ankle, knee, hip, shoulder, neck, and elbow) as $\textbf{J}_{j}$ $(j=1,2,\ldots,6)$ .

The reference joints are set for calculating $ q_j$ which are adjacent to end effectors. They do not have an actuation mechanism but need to be included in joint angle calculation for biological similarity, as head and wrist joints in this model, named as $R_1, R_2$ (head, wrist).

The $\mathbf{q_{ini}}$ is a starting position where all joint angles are set as $[\pi/2, 0, 0, 0, 0, 0]$ , serving as a reference posture for measuring subsequent movements. The joint angle $ q_j$ is defined to measure the rotation at the active joints, calculated based on the angles between vectors formed by structurally connected joints, the vector relationships of $q_j$ are shown in Fig. 5, and calculated as the following formula:

(1) \begin{equation} q_{\text{j}} = \arctan \left ( \text{v}_{a} \right ) - \arctan \left ( \text{v}_{b}\right ) \end{equation}

For the calculation of $\text{v}$ , refer to Appendix A for the details of the inverse kinematics.

Vise versa, the joint coordinates also can be calculated by the input joint angles with the human model, for error comparison in the optimization. The calculation progress of the direct kinematics is given in Appendix B.

3.3. Joint angle synergy

The purpose of extracting synergy in a multiple-joint robot system is to describe the higher dimensional joint angle space with a low-dimensional synergy space [21], while maximizing the variance of the projected data. For the control simplicity, only dominant synergies are retained.

Principal component analysis (PCA) [22], is widely used in extracting the joint angle synergy of robots [12, 23]. As its efficacy is validated in the literature, in this research, we use PCA to extract the joint angle synergy of the human model.

The inputs of PCA, a dataset of frames of joint angles $\textbf{q}_{\textbf{n}}= \{\textbf{q}_{\textbf{1}}, \textbf{q}_{\textbf{2}}, \textbf{q}_{\textbf{3}}, \textbf{q}_{\textbf{4}}, \textbf{q}_{\textbf{5}}, \textbf{q}_{\textbf{6}}\}$ , where multiple frames $n = 1, \ldots, N$ . The $\textbf{q}_{\textbf{1}}$ to $\textbf{q}_{\textbf{6}}$ is the $n$ frames of joint angles from $J_1$ to $J_6$ .

Implement PCA on the data set, the decomposition result of the $\textbf{q}_{\textbf{n}}$ is

(2) \begin{equation} \mathbf{q_n}=\bar{\mathbf{q}}+\tilde{\mathbf{q}}_{{\textbf{n}}}\mathbf{U} \end{equation}

the dataset joint angle mean value $\bar{\mathbf{q}}$ , eigenvector matrix $\mathbf{U}=\left \{ \mathbf{u_1}, \mathbf{u_2}, \mathbf{u_3}, \mathbf{u_4}, \mathbf{u_5}, \mathbf{u_6}\right \}^T$ , the projection matrix $\tilde{\mathbf{q}}_{\textbf{n}}$ .

An over 80% cumulative contribution rate decides the number of components to retain. Suppose retaining 2 components, obtaining the reconstructed joint angles

(3) \begin{equation} \mathbf{q}_{rec} = \bar{\mathbf{q}} + \mathbf{q}_\boldsymbol{r}\mathbf{V}_{\textbf{s}} \end{equation}

the synergy coordinates $\mathbf{V_s}$

(4) \begin{equation} \mathbf{V}_s = \left \{\mathbf{u}_1, \mathbf{u}_2, \mathbf{0}, \mathbf{0}, \mathbf{0},\mathbf{0}\right \}^T \end{equation}

and the joint inputs

(5) \begin{equation} \mathbf{q}_r = \left \{\tilde{\mathbf{q}}_{\textbf{1}}, \tilde{\mathbf{q}}_{\textbf{2}}, \mathbf{0}, \mathbf{0}, \mathbf{0},\mathbf{0}\right \} \end{equation}

As the $\mathbf{V_s}$ are not time-variant and $\mathbf{q}_\boldsymbol{r}$ is a lower dimensional input signal, in a robot system, such extracted joint angle synergy can reduce the control dimensions of multiple joint inputs, while $\mathbf{q}_{rec} \sim \mathbf{q}_n$ .

The Detailed method of PCA decomposition can be found in Appendix C.

3.4. Fascia-inspired nonlinear constraints

The concept of the proposed method is inspired by the observation that human muscles are grouped and interconnected through fascia. Also, the feature of fascia has been studied by researchers in the robotics field [24], by using a robot to study how the foot’s windlass mechanism, comprising plantar fascia, affects jumping, resulting in a simplification of the buliding foot mechanisms of a bipedal robot.

3.4.1. Fascia and fascia constraint model

We suppose that there are multiple fascia lines surrounding the joints of the robot. These “fascia lines” do not exist physically, yet provide morphology criterion by their math models with joint angle inputs. When the robot system tends to perform malformed motion, similar to the human fascia network, the fascia aids and generates responsive constraints or feedback based on the predefined math models. To achieve this, the fascia lines are represented as an extractable line constraint, which is utilized in determining the activation conditions for fascia.

The fascia constraint model is defined as a set of fascia line models $\textbf{L}$ . For a specific fascia line $L \in \textbf{L}$ , connecting designated joints $J \in \textbf{J}_{j}$ referring to the anatomy of human fascia, will increase the tension when the relative joints rotate tending to extract the fascia line, shown in Fig. 6. The extraction direction $\beta$ is defined as $1$ when the placed fascia segment extracts when the joint rotates counterclockwise, $-1$ when the placed fascia segment extracts when the joint rotates clockwise. $\beta$ can only be $+1$ and $-1$ .

Figure 6. The fascia constraint model. The extraction of a segment fascia line is based on its placement and its alignment with the rotation direction. (a) The relationship of rotation direction $q_i - q_{i(0)}$ and the extraction direction $\beta$ . (b) When the joint rotation direction toward $q_{i(0)}$ and extraction direction are the same, the fascia segment extracts. The $\lambda _i$ are assigned first and used to calculate $\kappa$ . (c) For all selected $J_i$ in $J$ , the set of designated joint in a specific fascia line $L$ . (d) The placement of lower SBL and the lower SFL.

A segment fascia tension $\lambda _i$ is defined as the extraction of a segment fascia line of $J_i$ , as a part of the fascia line that surrounds a joint $J_i$ , shown in Fig. 6 (b). Suppose a segment fascia tension $\lambda = 0$ is the initial state of a segment fascia line. For a certain fascia line $L_{linename}$ , the calculation of the fascia tension $\kappa$ of one frame of input angles $\mathbf{q}$ , using the designated joints $J$ , is a cosine value of equally weighted sum up of the designated joints’ segment tension $\{\lambda _i\}$ , For all selected $J_i$ in $J$ , as

(6) \begin{equation} \lambda _i = \begin{cases} \lambda _{0}\cdot \left ({q_i - q_{i(0)}}\right ) & \left ({q_i - q_{i(0)}}\right )\beta \gt 0 \\ \lambda = 0 & \left ({q_i - q_{i(0)}}\right )\beta \leq 0 \end{cases},\forall J_i \in J \end{equation}

(7) \begin{equation} \kappa = \cos \left ( \frac{\sum _{i\in J}\lambda _i J_i}{P} \right ), \forall J_i \in J \end{equation}

where the segment tension is a deviation of input angles of $J_i$ and the corresponding initial angle $q_{i(0)}$ , where $\lambda _{0}=1$ in this research. $P$ is the total number of the designated joints in $J$ . $i$ is the index of the designated joints for $J$ . When there is no extraction on a fascia line, $\kappa =1$ . When there is extraction on a fascia line, $\kappa \lt 1$ .

In this research, we focus on the whole-body motion from the head to the toes. Therefore, we chose the superficial back line (SBL) and the superficial front line (SFL), shown in Fig. 3 as the target of modeling, which has a reciprocal relationship in providing tensile support to lift the rigid which extend backward and forward the gravity line.

Subsequently, we discuss the fascia constraints of the lower SBL and the lower SFL. In the remaining paper, when the fascia line is referred to, we are indicating the fascia constraint model proposed in this research.

3.4.2. Constraint activation conditions for fascia

The constraints of fascia are implemented according to the conditions of the joint angles of every control cycle. The conditions are identified by checking if the tension of a fascia line has surpassed its threshold preset. If surpassed, the correspondent line is treated as "malformed" and labeled as classifying evidence of the current condition. Theoretically, with $C$ fascia lines adapted, $2^C$ conditions are classifiable, within 1 condition of normal, of not malformed condition.

Giving a series of joint angles of a motion set, by using the calculation formula of the fascia tension of a specific line, the time-variant tension can be shown in Fig. 7. For the classified states $S_1, S_2, S_3, S_4$ , the frame sequences that belong to each state are defined as sets $s_1, s_2, s_3, s_4$ , correspondingly, as $s_1, s_2, s_3, s_4 \in \{1, 2, \ldots, N\}$ , while they are independent. The threshold preset is set by human observations, and its configuration is not discussed in this research.

Figure 7. The constraint activation mechanism using fascia tense model. The condition classification is decided by the labels of fascia lines. In this research; 2 fascia lines (lower SBL, lower SFL) are used to constrain the motion frames. Therefore, the motion frames are categorized into 4 conditions: $S_1, S_2, S_3, S_4$ .

Based on the reconstructed joint angle, for every frame (control cycle), the condition of the joint angle is checked in real time. For each time the condition $S_k$ changes, the constraint will switch to the connection pattern $\mathbf{T_k}$ corresponding to the condition, which will be explained in the next subsection. $k$ is the index of the constraint condition pattern, and $K$ is the total number of constraint condition patterns. In this research, we use two lines to activate the constraint condition: lower SBL and lower SFL.

3.4.3. Implementing nonlinear movement constraints on the human model

When the condition of the input motion transits into a condition that the fascia lines are malformed, the nonlinear movement constraint is implemented, to admit a part of joint angles to be transmitted among the joints, pursuing the natural balance of the motion.

A connection pattern is defined for each classified condition, enabling the rotation of one joint to affect the rotation of another one, representing the transition of joint angles. Therefore, in the hardware aspect, the constraint of one joint is a proportional angular displacement from another joint, according to the joint connection pattern configuration at the moment of every control cycle. Figure 8 shows the math representation of the displacement progress.

Figure 8. The math representation of one joint angular displacement from another joint. (a) The physical illustration of $w_{1}\delta{q_2}$ , the displacement using a specific pattern $\mathbf{T_k}$ . (b) The physical meaning of $w_{1}\delta{q_2}$ in different connection patterns $\mathbf{T_K}$ . (c) The physical meaning of $\mathbf{T_k} \cdot \boldsymbol{\delta }\textbf{q}$ , where a specific pattern is illustrated as an example.

In a time-variant system, the whole movement constraint process by initially inputting $\textbf{q}_{\textrm{rec}}$ is, physically, integration progress. In the math representation, for simplicity in a discrete form of the model, we first introduce the increment $\boldsymbol{\delta }\textbf{e}$ , as the cumulative term of angular displacement is influenced by the implemented constraints of every control cycle, a transmission matrix $\mathbf{T_k}$ . Note that $S_k$ and $\mathbf{T_k}$ are bijective, $S_k \to \mathbf{T_k}$ , $\mathbf{T_k} \to S_k$ .

For one $\mathbf{T_k}$ , When the constraint is activated, in every control cycle, the increment $\boldsymbol{\delta }\textbf{e}$ is defined as

(8) \begin{equation} \boldsymbol{\delta }\textbf{e} = T_k \cdot \boldsymbol{\delta }\textbf{q} \end{equation}

where $\boldsymbol{\delta }\textbf{q}$ is the joint angle difference compared to the previous control cycle. And $\mathbf{T_k}$ , $\boldsymbol{\delta }\textbf{q}$ is consists of form

(9) \begin{equation} \begin{bmatrix} 0 & w_{1}^{k} & 0 & 0& 0 & 0\\[5pt] 0 & 0 & w_{2}^{k} & 0& 0 & 0\\[5pt] 0 & 0 & 0 & 0 & w_{3}^{k} & 0\\[5pt] 0 & 0 & 0 & 0 & 0 & 0\\[5pt] 0 & 0 & 0 & 0 & 0 & 0\\[5pt] 0 & 0 & 0 & 0 & 0 & 0 \end{bmatrix} \begin{bmatrix} \delta q_{1}\\[5pt] \delta q_{2}\\[5pt] \delta q_{3}\\[5pt] \delta q_{4}\\[5pt] \delta q_{5}\\[5pt] \delta q_{6} \end{bmatrix} = \begin{bmatrix} w_{1}^{k} \delta q_{2}\\[5pt] w_{2}^{k} \delta q_{3}\\[5pt] w_{3}^{k} \delta q_{5}\\[5pt] 0\\[5pt] 0\\[5pt] 0 \end{bmatrix} \end{equation}

$w$ indicates the transmission proportion of one joint from another joint. In the case of a parameter configuration, as $\delta q_1 = w_{1}^{k} \delta q_{2}$ , $\delta q_2 = w_{2}^{k} \delta q_{3}$ , $\delta q_5 = w_{3}^{k} \delta q_{5}$ , meaning the $J_1$ (ankle) will be constrained by $J_2$ (knee), $J_2$ (knee) will be constrained by $J_3$ (hip), $J_3$ will be constrained by $J_5$ (neck), consistent with the illustration in Fig. 8 (c).

Note the picked-up joints to represent $\mathbf{T_K}$ are different from those in the activation of the fascia constraint model $J$ . The parameter $w$ is what we configure in the next section, meanwhile, the parameter settings and configurations can be varied, although configured as Equation (9) in this research.

To sum up, the constraint method is a time-variant process from the start of a motion depending on the nonlinear switch of the condition patterns as $S_K$ . Consequently, the constrained motion $\textbf{q}_{\textrm{cst}}$ at frame $n$ , $\textbf{q}_{\textrm{cst(n)}}$ is composed of two parts: 1) the reconstructed joint angles of synergy and 2) the nonlinear movement constraint, representing as

(10) \begin{equation} \mathbf{q_{cst(n)}} = \mathbf{q_{rec(n)}} + \sum _{1}^{n}\sum _{k=1}^{K}\boldsymbol{\delta }\textbf{e}\times \mathcal{G}(n) \end{equation}

while

\begin{equation*} \mathcal {G}(n) = \begin {cases} 1 & n \in s_k \\[5pt] 0 & \text {others} \end {cases} \end{equation*}

where $s_k$ is the state set that includes the frame index of the $k$ condition, as mentioned the constraint will adjust the connection pattern that corresponds to the switch of the condition.

3.5. Optimization of the nonlinear movement constraint parameters

We optimize the nonlinear constraint parameters to make the constrained motion become closer to human-performed motion, comparing it with the motion reconstructed by synergy.

To achieve it, the loss function of the whole optimization $\mathcal{L}$ is defined as the comparison of two error functions: 1) between the synergy motion and the original motion $\overline{E_{syn}}$ and 2) between the constrained motion and the original motion $\overline{E_{cst}}$ . This can be represented as

(11) \begin{equation} \mathcal{L} = \overline{E_{cst}} - \overline{E_{syn}} \end{equation}

To calculate them, their general form, the error function $\overline{E}$ , is used to estimate the overall motion performance of all motion sets. Therefore, it is defined as the average of the sum of $E$ . which is the overall error of $N$ frames of all motion sets in Cartesian coordinates by using the direct kinematics human model, inputting the joint angles, as

(12) \begin{equation} \overline{E} = \frac{1}{N}\sum _{i=1}^{N}E \end{equation}

where $N$ is the total frames of all motion sets. $E$ , as the error of one motion frame, is calculated as

(13) \begin{equation} E=\sum _{j=1}^{Z}e_{j(n)} \end{equation}

which $e_{j(n)}$ is the standard square error of the human model for $n$ the frame in the Cartesian coordinates of one joint $J_j$ as

(14) \begin{equation} e_{j(n)}=\sqrt{\left ( y_{j(n)}-\widehat{y_{j(n)}} \right )^2} \end{equation}

$j$ is the joint index, $Z$ is the counts of all joints of human model (including active ones and reference ones), and $n$ is the frame index.

For a series of motion sets, as shown in Fig. 9, $\widehat{\textbf{y}_{\textrm{j(n)}}}$ is the joint coordinates with the original joint angles input, $\textbf{y}_{\textrm{j(n)}}$ is obtained using the human model of direct kinematics with the input $\textbf{q}_{\textrm{(n)}}$ , the joint angle of $n$ th frame. In this research, the joint input of the proposed model $\textbf{q}_{\textrm{cst}}$ and the reconstructed joint angles of synergy model $\textbf{q}_{\textrm{rec}}$ are used.

Figure 9. The error calculation for a series of motion inputs.

The L-BFGS-B [25] solver is used, as this method optimizes a large number of parameters, using the L-BFGS-B method provides a balance between computation time and accuracy.

To summarize, the loss function, that is, the function to be optimized, is minimized by configuring $w$ in Section 2.4.4, represented as formula

(15) \begin{equation} \begin{aligned} \frac{1}{N}\sum _{n=1}^{N}\Bigg \{ & \sqrt{\sum _{j=1}^{Z}\Bigg [ f\Bigg ( \mathbf{q_{\text{rec(n)}}} + \sum _{1}^{n}\sum _{k=1}^{K}\mathbf{\mathbf{T_k}} \cdot \boldsymbol{\delta }\textbf{q} \times \mathcal{G}(n) \Bigg ) -f(\mathbf{q_{(n)}}) \Bigg ]^{2}} - \sqrt{\sum _{j=1}^{Z}\Bigg [ f\left (\mathbf{q_{\text{rec(n)}}} \right )-f(\mathbf{q_{(n)}}) \Bigg ]^{2}} \Bigg \} \end{aligned} \end{equation}

As $f$ is a continuous function of calculating direct kinematics of the human model, and $\mathbf{T_k}$ is composed of $w^k$ and their derivatives exist. Therefore, the function can be partially differentiated.

The overall optimization step is shown in Algorithm 1.

Algorithm 1 Optimization Using Human and Error Models

4. Experiments and evaluation

To to assess the effect of the presence of the fascia constraint model and to show its effectiveness in the motion generation of a multijoint humanoid robot, we carried out experiments by acquiring the original human motion set and evaluated our proposed method using the collected data. In this study, the study basically addressed movements within the sagittal plane.

Figure 10. The configuration of the motion capture system and the captured subjects. The $OptiTrack$ motion capture system is used to detect the markers’ locations in the set coordinate frame. Capture frame rate: 120fps, export frame rate: 120fps. Rotation type: quaternion.

4.1. Collection of human movement data via motion capture

We use a motion capture system (Trio tracking system V120, NaturalPoint, Inc. DBA $OptiTrack$ ) to obtain the estimated positions of the joints of a human, by acquiring reflective marker positions when a human subject is performing motions, as shown in Fig. 10. Consequently, we collected 10 kinds of positional motion sets with 11,551 frames of humans using a motion capture system, shown in Table I, the data is the joint angles on the 2-D lateral side of a human, showing a human using his whole body to perform different motions. Each motion is performed 3 to 4 times. Assume one frame is a control cycle.

The marker position using the motion capture system is shown in Fig. 11, in this research, $\mathcal{Q}_1, \mathcal{Q}_2, \mathcal{Q}_4, \mathcal{Q}_{10}$ set as the joint position of $J_1, J_2, J_3, J_6$ of the human model, respectively, $\mathcal{Q}_6$ as the joint position of $J_4, J_5$ , $\mathcal{Q}_8$ as $R_1$ , $\mathcal{Q}_{11}$ as $R_2$ .

Table I. The motion descriptions.

Figure 11. Marker placements on the human body.

4.2. Extraction of joint angle synergy

By adapting the method introduced in Section 2, we extracted the joint synergy from the obtained motion sets. The dataset mean $\bar{\mathbf{q}}=$ [1.32, 0.80, −0.98, −2.50, −0.66, 0.57] (rad), the first 2 synergies (shown in Fig. 12) count for 91.3% cumulative contribution rate. Hence, we choose the first 2 synergies to reconstruct the joint angles as $\textbf{q}_{\textrm{rec}}$ , the reconstruct method is explained in Section 2.

4.3. Optimization of fascia-inspired nonlinear constraints

According to the constraint activation condition calculation method in 2.4.3, for chosen fascia line lower SBL and lower SFL, $\kappa _{LSBL}$ of lower SBL, the designated joint angles $J$ are $J_1, J_2$ : $\kappa _{LSFL}$ of lower SFL, the designated joint angles $J$ are $J_1, J_2, J_3$ . The $\kappa _{LSBL}$ , $\kappa _{LSFL}$ toward frames for M1 to M10 is calculated.

According to the classification method of constraint activation condition in Section 2.4.3, we categorize the motion frames into 4 conditions as mentioned in Fig. 7. Their switching relationship toward time is shown in Fig. 13.

Figure 12. Cumulative contribution rate from the 6 decomposed eigenvectors, 91.3% of the data can be represented by using 2 principal components.

Figure 13. The M1 to M10 motion sets are activated into 4 conditions $S_K$ by frame. The red part demonstrates the scopes of the activated condition $s_k$ .

To achieve a seamless and physically feasible transition when switching motion conditions as in the real world, it is essential to interpolate the state weight correlating with different states. The calculation process is given in Appendix D.

After these configurations, we process the optimization. According to 2.4.4, parameters in $T_1$ , $T_2$ , $T_3$ ( $w_{1}^{1}, w_{2}^{1}, w_{3}^{1}, w_{1}^{2}, w_{2}^{2},w_{3}^{2},w_{1}^{3},w_{2}^{3},w_{3}^{3}$ ) are the optimization variables ( $T_4=0$ as a normal condition). The $T_1$ , $T_2$ , $T_3$ , $T_4$ correspond to 4 conditions. The inputs are the reconstructed joint angles $\mathbf{q_{rec}}$ and constrained joint angles $\mathbf{q_{cst}}$ . Minimize $\mathcal{L} = \overline{E_{cst}} - \overline{E_{syn}}$ of the whole motion set within 200 iterations. The calculation of L-BFGS-B is given in Appendix E.

4.4. Evaluation methodology

We aim to optimize the nonlinear movement constraint parameters to achieve motion that closely resembles human-performed motion.

The optimization involves adjusting parameters $w$ to minimize the loss function defined as the difference between two error functions: the error between the synergy motion and the original motion and the error between the constrained motion and the original motion.

The original motion is a part of collected human movement data, as a one-time performance of each motion collected. The reason for only using a part is to increase the optimization speed in the observation of the homogeneity of multiple times of movements in each kind of motion.

4.4.1. Experiment condition

To validate our approach, a series of optimizations were executed on a computer possessing the following specifications: CPU: AMD 7950X, GPU: NVIDIA RTX 4090, RAM: 32 GB, Software platform: python 3.9.13 on Visual Studio Code. Each optimization took around 40 minutes, including 200 iterations, $N$ = 3755.

4.4.2. Evaluation metrics

With the above conditions in mind, our evaluation comprises the following metrics:

1) The loss function $\mathcal{L} = \overline{E_{cst}} - \overline{E_{syn}}$ at the convergent solution.

2) The median error from M1 to M10, comparing the two human model errors of input $\mathbf{q_{rec}}$ and optimized $\mathbf{q_{cst}}$ , $w$ .

3) The error distribution, depicted as violin plots, comparing the two human model errors of input $\mathbf{q_{rec}}$ and $\mathbf{q_{cst}}$ , $w$ .

4.5. Results

After 200 iterations, the optimization can get a convergent solution with an average error difference of −0.058, which $\overline{E_{cst}}$ = 0.5197, $\overline{E_{syn}}$ = 0.5781.

From the optimization result, we got the solution of $w$ of $w_{1}^{1} =0.26, w_{2}^{1}=-0.24, w_{3}^{1}=-0.01, w_{1}^{2}=-0.24, w_{2}^{2}=2.35,w_{3}^{2}=-2.02,w_{1}^{3}=-1.09,w_{2}^{3}=-5.78,w_{3}^{3}=3.53$ .

Median errors for each motion set in the proposed and synergy models: Proposed Model/ Synergy Model: M1: 0.362/ 0.377, M2: 0.309/ 0.514, M3: 0.630/ 0.630, M4: 0.543/ 0.595, M5: 0.550/ 0.550, M6: 0.302/ 0.302, M7: 0.256/ 0.263, M8: 0.368/ 0.427, M9: 0.511/ 0.813, M10: 0.403/ 0.738.

We use the optimized parameters to reconstruct the robot motion and visually compare the distribution of error between the proposed model and the synergy model across various motion sets (M1 to M10) with a violin plot shown in Fig. 14.

The human model of M1 motions of the synergy model and the constraint model, which is rebuilt as a series of motion capture clips, is shown in Fig. 15.

Figure 14. The error comparing the synergy model $E_{syn}$ and the proposed model $E_{cst}$ . The distribution is calculated by the $E$ of each frame. The lower the error, the closer to the original motion.

Figure 15. The comparison of video clips of the human model of M1 motions. From left to right: original motion, reconstructed joint synergy model, the proposed model. They are rebuilt with the forward kinematics of the human model. The clips are captured every 30 frames. The connection pattern start switching at f = 161.

Comparing the synergy model, the proposed model is more effective in performing original human motion with a lower median error and lower error distribution.

5. Discussion

As a result, the average error difference of the proposed model is smaller than that of the reconstructed synergy model. It means that in the aspect of postures of the human model during the movement performance, the motion generated by the proposed model is closer to the original human motion. This is because introducing nonlinear constraint adjusts the joint space of the linear synergy, resulting in the complementary toward the essential characteristics of movements.

In the evaluation, we not only focus on the decline in the average error of overall movements but also attention is paid to the specific movements where the error has been significantly reduced. By examining the nature of these movements, we considered the reason for a substantial decrease in error, of whether the movements are closely related to the fascia being studied. If a movement is unrelated to the fascia in focus, its error is unlikely to be considerably influenced. Conversely, it would be peculiar if the error diminished substantially across a wide array of movements, irrespective of their relevance to the fascia being studied.

In the case of this research, we observed from the results that after the implementation of the nonlinear constraint, a significant decrease in error was noted in M1. This outcome is attributed to the activation of the lower SBL and lower SFL effect, which facilitated the adjustment of motion using the optimized connection pattern, thereby enhancing the support effect.

For motions M5 and M6, no noticeable change was observed. This lack of variation is due to the non-activation of the nonlinear constraint effect, which implies that the predefined conditions necessary to trigger the nonlinear constraint were not met in these motions.

In instances where the nonlinear constraint was activated, but the effect was not pronounced, for instance, M7 and M8. This is likely due to the insufficient magnitude of the applied constraints to create a notable alteration in the motion patterns or due to the inherent characteristics of the motions themselves that make them less susceptible to modifications by the nonlinear constraints.

On the other hand, this method does not add an extra control signal based on the input of the joint synergy. It works by switching the pre-configured connection pattern among the joints, as the $\mathbf{T_K}$ are constant matrices and do not change for each condition once pre-configured.

The feature of the proposed system is its capacity for minor adjusting the joint angles. This is achieved by utilizing the intrinsic rotation of the joints, which allows for minor adjustments compared to traditional linear joint synergy control methods. This is beneficial in applications where similar reduced dimension systems or systems require a single numeral signal to adjust the movements. In previous systems, the synergy approach has been shown to effectively facilitate upright postures and maintain balance [26]. This is evident in the context of standing-up movements [27].The connection constraints to this framework can additionally enhance the performance of human-like movements with nonlinear features, with the results shown in Section 4.5.

5.1. Potential hardware realization for nonlinear constraints

In this research, the proposed nonlinear constraints can be implemented in either the software or hardware aspect. The proposed model is consistent with the hardware mechanism of the ANS. The configuration of intrinsic constraint condition can be realized by exploiting ANS without increasing the computation loads as explained in Section 2.2. Using ANS, the response delay does not exist as it can be realized by physical constraints among joints without electrical sensors or devices.

Figure 16. The proposed hardware mechanism. A rotatable joint, equipped with a pulley, link, free-rotation pulley, and cylinder, functions by having its pulley rotated by a motor through input angles, which subsequently moves the cylinder piston via the pulley-tendon mechanism. This process facilitates force transmission, providing incremental input to the motor of the adjacent rotatable joint, which receives both its supposed input angle and the incremental input generated by the previous joint.

Specifically, it is realized by connecting cylinders and tubes with the ANS. We connect the cylinders by tubes and solenoid valves to switch the fluid connection patterns. Based on this concept, we introduce the implementation method of proposing fascia-inspired constraints using hardware structure instead of software, as shown in Fig. 16.

By exploiting the fluid transmission among the chambers, the rotation angle of one joint can be transmitted to another using the fluid linkage-pulley-tendon mechanism. Each rotatable joint is driven by a motor, and it is programmed to respond to joint synergy-based commands. To simplify the mechanism description, we use motors to rotate the joints here, as the simplest mechanism of generating joint rotations. However, it can be replaced by any form of mechanism that can generate a rotation angle, for example, linear actuators [28].

Note that in the illustrated mechanism, for the pulley on the transmitted joint, of the motor on the pulley (the orange joint), the motor’s stator is attached to the pulley. When the pulley rotates, it consequently induces the rotation on the stator.

Controlling different constraint conditions can be hardware achieved by setting different connection patterns. A minimal system showing switching 2 patterns is shown in Fig. 17, with a valve control strategy shown in Table. II.

Figure 17. The method of controlling different connection patterns. The figure shows a minimal system of switching 2 patterns. This figure shows a case for a joint angle transmitted from the upper vertical cylinder to the lower horizontal cylinder. As the sectional areas $A$ of different chambers of the cylinders can be configured by adjusting the piston diameter, thus, the $w$ in $T_k$ be adjusted.

Also, the $w$ in $\mathbf{T_K}$ can be hardware represented as a proportion of cylinder segment area ratio $\omega$ by changing the diameter of the chamber piston. This is shown in Fig. 17 and Table II. $\omega$ can be changed by configuring $A$ of each chamber of the cylinders.

As the proposed mechanism that realizes the constraint is based on hardware, it can realize closer human-like movements by only using the control input of the joint synergy.

5.2. Limitations

When implementing the proposed hardware structure, as the complexity of $\mathbf{T_k}$ and the number of activation conditions, the mechanism will become more complex because of the increase of cylinders, valves, and pulleys. When designing the hardware, the trade-off between hardware complexity and control complexity must be considered.

One more thing that needs to be mentioned is that as the hardware becomes more complex, there will be other physical factors that affect the actual performance of the robot, e.g., the gravity issue. Therefore, a gravity compensation mechanism [29, 16] needs to be further studied. In this research, we only consider the whole-body kinematics, the dynamics of a robot system are not considered.

Table II. The pattern switching method by manipulating the state of the 2-port valve and 4-port valve. By changing the valve states, the connection pattern can be adjusted; by pre-configuring the section area $\mathrm{A}$ inside the cylinders, the $\unicode{x03C9}$ can be changed, meaning that the $\mathrm{w}$ in $\mathrm{T}_{\mathrm{k}}$ can be hardware realized.

We have investigated the specific effects that may arise from these additional optimization parameters of the proposing fascia constraint, including the connection pattern $\mathbf{T_k}$ . The configuration of the threshold will be discussed for the future feature study of the model. While the error indicates better human movements of the proposed method, it is acknowledged that the error may not fully capture the features of human-like motion. when evaluating the human-likeness using an actual robot, more quantitative evaluation indicators need to be used to study the natural movements from multiple aspects.

6. Conclusion

This research introduces a fascia-inspired nonlinear constraint into the joint synergy of humanoid robots, specifically for those with multiple DOFs. Drawing inspiration from the human fascia network, our constraint model is built to facilitate human-like movements in humanoid robots in a simple control manner of switching the connection pattern when keeping a reduced number of control inputs.

In this study, we integrated fascia-inspired constraints into joint synergy control of a 6-DOF humanoid robot model, enabling robots with multiple DOFs with fewer control inputs to resemble human-like movements more closely. Our experiments have shown that for the movements generated with the proposed constraints, lower average error, median error and their corresponding error distribution are obtained. It proves that the proposed model can generate human movements better than the joint synergy model, confirming the effectiveness of our approach in enhancing humanoid movements based on the joint synergy method. Besides the theoretical implementation, demonstration, and validation of the proposed constraints, we also outlined one possible method for applying our findings to actual robotic systems, which is crucial for transitioning from theory to partial application in humanoid robotics.

Our future work will include actual robot experiments considering the effects of other physical factors such as implementing a gravity compensation mechanism on the robot and a broader exploration of the varied conditions of the fascia constraint model, including asymmetrical motions across multiple planes. We are optimistic about the potential impact on the development of more advanced humanoid robots, especially in terms of achieving more natural movements within existing mechanical limitations.