Online robust self-learning terminal sliding mode control for balancing control of reaction wheel bicycle robots

Abstract

This paper proposes an online robust self-learning terminal sliding mode control (RS-TSMC) with stability guarantee for balancing control of reaction wheel bicycle robots (RWBR) under model uncertainties and disturbances, which improves the balancing control performance of RWBR by optimising the constrained output of TSMC. The TSMC is designed for a second-order mathematical model of RWBR. Then robust adaptive dynamic programming based on an actor-critic algorithm is used to optimise the TSMC only by data sampled online. The system closed-loop stability and convergence of the neural network weights are guaranteed based on the Lyapunov analysis. The effectiveness of the proposed algorithm is demonstrated through simulations and experiments.

1. Introduction

In recent years, there has been growing interest in the research of agile and high-speed mobile robots designed for rugged or narrow terrain [1–4]. Among these, bicycle robots have emerged as a promising platform due to their ability to achieve high-speed locomotion and agile manoeuvres on varied terrains. Reaction wheel bicycle robots (RWBR) are a type of bicycle robot that relies on reaction wheels as auxiliary balancing mechanisms. Compared to other auxiliary balancing mechanisms, such as control moment gyroscopes [5, 6] and mass pendulums [7, 8], reaction wheels offer advantages such as simple mechanism design and rapid response [9, 10].

Previous studies have investigated the effects of strategies on the RWBR balancing control. The proportional-integral-derivative (PID) control was designed to stabilise the roll angle [11]. Linear quadratic regulator (LQR) controller was used to achieve balancing control by approximating the linearisation around the equilibrium point [12]. The control of RWBR presents significant challenges, particularly in dealing with inherent uncertainties and disturbances. Traditional control methods often struggle to address these complexities effectively, leading to suboptimal performance and limited adaptability. To address these problems, various robust control strategies were proposed to balance the RWBR, such as robust LQR [13] and disturbance observers [14]. Sliding mode control (SMC) has an excellent ability to deal with uncertainties [15–17], which has been developed for balancing control of RWBR [18–20]. However, the robustness of the sliding mode controller to uncertainties typically comes at the cost of conservative control performance. This trade-off between robustness and control performance remains an open problem.

Many researchers have been striving to combine SMC with other methods to tackle this challenge, such as fuzzy control [18], adaptive control [21] and reinforcement learning [22–24]. A fuzzy sliding mode controller was designed to deal with impulse disturbance and system uncertainty in [18], but the determination of fuzzy rules was rather complicated. In [21], an adaptive sliding mode controller was proposed, which dynamically adjusts the parameters of the sliding mode controller to optimise the performance of the control. This work only make monotonic adjustments in certain scenarios, which may lead to excessively high system gain and more severe chattering. Our previous work has confirmed that reinforcement learning can improve the control performance of the SMC online [22, 23], while this combination cannot provide sufficient theoretical stability guarantee.

Adaptive dynamic programming (ADP) algorithm, a kind of reinforcement learning technique, has been used to address various optimal control problems [25–29]. It not only improves control performance while maintaining robustness but also provides theoretical stability guarantee. The linear controller with the offline ADP algorithm was proposed to balance a bicycle robot in [25]. The online ADP algorithm was studied to deal with the optimal control problem with known dynamics in [28]. Ref. [26] proposed a method to adjust the sliding mode controller of ADP online to optimise the trajectory tracking of mobile robots. However, its online optimisation was based on the prediction of the states of the nominal model, which greatly limits the applicability under uncertainty. In order to directly utilise online data for ADP solutions, researchers have conducted a significant amount of work, which has led to the developments of two main methods. One involves using the model obtained from online data fitting for online prediction [30]. The other directly uses online data to optimise the controller, including integral reinforcement learning [31] and robust adaptive dynamic programming (RADP) [27].

To address above problems, we introduce RADP to optimise the TSMC online for balancing control of RWBR. First, the nonlinear dynamics of RWBR with uncertainties and disturbances are established and the terminal sliding mode controller is set. Then, the problem of optimising the TSMC with stability constraints is formulated. An online actor-critic-based RADP algorithm is proposed to solve optimal control problems. The stability and convergence of the proposed control strategy are proven. The algorithm comparison in simulation demonstrates the advantages of the proposed control strategy. Prototype experiments also validate the control strategy. The main contributions of this paper are summarised as follows.

• • An online robust actor-critic-based RADP algorithm with robust self-learning terminal sliding mode control

  (RS-TSMC) is proposed to optimise the control performance while maintain the robustness of balancing controller for RWBR. The optimisation process is directly based on data collected online without the need for system dynamics.

• • The controller optimisation problem is transformed into solving the Hamilton–Jacobi–Bellman (HJB) equation, and the system output generated by ADP is constraint according to the range of TSMC parameters. Compared to [26], this mechanism improves the conditions for solving the constrained HJB equation, providing a more flexible and adaptable strategy for designing control strategies.

• • Experimental studies conducted in simulation platform and on a prototype RWBR compared with several recently proposed control strategies show the effectiveness of the algorithm proposed in this paper.

The rest of the paper is organised as follows. The dynamics of the RWBR and the problem formulation are given in Section 1. The online self-learning sliding mode control strategy is proposed with the stability and convergence proof in Section 3. In Section 4, various simulation experiments are performed, and the experimental results for a RWBR prototype are presented. The conclusion is addressed in Section 5. The video of the simulation and the experiments of RWBR prototype are available at the following website: https://github.com/ZhuXianjinGitHub/RSTSMC. (accessed on 30 August 2024).

Throughout the paper, $\left \| \cdot \right \|$ denotes the Euclidean norm, $ \mathrm{diag}\left \{ \cdot \right \}$ represents a diagonal matrix, and $ \otimes$ denotes the Kronecker product.

2. Problem formulation

In this section, the dynamic model of RWBR with uncertainty and disturbance is derived. We also introduce the feedback transformation. In addition, a TSMC is designed. Furthermore, the online optimisation problem for this controller is presented.

2.1. Dynamics model of RWBR

Figure 1 presents the prototype of RWBR, while Figure 2 shows notations. It can be seen that the RWBR consists of five parts, including a rear wheel, body frame, reaction wheel, handlebar and a front wheel (simplified as $R$ , $B$ , $W$ , $H$ and $F$ , respectively) in Figure 2. The details of the notation are shown in Table I.

Figure 1. Side view of the RWBR prototype.

Figure 2. Notations of the RWBR.

Table I. Diagram of bicycle structure.

Following [23], the roll dynamics of the RWBR is presented as follows:

(1) \begin{equation} \begin{aligned} J\ddot{\varphi }+I_2\ddot{\theta }-Mg\sin\! \left ( \varphi \right ) = d_1\\[3pt] I_2\ddot{\varphi }+I_2\ddot{\theta }=\tau +d_2 \end{aligned} \end{equation}

where $J=m_1l_{1}^{2}+m_2l_{2}^{2}+I_1+I_2$ , $M=m_1I_1+m_2I_2$ , $d_1$ and $d_2$ represent unmodelled dynamics and uncertainty.

To make full use of the known dynamics of the system, the dynamics parameters are divided into a nominal part and an uncertainty part.

(2) \begin{equation} \begin{aligned} \varDelta J=\left | J-J_N \right |\lt \overline{\varDelta J}\\[3pt] \varDelta I_2=\left | I_2-{I_2}_N \right |\lt \overline{\varDelta I_2}\\[3pt] \varDelta M=\left | M-M_N \right |\lt \overline{\varDelta M} \end{aligned} \end{equation}

where $J_N$ , ${I_2}_N$ and $M_N$ are the nominal parameter values, $\overline{\varDelta J}$ , $\overline{\varDelta I_2}$ and $\overline{\varDelta M}$ are the upper bounds of the uncertainties $\varDelta J$ , $\varDelta I_2$ and $\varDelta M$ .

Further, equation (1) can be re-written as

(3) \begin{align} \left ( J_N-I_{2N} \right ) \ddot{\varphi }-M_Ng\sin\!( \varphi) =-\tau +d_{1N}-d_{2N} \nonumber\\[-8pt] \end{align}

where $d_{1N}=d_1+\varDelta Mg\sin\!(\varphi) -\varDelta J\ddot{\varphi }-\varDelta I_2\ddot{\theta }$ and $d_{2N}=d_2-\varDelta I_2\ddot{\varphi }-\varDelta I_2\ddot{\theta }$ .

2.2. Design of TSMC controller

For the controller design, we first define $\varphi _d$ as the reference roll angle. The $\varphi _d$ , $\dot{\varphi }_d$ and $\ddot{\varphi }_d$ can be obtained as shown in our previous work [23]. Based on the Olfati–Saber transformation mentioned in [33], the following state variables and the feedback transformation are classified.

(4) \begin{equation} \begin{aligned} \dot{x}_1=x_2\\[3pt] \dot{x}_2=u+d^* \end{aligned} \end{equation}

where $x_1=\varphi -\varphi _d$ , $x_2=\dot{\varphi }-\dot{\varphi }_d$ , $u=\frac{I_{2N}}{\left ( J_N-I_{2N} \right )}M_Ng\sin \left ( x_1 \right ) -\ddot{\varphi }_d-\frac{I_{2N}}{\left ( J_N-I_{2N} \right )}\tau$ and $d^*=\frac{I_{2N}}{\left ( J_N-I_{2N} \right )}\left ( d_{1N}-d_{2N} \right )$ .

Assumption 1. Assuming $d_1$ and $d_2$ are bounded, it is can be get that $d_{1N}$ and $d_{2N}$ are bounded. Then, it is can be easily proved that $d^*$ is bounded. Consider that $\left | d^* \right |\lt L$ , and note that L is an unknown constant.

The sliding mode surface $s$ , the equivalent control $u_{eq}$ and the reaching control $u_r$ of TSMC are designed according to [32]. The fractional-order terminal attractor replaces the sign item in the classical sliding mode controller, which is beneficial to attenuate chattering.

(5) \begin{equation} \begin{aligned} s=x_2+\alpha _0x_1+\beta _0x_{1}^{q_0/p_0}\\[3pt] u_{eq}=-\left ( \alpha _0\dot{x}_1+\beta _0\frac{d}{dt}x_{1}^{q_0/p_0} \right ) \\[3pt] u_r=-\left ( \alpha _1s+\beta _1s^{q_1/p_1} \right ) \\[3pt] u_{tsmc}=u_{eq}+u_r \end{aligned} \end{equation}

where $\alpha _i\gt 0$ , $\beta _i\gt 0$ , $q_i$ and $p_i$ $\left ( q_i\lt p_i \right )$ $\left ( i=0,1 \right )$ are positive odd integers.

By selecting appropriate gains, the system will converge to the sufficiently small neighbourhood of the system equilibrium in finite time. According to [32], $\beta _1=\frac{L}{\left | s^{q_1/p_1} \right |}+\gamma$ and $\gamma \gt 0$ , the sliding mode variable will reach the neighbourhood $\left | s \right |\lt \left ( \frac{L}{\beta _1} \right ) ^{p_1/q_1}$ of the equilibrium in finite time $t_s$ .

(6) \begin{equation} t_s=\frac{p_1}{\alpha _1\left ( p_1-q_1 \right )}ln\frac{\alpha _1s\left ( 0 \right ) ^{\left ( p_1-q_1 \right ) /p_1}+\gamma }{\gamma } \end{equation}

Then, define $\xi _s=\left | \left ( \frac{L}{\beta _1} \right ) ^{p_1/q_1} \right |\lt L\prime$ ,

(7) \begin{equation} \dot{x}_1=-\alpha _0x_1-\beta _0x_{1}^{q_0/p_0}+L\prime \end{equation}

the system state $x_1$ will converge to the sufficiently small neighbourhood $\left | x_1 \right |\lt \left ( \frac{L\prime }{\beta _0} \right ) ^{p_0/q_0}$ of the system equilibrium in finite time $t_{x_1}$ the system equilibrium in finite time with $\beta _0=\frac{L\prime }{\left | x_{1}^{q_0/p_0} \right |}+\gamma \prime$ , $\gamma \prime \gt 0$ .

(8) \begin{equation} t_{x_1}=\frac{p_0}{\alpha _0\left ( p_0-q_0 \right )}ln\frac{\alpha _0x_1\left ( 0 \right ) ^{\left ( p_0-q_0 \right ) /p_0}+\gamma \prime }{\gamma \prime } \end{equation}

Remark 1. The parameters $\alpha _1$ and $\beta _1$ influence the reaching process of sliding mode variables. The larger parameters can reduce the time required for convergence and improve the robustness of the controller to uncertainties, while the burden of the actuator is increased and the performance of the controller is more conservative. In this paper, the RADP is introduced to online tune parameters $\alpha _1$ and $\beta _1$ of the TSMC controller (5) with constraints $\kappa =\left [ \varDelta \alpha _1,\varDelta \beta _1 \right ] ^T$ . The main motivation is to improve the control performance while maintain stability and robustness.

Assumption 2. Assuming $\kappa \in \mathcal{K} =\left \{ \kappa _{i\min }\leqslant \kappa _{i}\leqslant \kappa _{i\max } \right \}$ , $\left ( i=1,2 \right )$ . $\mathcal{K}$ is set to guarantee the finite-time convergence. $\mathcal{K}$ and $L$ generally can be obtained through experiments. And the the stability proof is given in [26].

3. Online robust self-learning TSMC

In this section, an online robust self-learning TSMC for RWBR is proposed to improve the control performance and retain the robustness. First, the optimal control problems with stability constraints are formulated. Then, an online actor-critic-based RADP algorithm is designed to approximate the HJB solutions.

Define $u_{adp}$ as the self-learning part of the control, the output of the controller as follows:

(9) \begin{equation} \begin{aligned} u=u_{tsmc}+u_{adp}\\[3pt] \left [ \kappa _{1\min },\kappa _{2\min } \right ] \zeta \lt \left | u_{adp} \right |\lt \left [ \kappa _{1\max },\kappa _{2\max } \right ] \zeta \\[3pt] \end{aligned} \end{equation}

where $\zeta =\left [ s,s^{q_1/p_1} \right ] ^T$ .

Taking (9) into (4), the system can be written as

(10) \begin{align} \dot{X}=AX+Bu+D \nonumber\\[-12pt] \end{align}

where $X=\left [ \begin{array}{c}x_1\\[3pt] x_2\\[3pt] \end{array} \right ]$ , $A=\left [ \begin{matrix}0& 1\\[3pt] 0& 0\\[3pt] \end{matrix} \right ]$ , $B=\left [ \begin{array}{c}0\\[3pt] 1\\[3pt] \end{array} \right ]$ and $D=\left [ \begin{array}{c}0\\[3pt] d^*\\[3pt] \end{array} \right ]$ .

The optimal problem is considered to be solved by minimising the value function $V_c$ to obtain the optimal policy function $u$ . $V_c$ is defined as

(11) \begin{equation} V_c=\int _0^{\infty }{\left ( X^TQX+r\left ( u_{tsmc}+u_{adp} \right ) ^2 \right ) dt},X\left ( 0 \right ) =X_0 \end{equation}

where $Q$ is symmetric positive definite matrices and $r$ is a positive constant. Taking the derivative of (11) along the trajectory of (10), the following Hamiltonian function can be obtained

(12) \begin{equation} H=V_{cX}^{T}\dot{X}+X^TQX+r\left ( u_{tsmc}+u_{adp} \right ) ^2 \end{equation}

where $V_{cX}=\frac{\partial V_c}{\partial X}$ . Define $V_{c}^{*}=\underset{U\prime }{\min }\left ( V_c \right )$ to denote the optimal value function, which satisfies

(13) \begin{equation} 0=H^*=\underset{u_{adp}}{\min }\left \{ H \right \} =V_{cX}^{*T}\dot{X}+X^TQX+r\left ( u_{tsmc}+u_{adp} \right ) ^2 \end{equation}

where $V_{cX}^{*}=\frac{\partial V_{c}^{*}}{\partial X}$ . Assuming the minimum of (13) exists and is unique, then we can obtain the optimal control policy $u_{adp}^{*}=\underset{u_{adp}}{arg\min }\left \{ H \right \}$ by $\frac{\partial H}{\partial u_{adp}}=0$ , which is described as

(14) \begin{equation} u_{adp}^{*}=-\frac{1}{2r}V_{cX}^{*T}B-u_{tsmc} \end{equation}

Taking (14) into (13),

(15) \begin{equation} 0=V_{cX}^{*T}\dot{X}+X^TQX+r\left ( -\frac{1}{2r}V_{cX}^{*T}B-u_{tsmc} \right ) ^2 \end{equation}

Traditionally, (15) is difficult to get the solution directly. The policy iteration algorithm [34] is adopted to iteratively solve in traditional ADP by the following two steps:

a) given $u^{\left ( i \right )}$ , solve for the $V_{c}^{\left ( i \right )}$ using

(16) \begin{equation} \begin{aligned} 0=V_{cX}^{\left ( i \right ) T}\dot{X}+X^TQX+r\left ( u_{tsmc}+{u_{adp}}^{\left ( i \right )} \right ) ^2\\[3pt] V_{c}^{\left ( i \right )}\left ( 0 \right ) =0 \end{aligned} \end{equation}

b) update the control policy using

(17) \begin{equation} u_{adp}^{\left ( i+1 \right )}=-\frac{1}{2r}V_{cX}^{\left ( i \right ) T}B-u_{tsmc} \end{equation}

where $i=1,2,\cdots$ denotes the iterations. When $i\rightarrow \infty$ , then $V_c\rightarrow V_{c}^{*}$ , $u_{adp}\rightarrow u_{adp}^{*}$ .

It can be seen that the system dynamic is needed in (16) to get $\dot{X}$ . When there is a certain deviation between the nominal model of the system and the actual scene, the optimisation effect based on the nominal model of the system may be affected. In this paper, RADP [27] is used to solve the optimal control problem only by data sampled online.

Consider an arbitrary control input $u=u_{tsmc}+u_s$ and differentiate the value function $V_{c}^{\left ( i \right )}$ .

(18) \begin{equation} \begin{aligned} \dot{V}_{c}^{\left ( i \right )}=V_{cX}^{\left ( i \right ) T}\left ( AX+B\left ( u_{tsmc}+{u_{adp}}^{\left ( i \right )} \right ) +B\left ( u_s-{u_{adp}}^{\left ( i \right )} \right ) \right ) \\[3pt] =-2r\left ( u_{tsmc}+u_{adp}^{\left ( i+1 \right )} \right ) \left ( u_s-{u_{adp}}^{\left ( i \right )} \right ) -X^TQX-r\left ( u_{tsmc}+{u_{adp}}^{\left ( i \right )} \right ) ^2 \end{aligned} \end{equation}

Integral (18) over an arbitrary interval as follows,

(19) \begin{equation} \begin{aligned} V_{c}^{\left ( i \right )}\left ( X_t \right ) -V_{c}^{\left ( i \right )}\left ( X_{t-T} \right ) = \\[3pt] -\int _{t-T}^t{\left ( 2r\left ( u_{tsmc}+u_{adp}^{\left ( i+1 \right )} \right ) \left ( u_s-{u_{adp}}^{\left ( i \right )} \right ) +X^TQX+r\left ( u_{tsmc}+{u_{adp}}^{\left ( i \right )} \right ) ^2 \right ) d\tau } \end{aligned} \end{equation}

The closed-loop stability of the system is ensured by (9). $V_{c}^{\left ( i \right )}$ and the improved policy $u_{adp}^{\left ( i+1 \right )}$ can be obtained in one calculation, and it does not need knowledge of the system dynamics.

The value function and the policy function are defined as neural network (NN),

(20) \begin{align} \hat{V}_{c}^{*}\left ( X \right ) =\hat{W}_{c}^{T}\phi \left ( X \right ) \end{align}

(21) \begin{equation} \hat{u}_{adp}^{*}\left ( X \right ) =\hat{W}_{a}^{T}\varphi \left ( X \right ) \end{equation}

After inserting into (19),

(22) \begin{equation} \begin{aligned} \epsilon \left ( t \right ) =\hat{W}_{c}^{T}\left ( \phi \left ( x_t \right ) -\phi \left ( x_{t-T} \right ) \right ) + \\[3pt] \int _{t-T}^t{\left ( 2r\left ( u_{tsmc}+\hat{W}_{a}^{T}\varphi \left ( X \right ) \right ) \left ( u_s-\hat{W}_{a}^{T}\varphi \left ( X \right ) \right ) +X^TQX+r\left ( u_{tsmc}+\hat{W}_{a}^{T}\varphi \left ( X \right ) \right ) ^2 \right ) d\tau } \end{aligned} \end{equation}

Under the gradient descent method, the updating laws for the weights of the critic NN and the actor NN as follows,

(23) \begin{equation} \dot{\hat{W}}_c=-\lambda _1\frac{\phi \left ( x_t \right ) -\phi \left ( x_{t-T} \right )}{m_{s}^{2}\left ( t \right )}\epsilon \left ( t \right ) \end{equation}

(24) \begin{equation} \dot{\hat{W}}_a=-\lambda _2\frac{\eta \left ( t \right )}{m_{s}^{2}\left ( t \right )}\epsilon \left ( t \right ) \end{equation}

where $\eta \left ( t \right ) =2\int _{t-T}^t{\left ( \left ( ru_s \right ) \otimes \varphi \left ( x \right ) \right ) d\tau }-\int _{t-T}^t{\left ( \varphi \left ( x \right ) \otimes r \right ) \otimes \varphi \left ( x \right ) d\tau }\mathbf{vec}\left ( \hat{W}_{a}^{T} \right )$ , $m_s = \left(\phi(x_t) -\phi(x_{t-T})\right)^T \left ( \phi \left ( x_t \right ) -\phi \left ( x_{t-T} \right ) \right ) +\eta ^T\eta +1$ and $m_s$ is used for normalization.

Remark 2. The differences between RS-TSMC proposed in this paper and self-TSMC (S-TSMC) in [26] and R-TSMC (rosbust-TSMC) in [27] are listed as follows. First, the optimisation process in S-TSMC is based on the state prediction of the nominal model, which is not conducive to the online application of the algorithm. To address this problem, this paper employs an iterative form of RADP to optimise TSMC using online data. Second, the optimisation in S-TSMC is performed directly for the state variable $s$ , which is not exactly equivalent to the optimisation for the state $X$ . The optimisation objective in R-TSMC considers only the part of the $u_{adp}$ and not the overall output of the controller. However, the optimisation is directly based on the system state and controller output in RS-TSMC. Third, the optimisation solution in S-TSMC is performed provided that the constraints of the HJB equations have a solution, whereas R-TSMC does not consider the constraints, but RS-TSMC first solves the unconstrained problem of the HJB and subsequently constrain the controller outputs.

The proposed control strategy schemes are illustrated in Algorithm 1 and Figure 3. The stability and the convergence of the proposed control strategy are given in the Appendix.

Algorithm 1. Online robust self-learning TSMC for RWBR

Figure 3. Schematic of control system.

4. Simulations and experiments

4.1. Simulations

In order to demonstrate the effectiveness of the RS-TSMC controllers proposed in this paper, two cases built in a simulation platform shown in Figure 4 as one of our previous works [23]. And two recently developed methods: S-TSMC [26] and R-TSMC [27] are used for comparison. The other simulation factors are the same except for the distinction mentioned in Remark 2. The RWBR is placed on a curved pavement with white noise. The nominal parameters of RWBR are $J_N=0.0368$ , ${I_2}_N=0.0035$ and $M_N=0.2544$ . The true parameters of RWBR used for simulation are $J=0.033$ , $I_2=0.0040$ and $M=0.2742$ . The control period of the controllers is 0.01s. The other parameters are given as follows:

Figure 4. Simulation environment of RWBR in Matlab Simscape.

(25) \begin{equation} \begin{aligned} Q=\mathrm{diag}\left \{ 1,1 \right \}, r=1\\[3pt] \alpha _0=3, \alpha _1=2, \beta _0=1, \beta _1=1, q_0=q_1=17\ and\ p_0=p_1=19\\[3pt] \kappa _{1\max }=\kappa _{2\max }=0.8, \kappa _{1\min }=\kappa _{2\min }=-0.8\\[3pt] \lambda _1=0.2, \lambda _2=0.1 \end{aligned} \end{equation}

The activation functions of the critic NN and the actor NN are considered as

(26) \begin{equation} \begin{aligned} \phi \left ( X \right ) =\left [ \begin{array}{l}x_{1}^{2}, x_{2}^{2}, x_1x_2, x_{1}^{4}, x_{2}^{4}, x_{1}^{2}x_{2}^{2}\\[3pt] \end{array} \right ] ^T\\[3pt] \varphi \left ( X \right ) =\left [ \begin{array}{l}x_1, x_2, x_{1}^{2}, x_{2}^{2}, x_1x_2, x_{1}^{4}, x_{2}^{4}, x_{1}^{2}x_{2}^{2}\\[3pt] \end{array} \right ] ^T \end{aligned} \end{equation}

An overturning moment $d_2$ is added to the system of RWBR. In case 1,

(27) \begin{equation} d_2=0.02\sum _j{\sin \left ( jt \right )} \end{equation}

where $j=\left [ \begin{array}{l}1, 3, 7, 11, 13, 15\\[3pt] \end{array} \right ]$ . In case 2,

(28) \begin{equation} d_2=0.02\sum _j{\sin \left ( jt \right )}+\begin{cases} 0.2,t\in \left [ 10,12 \right ) \cup t\in \left [ 30,32 \right )\\[3pt] -0.2,t\in \left [ 20,22 \right ) \cup t\in \left [ 40,42 \right )\\[3pt] 0,else\\[3pt] \end{cases} \end{equation}

To clearly demonstrate the superiority of the proposed method, $V_c$ defined in (11) are used to quantitatively estimate the performance, which are shown in Table II. As seen in this table, RS-TSMC reduced the criteria by $39.79\%$ in case 1 and by $15.91\%$ in case 2 to TSMC. It is less than the other two recently developed methods (R-TSMC, S-TSMC), which implies that the proposed method can achieve better control performance with less control effort. Then, details of the simulations of the two cases are discussed.

Table II. Assessment of control performance under different cases.

The simulation results of Case 1 are demonstrated in Figure 5 and Figure 6. Figure 5 gives the norms $\left \| \hat{W}_c \right \|$ and $\left \| \hat{W}_a \right \|$ with respect to the time under RS-TSMC. As shown in Figure 5, $\left \| \hat{W}_c \right \|$ converges after 12 s, and $\left \| \hat{W}_a \right \|$ converges after 20 s. Figure 6 gives the states, the control output, and $V_c$ of four methods. As can be seen, the proposed method has the smallest value of $V_c$ among the four controllers. In sum, it can be concluded that the control performance of the proposed method (RS-TSMC) outperforms the other three methods, which illustrates the superiority of the proposed method.

Figure 5. $\| \hat{W}_c\|$ and $\| \hat{W}_a \|$ with respect to the time under RS-TSMC in case 1.

Figure 6. The states, output and $V_c$ with respect to the time under four different algorithms in case 1.

Figure 7. $\| \hat{W}_c \|$ and $\| \hat{W}_a \|$ with respect to the time under RS-TSMC in case 2.

Figure 8. The states, output and $V_c$ with respect to the time under four different algorithms in case 2.

Figure 7 gives the norm $\left \| \hat{W}_c \right \|$ and $\left \| \hat{W}_a \right \|$ with respect to the time under RS-TSMC. The pulse perturbation has a significant effect on $\left \| \hat{W}_c \right \|$ at 10 s. The $\left \| \hat{W}_a \right \|$ shows regular changes with the pulse disturbance, indicating the regulation effect of the online learning algorithm on the controller output. Figure 8 illustrates the simulation results in Case 2. Similarly, we can conclude that the better control performance is reached and the less control effort is needed with the proposed method in this case.

4.2. Experiments

The RWBR prototype is used to verify the effectiveness of the proposed controller in this subsection. We presented the experiment results of the proposed RS-TSMC controller. We also performed TSMC, R-TSMC and S-TSMC for performance comparisons, which can be found in Figure 9. In the experimental studies, the TSMC algorithm works on ESP32 control board at 50 Hz and the optimising algorithm works on a PC at 25 Hz. Wireless data transmission between ESP32 and PC is achieved via UDP communication protocol. We consider the swing of the handlebars to generate disturbances for the control of the roll angel. The other settings are the same as in the simulations.

Figure 9 demonstrates the experimental results. Within the first 10 s, it can be seen that the $V_c$ of the three optimisation algorithms is slightly higher than that in TSMC, which can also be seen from the curves of $x_1$ , $x_2$ and $\tau$ . The reasons may be as follows: 1) The experimental factors such as initial roll Angle and initial roll angular velocity of RWBR are not completely consistent in different experiments. 2) The processing power of RWBR and PC is limited. With the iterative optimisation of the controller, it is only after 15 s that the three optimisation algorithms gradually outperform TSMC. The main reason lies in the fact that the control period in the RWBR prototype is much lower than that in the simulation experiment. In addition, it is not difficult to find that RS-TSMC almost outperforms the other two optimisation algorithms throughout the experiment. The proposed controller (RS-TSMC) reduced the criteria by 21.79 $\%$ , while R-TSMC and S-TSMC reduced by about 10 $\%$ to TSMC. The experimental results also validate the effectiveness and feasibility of the proposed control strategy.

Figure 9. The states, output and $V_c$ with respect to the time under four different algorithms of RWBR prototype.

5. Conclusions

This paper proposes an online RS-TSMC with stability guarantee for balancing control of RWBR under uncertainties, which improves the balancing control performance of RWBR by optimising the constrained output of TSMC. The robust adaptive dynamic programming (RADP) is used to optimise the TSMC only based on data sampled online without system dynamic. The constraint on the parameters of the sliding mode controller is utilised to derive the constraint on the control output at each time step to maintain the stability of the closed-loop system. Experimental studies conduct a simulate platform and on a prototype RWBR compared with several recently proposed control strategies show the effectiveness of the algorithm proposed in this paper.