Nonexistence of anti-symmetric solutions for fractional Hardy–Hénon system

Abstract

We study anti-symmetric solutions about the hyperplane $\{x_n=0\}$ for the following fractional Hardy–Hénon system:

\[ \left\{\begin{array}{@{}ll} (-\Delta)^{s_1}u(x)=|x|^\alpha v^p(x), & x\in\mathbb{R}_+^n, \\ (-\Delta)^{s_2}v(x)=|x|^\beta u^q(x), & x\in\mathbb{R}_+^n, \\ u(x)\geq 0, & v(x)\geq 0,\ x\in\mathbb{R}_+^n, \end{array}\right. \]

where $0< s_1,s_2<1$, $n>2\max \{s_1,s_2\}$. Nonexistence of anti-symmetric solutions are obtained in some appropriate domains of $(p,q)$ under some corresponding assumptions of $\alpha,\beta$ via the methods of moving spheres and moving planes. Particularly, for the case $s_1=s_2$, one of our results shows that one domain of $(p,q)$, where nonexistence of anti-symmetric solutions with appropriate decay conditions at infinity hold true, locates at above the fractional Sobolev's hyperbola under appropriate condition of $\alpha, \beta$.

1. Introduction

In this paper, we study anti-symmetric solutions about the hyperplane $\{x_n=0\}$ for the following system involving fractional Laplacian

(1.1)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s_1}u(x)=|x|^\alpha v^p(x), & x\in\mathbb{R}_+^n, \\ (-\Delta)^{s_2}v(x)=|x|^\beta u^q(x), & x\in\mathbb{R}_+^n, \\ u(x)\geq 0, v(x)\geq 0, & x\in\mathbb{R}_+^n, \\ u(x',x_n)={-}u(x',-x_n), v(x',x_n)={-}v(x',-x_n), & x=(x',x_n)\in\mathbb{R}^n, \end{array}\right. \end{equation}

where $s_1,s_2\in (0,1)$, $n>\max \{2s_1,2s_2\}$, $\mathbb {R}_+^n=\{x=(x',x_n)\in \mathbb {R}^n|x_n>0\}$ and $x'=(x_1,x_2,\ldots,x_{n-1})$.

The fractional Laplacian $(-\Delta )^s \, (0< s<1)$ is a nonlocal operator defined by

\[(-\Delta)^su(x)=C(n,s)P.V.\int_{\mathbb{R}^n}\frac{u(x)-u(y)}{|x-y|^{n+2s}}\,{\rm d}y, \]

where $P.V.$ stands for the Cauchy principal value and $C(n,s)=(\int _{\mathbb {R}^n}\frac {1-\cos \xi }{|\xi |^{n+2s}}\,{\rm d}\xi )^{-1}$ (see [2, 11]). Let

\[ L_{2s}=\left\{u:\mathbb{R}^n\to\mathbb{R}|\int_{\mathbb{R}^n}\frac{|u(x)|}{1+|x|^{n+2s}}\,{\rm d}x<{+}\infty\right\}. \]

Then for $u\in L_{2s}, (-\Delta )^su$ can be defined in distributional sense (see [34])

\[ \int_{\mathbb{R}^n}(-\Delta)^su \varphi\,{\rm d}x=\int_{\mathbb{R}^n}u(-\Delta)^s\varphi\,{\rm d}x,\quad\text{for any}\ \varphi\in \mathcal{S}.\]

Moreover, $(-\Delta )^su$ is well defined for $u\in L_{2s}\cap C_{loc}^{1,1}(\mathbb {R}^n)$. We call $(u,v)$ a classical solution of (1.1) if $(u,v)\in ( L_{2s_1}\cap C^{1,1}_{loc}(\mathbb {R}_+^n)\cap C(\mathbb {R}^n)) \times (L_{2s_2}\cap C^{1,1}_{loc}(\mathbb {R}_+^n)\cap C(\mathbb {R}^n))$ and satisfies (1.1).

As is well known, the method of moving planes and moving spheres play an important role in proving the nonexistence of solutions. Chen et al. [3, 4] introduced a direct method of moving planes and moving spheres for fractional Laplacian, which have been widely applied to derive the symmetry, monotonicity and nonexistence and even a prior estimates of solutions for some equations involving fractional Laplacian. In such process, some suitable forms of maximum principles are the key ingredients. The method of moving planes in integral forms is also a vital tool for classification of solutions (see [5]).

Recently, Li and Zhuo [20] classified anti-symmetric classical solutions of Lane–Emden system (1.1) in the case of $s_1=s_2=:s\in (0,1)$ and $\alpha =\beta =0$. They established the following Liouville type theorem.

Proposition 1.1 ([20])

Given $0< p,q\leq \frac {n+2s}{n-2s},$ assume that $(u, v)$ is an anti-symmetric classical solution of system (1.1). If $0 < pq < 1$ or $p + 2s > 1$ and $q + 2s > 1$, then $(u,v)\equiv (0,0)$.

As a corollary of proposition 1, the nonexistence results in the larger space $L_{2s+1}$ follows immediately for the case $p+2s>1$, $q+2s>1$.

Proposition 1.2 ([20])

Assume that $u$ and $v \in L_{2s+1}\cap C^{1,1}_{loc}(\mathbb {R}^n_+)\cap C(\mathbb {R}^n)$ satisfy system (1.1). Then if $0 < p, q \leq \frac {n+2s}{n-2s}$, $p+2s > 1$ and $q + 2s > 1$, $(u,v)\equiv (0,0)$ is the only solution.

The nonexistence of anti-symmetric classical solutions to the corresponding scalar problem were given in [37].

The following Hardy–Hénon system with homogeneous Dirichlet boundary conditions has been investigated widely

(1.2)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s_1}u(x)=|x|^\alpha u^p,\ u(x)\geq 0, & x\in\Omega, \\ (-\Delta)^{s_2}v(x)=|x|^\beta v^q, \ v(x)\geq 0, & x\in\Omega, \\ u(x)=v(x)=0, & x\in\mathbb{R}^n\setminus\Omega. \end{array}\right. \end{equation}

There are enormous nonexistence results of (1.2) for the case $\Omega =\mathbb {R}^n$. We list some main results as follows.

If $s_1=s_2=1$, for $\alpha,\beta \geq 0$, system (1.2) is the well-known Hénon–Lane–Emden system. It has been conjectured that the Sobolev's hyperbola

\[ \left\{p>0,q>0:\frac{n+\alpha}{p+1}+\frac{n+\beta}{q+1}=n-2\right\} \]

is the critical dividing curve between existence and nonexistence of solutions to (1.2). Particularly, the Hénon–Lane–Emden conjecture states that system (1.2) admits no nonnegative non-trivial solutions if $p>0$, $q>0$ and $\frac {n+\alpha }{p+1}+\frac {n+\beta }{q+1}>n-2$. For $\alpha =\beta =0$, this conjecture has been completely proved for radial solutions (see [23, 32]). However, for non-radial solutions, the conjecture is only fully answered when $n\leq 4$ (see [29, 33, 35]). In higher dimensions, the conjecture was partially solved. Figueiredo and Felmer [14] showed that system (1.2) admits no classical positive solutions if

\[0< p,q\leq \frac{n+2}{n-2}\quad \text{and}\quad (p,q)\neq\left(\frac{n+2}{n-2},\frac{n+2}{n-2}\right). \]

Busca and Manásevich [1] proved the conjecture if

\[ \alpha_1,\alpha_2\geq \frac{n-2}{2}\quad \text{and}\quad (\alpha_1,\alpha_2)\neq \left(\frac{n-2}{2},\frac{n-2}{2}\right), \]

where

\[ \alpha_1=\displaystyle\frac{2(p+1)}{pq-1},\quad \alpha_2=\frac{2(q+1)}{pq-1},\ pq>1. \]

When $\alpha,\beta >0$, Fazly and Ghoussoub [13] showed that the conjecture holds for dimension $n = 3$ under the assumption of the boundedness of positive solutions, Li and Zhang [22] removed this assumption and proved this conjecture for dimension $n = 3$. When $\min \{\alpha,\beta \}>-2$, the conjecture is proved for bounded solutions in $n=3$ (see [27]).

If $s_1=s_2=:s\in (0,1), \alpha, \beta \geq 0$, there are fewer nonexistence results of solutions to system (1.2) in the case of $p>0$, $q>0$ and $\frac {n+\alpha }{p+1}+\frac {n+\beta }{q+1}>n-2s$, namely the case that $(p,q)$ locates at bottom left of the fractional Sobolev's hyperbola $\left \{p>0,q>0:\frac {n+\alpha }{p+1}+\frac {n+\beta }{q+1}=n-2s\right \}$. For $\alpha =\beta =0$, Quaas and Xia in [30] proved that there exist no classical positive solutions to (1.2) provided that

(1.3)\begin{equation} \alpha^s_1,\alpha^s_2\in\left[\frac{n-2s_1}{2},n-2s_1\right),\quad \text{and}\quad (\alpha^s_1,\alpha^s_2)\neq \left(\frac{n-2s}{2},\frac{n-2s}{2}\right), \end{equation}

where

\[ \alpha_1^s=\frac{2s(q+1)}{pq-1},\quad \alpha_2^s=\frac{2s(p+1)}{pq-1},\quad p,q>0,\ pq>1. \]

Note that region (1.3) of $(p,q)$ contains the following region:

\[ \left\{(p,q):\frac{n}{n-2s}< p,q\leq \frac{n+2s}{n-2s},\quad\text{and}\quad(p,q)\neq \left(\frac{n+2s}{n-2s},\frac{n+2s}{n-2s}\right)\right\}. \]

As $\min \{\alpha,\beta \}>-2s$, Peng [26] derived that system (1.2) admits no nonnegative classical solutions if $0< p<\frac {n+2s+2\alpha }{n-2s}$ and $0< q<\frac {n+2s+2\beta }{n-2s}$.

For scalar equation (i.e. $s_1=s_2:=s, \alpha =\beta, p=q, u=v$), in the Laplacian case, if $\alpha =0$, a celebrated Liouville type theorem was showed by Gidas and Spruck [17] for $1< p<\frac {n+2}{n-2}$; if $\alpha \leq -2$ and $p>1$, there is no any positive solution (see [16, 25]); if $\alpha >-2$ and $1< p<\frac {n+2+2\alpha }{n-2}$, Phan and Souplet [28] derived a Liouville theorem for bounded solutions; if $0 < p \leq 1$, the nonexistence result was proved by Dai and Qin [8] for any $\alpha$. We also refer to [15, 24] and references therein. For the scalar fractional Laplacian case, Chen et al. [3], Jin et al. [18] proved the nonexistence results for $\alpha =0$ and $0< p<\frac {n+ 2s}{n-2s}.$ If $\alpha >-2s$, Dai and Qin [9] showed a Liouville type theorem for optimal range $0< p<\frac {n+2s+2\alpha }{n-2s}$.

For the case $\Omega =\mathbb {R}^n_+$, the following several main results exist.

If $s_1=s_2=1$, $\alpha =\beta =0$, $\min \{p,q\}>1$, a Liouville theorem is proved for bounded solutions by Chen et al. [6]. If $s_1=s_2=:s \in (0,1)$, for $\alpha =\beta =0$, the nonexistence of positive viscosity-bounded solutions to system (1.2) was showed by Quaas and Xia in [31]. For $\alpha,\beta >-2s$, if $p\geq \frac {n+2s+\alpha }{n-2s}$ and $q\geq \frac {n+2s+\beta }{n-2s}$, Duong and Le [12] obtained the nonexistence of solutions satisfying the following decay at infinity:

\[ u(x) = \text{o}\left(|x|^{-\frac{4s+\beta}{q-1}}\right)\quad\text{and}\quad v(x) = \text{o}\left(|x|^{-\frac{4s+\alpha}{p-1}}\right). \]

For general $s_1, s_2\in (0,1)$, Le in [19] concluded a Liouville type theorem. Precisely, they obtained that if $1\leq p\leq \frac {n+2s_1+2\alpha }{n-2s_2}$, $1\leq q\leq \frac {n+2s_2+2\alpha }{n-2s_1}$ and $(p,q)\neq (\frac {n+2s_1+2\alpha }{n-2s_2},\frac {n+2s_2+2\alpha }{n-2s_1}$), $\alpha >-2s_1$ and $\beta >-2s_2$, then $(u,v)\equiv (0,0)$ is the only nonnegative classical solutions to system (1.2). More nonexistence results for general nonlinearities in a half space can be seen in [10, 36]. For the corresponding scalar problem of (1.2) with Laplacian and $\alpha =\beta =0$, Gidas and Spruck [17] obtained the nonexistence of nontrivial nonnegative classical solution of (1.2) for $1< p\leq \frac {n+2}{n-2}$. For the corresponding scalar problem (1.2) with fractional Laplacian and $\alpha =\beta =0$, Chen et al. [3] showed that $u\equiv 0$ is the only nonnegative solution to (1.2) for $1< p\leq \frac {n+2s}{n-2s}$. Recently, a Liouville type theorem of the corresponding scalar problem for $1< p<\frac {n+2s+2\alpha }{n-2s}, \alpha >-2s$ and $s\in (0,1]$ was established by Dai and Qin in [9].

In this paper, we will study nonexistence of anti-symmetric classical solutions to system (1.1) for general $\alpha, \beta, p, q$.

For $\alpha >-2s_1$ and $\beta >-2s_2$, we denote

\begin{align*} \mathcal{R}_{sub}& :=\left\{(p,q)|0< p\leq \displaystyle\frac{n+2s_1+2\alpha}{n-2s_2}, 0< q\right.\\& \left.\leq \frac{n+2s_2+2\beta}{n-2s_1}, (p,q)\neq \left(\frac{n+2s_1+2\alpha}{n-2s_2},\frac{n+2s_2+2\beta}{n-2s_1}\right)\right\}. \end{align*}

Note that for the case $s_1=s_2$, the set $\mathcal {R}_{sub}$ locates at bottom left of the preceding fractional Sobolev's hyperbola.

Throughout this paper, we always assume $s_1,s_2\in (0,1)$, $n>\max \{2s_1,2s_2\}$ and use $C$ to denote a general positive constant whose value may vary from line to line even the same line. Our main results are as follows.

Theorem 1.1 For $(p,q)\in \mathcal {R}_{sub}$, assume that $(u,v)$ is a classical solution of system (1.1). For either one of the following two cases:

1. (i) $\min \{p+2s_1,p+2s_1+\alpha \}>1$ and $\min \{q+2s_2,q+2s_2+\beta \}>1$,

2. (ii) $0< pq<1$ with $\alpha \geq -2s_1pq$, $\beta \geq -2s_2pq$,

   we have that $(u,v)=(0,0)$.

The nonexistence results (i) of theorem 1 can be extended to a larger space.

Theorem 1.2 Assume that $(p,q)\in \mathcal {R}_{sub}$ and $(u,v)\in (L_{2s_1+1}\cap C^{1,1}_{loc}(\mathbb {R}^n_+)\cap C(\mathbb {R}^n))\times (L_{2s_2+1}\cap C^{1,1}_{loc}(\mathbb {R}^n_+)\cap C(\mathbb {R}^n))$ satisfies system (1.1). Then for the case that $\min \{p+2s_1,p+2s_1+\alpha \}>1$ and $\min \{q+2s_2,q+2s_2+\beta \}>1$, $(u,v)\equiv (0,0)$ is the only solution.

Combining our anti-symmetric property, in the proof of theorem 1.1, we only utilized the extended spaces $L_{2s_1+1}, L_{2s_2+1}$ instead of the usual spaces $L_{2s_1}, L_{2s_2}$ in the case that $\min \{p+2s_1,p+2s_1+\alpha \}>1$ and $\min \{q+2s_2,q+2s_2+\beta \}>1$. One can see that theorem 1.2 is a direct corollary of (i) of theorem 1.1.

Remark 1.1 Our results of theorems 1.1 and 1.2 are the extension to general $s_1,s_2,\alpha,\beta$ of the nonexistence results of Li and Zhuo [20] (see preceding propositions 1.1 and 1.2) except one critical point of $(p,q)$.

Remark 1.2 When $s_1=s_2$, $p=q$, $\alpha =\beta$ and $u=v$, the results of theorems 1.1 and 1.2 are the nonexistence of nontrivial classical solutions to the corresponding scalar problem.

Under appropriate decay conditions of $u$ and $v$ at infinity, we can extend the nonexistence result of classical solutions of (1.1) to an unbounded domain of $(p,q)$. Particularly, this unbounded domain, except at most a bounded sub-domain, locates at above the preceding fractional Sobolev's hyperbola for the case $s_1=s_2$.

Theorem 1.3 Suppose $p\geq \frac {n+2s_1+\alpha }{n-2s_2}$, $q\geq \frac {n+2s_2+\beta }{n-2s_1}$, $\alpha >-2s_2$, $\beta >-2s_1$. Assume $(u,v)$ is a classical solution of system (1.1) satisfying

\[ \mathop{\overline{\lim}}\limits_{x\to \infty}\displaystyle\frac{u(x)}{|x|^{a}}\leq C\quad \text{and}\quad \mathop{\overline{\lim}}\limits_{x\to \infty}\frac{v(x)}{|x|^{b}}\leq C, \]

for some $C>0$, where $a=-\frac {2s_1+2s_2+\beta }{q-1}$ and $b=-\frac {2s_1+2s_2+\alpha }{p-1}$. Then $(u,v)\equiv (0,0)$.

Remark 1.3 The results of theorem 1.3 are new even if for the corresponding scalar problem with $\alpha =0$.

Remark 1.4 When $\alpha,\beta$ are positive, define the region

\begin{align*} \mathcal{R}_{sup}& \,{:=}\left\{(p,q)|\frac{n+2s_1+\alpha}{n-2s_2}\leq p\leq \frac{n+2s_1+2\alpha}{n-2s_2},\frac{n+2s_2+\beta}{n-2s_1}\,{\leq}\, q\leq\frac{n+2s_2+2\beta}{n-2s_1}, \right.\\ & \quad\left. (p,q)\neq \left(\frac{n+2s_1+2\alpha}{n-2s_2},\frac{n+2s_2+2\beta}{n-2s_1}\right)\right\}. \end{align*}

Note that $\mathcal {R}_{sup}$ is contained in the nonexistence region of $(p,q)$ obtained in theorem 1.1. Hence, if $(p,q)\in \mathcal {R}_{sup}$, theorem 1.1 tells us that the results of theorem 1.3 still hold true without the decay conditions.

2. Preliminaries

In this section, we introduce and prove some necessary lemmas.

Proposition 2.1 ([12])

Let $s\in (0,1)$ and $w(y)\in L_{2s}\cap C_{loc}^{1,1}(\mathbb {R}^n)$ satisfy $w(y)=-w(y^-_\lambda )$, where $y^-_\lambda =(y', 2\lambda -y_n)$ for any real number $\lambda$. Assume there exists $x\in \Sigma _\lambda$ such that

\[ w(x)=\mathop{\inf}\limits_{\Sigma_\lambda }w(y)<0\quad\text{and}\quad (-\Delta)^sw(x)+c(x)w(x)\geq 0, \]

where $\Sigma _\lambda =\{x\in \mathbb {R}^n|x_n<\lambda \}$. Then we have the following claims:

1. (i) if

   \[\mathop{\lim\inf}\limits_{|x|\to \infty }|x|^{2s}c(x)\geq0, \]
   there exists a constant $R_0>0$ (depending on c, but independent of $w$) such that
   \[ |x|< R_0, \]

2. (ii) if $c$ is bounded below in $\Sigma _\lambda$, there exists a constant $\ell >0$ (depending on the lower bound of $c$, but independent of $w$) such that

   \[ {\rm d}(x,T_\lambda)> \ell, \]
   where $T_\lambda =\{x\in \mathbb {R}^n|x_n=\lambda \}$.

We want to point out that the constant $\ell$ is non-increasing about $\lambda$, since $\ell$ is non-decreasing about the lower bound of $c$, which can be seen from the proof of proposition 2.1 in [12].

In order to apply the method of moving planes to prove the nonexistence, we need to establish the following estimate.

Lemma 2.1 Let $s\in (0,1)$ and $w(y)\in L_{2s}\cap C_{loc}^{1,1}(\mathbb {R}^n)$ satisfy $w(y)=-w(y^-_\lambda )$. Assume there exists $x\in \Sigma _\lambda$ such that $w(x)=\mathop {\inf }\nolimits _{\Sigma _\lambda }w(y)<0$. Then we have

\begin{align*} (-\Delta)^sw(x)& \leq C(n,s)\left[\vphantom{\left(\displaystyle\frac{1}{|x-y|^{n+2s}}-\frac{1}{|x-y^-_\lambda|^{n+2s}}\right)}w(x)\,{d}^{{-}2s}\right.\\& \left.\quad+\displaystyle\int_{\Sigma_\lambda} (w(x)-w(y))\left(\displaystyle\frac{1}{|x-y|^{n+2s}}-\frac{1}{|x-y^-_\lambda|^{n+2s}}\right){\rm d}y\right],\end{align*}

where $d=d(x,T_\lambda )$ and the constant $C(n,s)$ is positive.

Proof. Applying the definition of fractional Laplacian, we have

(2.1)\begin{align} (-\Delta)^sw(x)& =C(n,s)\int_{\mathbb{R}^n}\frac{w(x)-w(y)}{|x-y|^{n+2s}}\,{\rm d}y\nonumber\\ & =C(n,s)\int_{\Sigma_\lambda}\frac{w(x)-w(y)}{|x-y|^{n+2s}}\,{\rm d}y+C(n,s)\int_{\mathbb{R}^n\setminus\Sigma_\lambda}\frac{w(x)-w(y)}{|x-y|^{n+2s}}\,{\rm d}y\nonumber\\ & =C(n,s)\int_{\Sigma_\lambda}\frac{w(x)-w(y)}{|x-y|^{n+2s}}\,{\rm d}y+C(n,s)\int_{\Sigma_\lambda}\frac{w(x)+w(y)}{|x-y^-_\lambda|^{n+2s}}\,{\rm d}y\nonumber\\ & =C(n,s)\left[\int_{\Sigma_\lambda}(w(x)-w(y))\left(\frac{1}{|x-y|^{n+2s}}-\frac{1} {|x-y^-_\lambda|^{n+2s}}\right){\rm d}y\right.\nonumber\\ & \left.\quad +\int_{\Sigma_\lambda}\frac{2w(x)}{|x-y^-_\lambda|^{n+2s}}\,{\rm d}y\right]. \end{align}

By an elementary calculation (see [7]), we derive

\[ \int_{\Sigma_\lambda}\frac{2w(x)}{|x-y^-_\lambda|^{n+2s}}\,{\rm d}y\cong C(n,s)w(x)\,{ d}^{{-}2s}. \]

Hence, combining this and (2.1), we complete the proof of lemma 2.1.

In order to apply the method of moving spheres to prove the nonexistence, we need to establish a similar estimate as that of lemma 2.1. To this end, we need to introduce some notations. For any real number $\lambda >0$, we denote

\begin{gather} S_\lambda=\{x\in\mathbb{R}^n \,|\, |x|=\lambda\},\nonumber \end{gather}

\begin{gather} B_\lambda^+{=}B_\lambda^+(0)=\{|x|<\lambda\,|\,x_n>0\}.\nonumber \end{gather}

Let $x^\lambda =\frac {\lambda ^2x}{|x|^2}$ be the inversion of the point $x=(x', x_n)$ about the sphere $S_\lambda$ and $x^*=(x', -x_n)$. Denote

\[ B_\lambda^-{=}\{x|x^*\in B_\lambda^+\},\quad (B_\lambda^+)^C=\{x|x^\lambda\in B_\lambda^+\}, \quad(B_\lambda^-)^C=\{x|x^\lambda\in B_\lambda^-\}. \]

Lemma 2.2 Let $w(x)\in L_{2s}\cap C_{loc}^{1,1}(\mathbb {R}_+^n)$ satisfy

(2.2)\begin{equation} w(x)={-}w(x^*) \text{ and } w(x)={-}\left(\frac{\lambda}{|x|}\right)^{n-2s}w(x^\lambda),\quad \forall x\in \mathbb{R}^n_+. \end{equation}

Assume there exists $\tilde {x}\in B_\lambda ^+$ such that $w(\tilde {x})=\inf _{B_\lambda ^+}w(x)<0$. Then we have

\begin{align*} (-\Delta)^sw(\tilde{x})& \leq C(n,s) \left[w(\tilde{x})\left((\lambda-|\tilde{x}|)^{{-}2s}+\frac{\delta^n}{\tilde{x}_n^{n+2s}}\right)\right.\\& \quad \left.+\int_{B_\lambda^+} (w(\tilde{x})-w(y))h_\lambda(\tilde{x},y)\,{\rm d}y\right], \end{align*}

where $h_\lambda (\tilde {x},y)=\frac {1}{|\tilde {x}-y|^{n+2s}}-\frac {1}{|\frac {|y|}{\lambda }\tilde {x}-\frac {\lambda }{|y|}y|^{n+2s}}+ \frac {1}{|\frac {|y|}{\lambda }\tilde {x}-\frac {\lambda }{|y|}y^*|^{n+2s}}-\frac {1}{|\tilde {x}-y^*|^{n+2s}}>0$ for $\tilde {x},y\in B_\lambda ^+$, $\delta =\min \{\tilde {x}_n,\lambda -|\tilde {x}|\}$ and $C(n,s)$ is a positive constant.

Proof. By the definition of fractional Laplacian and assumptions (2.2), we derive

(2.3)\begin{align} & (-\Delta)^sw(\tilde{x})\nonumber\\ & \quad=C(n,s)\int_{\mathbb{R}^n}\frac{w(\tilde{x})-w(y)}{|\tilde{x}-y|^{n+2s}}\,{\rm d}y\nonumber\\ & \quad=C(n,s)\left(\int_{B_\lambda^+}+\int_{(B_\lambda^+)^C}+\int_{B_\lambda^-}+\int_{(B_\lambda^-)^C}\right)\frac{w(\tilde{x})-w(y)}{|\tilde{x}-y|^{n+2s}}\,{\rm d}y\nonumber\\ & \quad=C(n,s)\left(\int_{B_\lambda^+}\frac{w(\tilde{x})-w(y)}{|\tilde{x}-y|^{n+2s}}\,{\rm d}y+\int_{B_\lambda^+}\frac{(\frac{\lambda}{|y|})^{n-2s}w(\tilde{x})+w(y)}{|\frac{|y|}{\lambda}\tilde{x}-\frac{\lambda}{|y|}y|^{n+2s}}\,{\rm d}y\right.\nonumber\\ & \qquad\left.+\int_{B_\lambda^-}\frac{w(\tilde{x})-w(y)}{|\tilde{x}-y|^{n+2s}}\,{\rm d}y +\int_{B_\lambda^-}\frac{\left(\frac{\lambda}{|y|}\right)^{n-2s}w(\tilde{x})+w(y)}{|\frac{|y|}{\lambda}\tilde{x}-\frac{\lambda}{|y|}y|^{n+2s}}\,{\rm d}y\right)\nonumber\\ & \quad=C(n,s)\left[\int_{B_\lambda^+}(w(\tilde{x})-w(y))h_\lambda(\tilde{x},y)\,{\rm d}y+\int_{B_\lambda^+}\frac{\left(1+\left(\frac{\lambda}{|y|}\right)^{n-2s}\right)w(\tilde{x})}{|\frac{|y|}{\lambda}\tilde{x}-\frac{\lambda}{|y|}y|^{n+2s}}\,{\rm d}y\right.\nonumber\\ & \qquad\left.+\int_{B_\lambda^+}\frac{2w(\tilde{x})}{|\tilde{x}-y^*|^{n+2s}}\,{\rm d}y +\int_{B_\lambda^+}\frac{\left(\frac{\lambda}{|y|}\right)^{n-2s}-1)w(\tilde{x})}{|\frac{|y|}{\lambda}\tilde{x}-\frac{\lambda}{|y|}y^*|^{n+2s}}\,{\rm d}y\right]. \end{align}

Using similar arguments in [21], we can obtain that $h_\lambda (x,y)>0$ for $x,y\in B_\lambda ^+$.

Furthermore, choose $r<\tilde {x}_n$ small such that $H:=\{x\in B_\delta (\tilde {x})|x_n>\tilde {x}_n\}\subset \{x\in \mathbb {R}^n|x_n>\tilde {x}_n\}\subset (B_r^+(0))^C$ where $\delta =\min \{\tilde {x}_n,\lambda -|\tilde {x}|\}$, then we calculate

(2.4)\begin{align} \int_{B_\lambda^+}\frac{\left(1+\left(\frac{\lambda}{|y|}\right)^{n-2s}\right)w(\tilde{x})}{\left|\frac{|y|}{\lambda}\tilde{x}-\frac{\lambda}{|y|}y\right|^{n+2s}}\,{\rm d}y& \leq\int_{B_r^+}\frac{\left(1+\left(\frac{\lambda}{|y|}\right)^{n-2s}\right)w(\tilde{x})}{\left|\frac{|y|}{\lambda}\tilde{x}-\frac{\lambda}{|y|}y\right|^{n+2s}}\,{\rm d}y\nonumber\\ & =\int_{(B_r^+)^C}\frac{\left(1+\left(\frac{\lambda}{|y^\lambda|}\right)^{n-2s}\right)w(\tilde{x})}{\left(\frac{|y^\lambda|}{\lambda}\right)^{n+2s}|\tilde{x}-y|^{n+2s}}\left(\frac{\lambda}{|y|}\right)^{2n}\,{\rm d}y\nonumber\\ & =w(\tilde{x})\int_{(B_r^+)^C}\frac{1}{|\tilde{x}-y|^{n+2s}}\left(1+\left(\frac{\lambda}{|y|}\right)^{n-2s}\right){\rm d}y\nonumber\\ & \leq w(\tilde{x})\int_{\{x\in\mathbb{R}^n|x_n>\tilde{x}_n\}\setminus H}\frac{1}{|\tilde{x}-y|^{n+2s}}\,{\rm d}y \nonumber\\ & \leq C(n) w(\tilde{x})\int_{\delta}^{+\infty}r^{{-}2s-1}\,{\rm d}r\nonumber\\ & \leq C(n,s)w(\tilde{x})\delta^{{-}2s}\nonumber\\ & \leq C(n,s)w(\tilde{x})(\lambda-|\tilde{x}|)^{{-}2s}. \end{align}

From the definition $\delta =\min \{\tilde {x}_n,\lambda -|\tilde {x}|\}$, we have $|\tilde {x}-y^*|< C\tilde {x}_n$ for any $y\in B_\delta ^+(\tilde {x})$. Simple calculations imply that

(2.5)\begin{equation} \int_{B_\lambda^+}\frac{2w(\tilde{x})}{|\tilde{x}-y^*|^{n+2s}}\,{\rm d}y\leq Cw(\tilde{x})\int_{B_\delta^+(\tilde{x})}\frac{1}{\tilde{x}_n^{n+2s}}\,{\rm d}y\leq C(n)w(\tilde{x})\frac{\delta^n}{\tilde{x}_n^{n+2s}}. \end{equation}

It is easy to see that

(2.6)\begin{equation} \int_{B_\lambda^+}\frac{\left(\left(\frac{\lambda}{|y|}\right)^{n-2s}-1\right)w(\tilde{x})}{\left|\frac{|y|}{\lambda}\tilde{x}-\frac{\lambda}{|y|}y^*\right|^{n+2s}}\leq 0. \end{equation}

Therefore, from (2.3)–(2.6), we conclude the proof.

Lemma 2.3 Let $\alpha,\beta >-n$. Suppose that $(u,v)$ is a nonnegative classical solution for the following system:

(2.7)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s_1}u(x)=|x|^\alpha v^p(x),\ u(x)\geq 0, & x\in\mathbb{R}_+^n, \\ (-\Delta)^{s_2}v(x)=|x|^\beta u^q(x),\ v(x)\geq 0, & x\in\mathbb{R}_+^n, \\ u(x)={-}u(x^*), \ v(x)={-}v(x^*), & x\in\mathbb{R}^n. \end{array}\right. \end{equation}

Then for $x\in \mathbb {R}^n_+$, we have

\[ \left\{\begin{array}{@{}l} u(x)\geq C\displaystyle\int_{\mathbb{R}_+^n}\left(\dfrac{1}{|x-y|^{n-2s_1}}-\dfrac{1}{|x^*-y|^{n-2s_1}}\right)|y|^\alpha v^p(y)\,{\rm d}y, \\ v(x)\geq C\displaystyle\int_{\mathbb{R}_+^n}\left(\dfrac{1}{|x-y|^{n-2s_2}}-\dfrac{1}{|x^*-y|^{n-2s_2}}\right)|y|^\beta u^p(y)\,{\rm d}y, \end{array}\right. \]

where $C$ is a positive constant.

Proof. Define a cut-off function $\eta (x)\in C_0^{\infty }(\mathbb {R}^n)$ satisfying $\eta (x)=0$ for $|x|>1$ and $\eta (x)=1$ for $|x|<\frac {1}{2}$. Denote $\eta _R(x)=\eta (\frac {x}{R})$ for large $R$ and

\begin{align*} u_R(x)& =C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right)\eta_R(y)|y|^\alpha v^p(y)\,{\rm d}y,\\ v_R(x)& =C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_2}}-\frac{1}{|x^*-y|^{n-2s_2}}\right)\eta_R(y)|y|^\beta u^q(y)\,{\rm d}y. \end{align*}

Note that $(u_R(x), v_R(x))$ is a solution for the following system:

(2.8)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s_1}u_R(x)=\eta_R(x)|x|^\alpha v^p(x), & x\in\mathbb{R}_+^n, \\ (-\Delta)^{s_2}v_R(x)=\eta_R(x)|x|^\beta u^q(x), & x\in\mathbb{R}_+^n, \\ u_R(x)={-}u_R(x^*), \ v_R(x)={-}v_R(x^*), & x\in\mathbb{R}^n. \end{array}\right. \end{equation}

Let $U_R(x)=u(x)-u_R(x)$ and $V_R(x)=v(x)-v_R(x)$. From (2.7) and (2.8), we derive

(2.9)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s_1}U_R(x)=|x|^\alpha v^p(x)-\eta_R(x)|x|^\alpha v^p(x)\geq 0, & x\in\mathbb{R}_+^n, \\ (-\Delta)^{s_2}V_R(x)=|x|^\beta u^q(x)-\eta_R(x)|x|^\beta u^q(x)\geq 0, & x\in\mathbb{R}_+^n, \\ U_R(x)={-}U_R(x^*),\ V_R(x)={-}V_R(x^*), & x\in\mathbb{R}^n. \end{array}\right. \end{equation}

By the definitions of $U_R(x)$ and $V_R(x)$, obviously, for $x\in \mathbb {R}^n_+$,

(2.10)\begin{equation} \mathop{\varliminf}\limits_{|x| \to \infty} U_R(x)\geq 0 \text{ and } \mathop{\varliminf}\limits_{|x|\to \infty} V_R(x)\geq 0, \end{equation}

where we used the assumptions $\alpha,\beta >-n$.

Next, we claim that $U_R(x)\geq 0$ and $V_R(x)\geq 0$ for $x\in \mathbb {R}_+^n$. If not, from (2.10) we know that there exists some $\hat {x}\in \mathbb {R}_+^n$ such that $U_R(\hat {x})=\mathop {\inf }\nolimits _{\mathbb {R}_+^n}U_R(x)<0$. Then,

\begin{align*} (-\Delta)^{s_1}U_R(\hat{x})& =C(n,s_1)\displaystyle\int_{\mathbb{R}^n}\frac{U_R(\hat{x})-U_R(y)}{|\hat{x}-y|^{n+2s_1}}\,{\rm d}y \\ & =C(n,s_1)\int_{\mathbb{R}_+^n}\frac{U_R(\hat{x})-U_R(y)}{|\hat{x}-y|^{n+2s_1}}\,{\rm d}y+C(n,s_1)\int_{\mathbb{R}_+^n}\frac{U_R(\hat{x}) +U_R(y)}{|\hat{x}-y^*|^{n+2s_1}}\,{\rm d}y \\ & =C(n,s_1)\left[\int_{\mathbb{R}_+^n}(U_R(\hat{x})-U_R(y))\left(\frac{1}{|\hat{x}-y|^{n+2s_1}}-\frac{1}{|\hat{x}-y^*|^{n+2s_1}}\right)\right.\\& \quad\left.+\frac{2U_R(\hat{x})} {|\hat{x}-y^*|^{n+2s_1}}\vphantom{\int_{\mathbb{R}_+^n}}\right]{\rm d}y \\ & <0. \end{align*}

This leads a contradiction with the first equation in (2.9). Thus, $U_R(x)\geq 0$ holds true for any $x\in \mathbb {R}_+^n$, that is, $u(x)\geq u_R(x)$ in $\mathbb {R}_+^n$. Letting $R \to \infty$, we obtain

\[ u(x)\geq C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right)|y|^\alpha v^p(y)\,{\rm d}y. \]

Similarly, one has

\[ v(x)\geq C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_2}}-\frac{1}{|x^*-y|^{n-2s_2}}\right)|y|^\beta u^q(y)\,{\rm d}y. \]

3. Proof of theorem 1.1

In this section, we are ready to prove theorem 1.1. For (i) of theorem 1.1, namely the case $\min \{p+2s_1,p+2s_1+\alpha \}>1$ and $\{q+2s_2,q+2s_2+\beta \}>1$, we use the method of moving spheres to derive a lower bound for $u(x)$ and $v(x)$. Then, lemma 2.3 and a ‘bootstrap’ iteration process will give the better lower-bound estimates which can imply the nonexistence result. For (ii) of theorem 1.1, namely the case $0< pq<1$, $\alpha \geq -2s_1pq$ and $\beta \geq -2s_2pq$, a direct application of lemma 2.3 and iteration technique may give its proof.

3.1 Proof of (i) of theorem 1.1

Proof. By contradiction, assume that $(u,v) \not \equiv (0,0)$, then we can derive that $u>0$ and $v>0$ in $\mathbb {R}^n_+$. Indeed, if there exists some $\hat {x}\in \mathbb {R}_+^n$ such that $u(\hat {x})=0$, from the anti-symmetry of $u$, we have

\[ (-\Delta)^{s_1}u(\hat{x})=\int_{\mathbb{R}^n}\frac{-u(y)}{|\hat{x}-y|^{n+2s_1}}\,{\rm d}y<0, \]

which contradicts with the equation

\[ (-\Delta)^{s_1}u(\hat{x})=|\hat{x}|^\alpha v^p(\hat{x})\geq 0. \]

Thus $u(x)>0,$ and using the same arguments as above, we easily obtain $v(x)>0$. Therefore, we may assume that $u(x)>0$ and $v(x)>0$ in the rest proof of (i) of theorem 1.

Let $u_\lambda (x)$ and $v_\lambda (x)$ be the Kelvin transform of $u(x)$ and $v(x)$ centred at origin, respectively

\begin{align*} u_\lambda(x)& =\left(\frac{\lambda}{|x|}\right)^{n-2s_1}u\left(\frac{\lambda^2x}{|x|^2}\right),\\ v_\lambda(x)& =\left(\frac{\lambda}{|x|}\right)^{n-2s_2}v\left(\frac{\lambda^2x}{|x|^2}\right) \end{align*}

for arbitrary $x\in \mathbb {R}^n\setminus \{0\}$. By an elementary calculation, $u_\lambda (x)$ and $v_\lambda (x)$ satisfy the following system:

\[ \left\{\begin{array}{@{}ll} (-\Delta)^{s_1}u_\lambda(x)=|x|^{\alpha}\left(\dfrac{\lambda}{|x|}\right)^{\tau_1}v_\lambda^p(x), & x\in \mathbb{R}_+^n, \\ (-\Delta)^{s_2}v_\lambda(x)=|x|^{\beta}\left(\dfrac{\lambda}{|x|}\right)^{\tau_2}u_\lambda^q(x), & x\in \mathbb{R}_+^n, \end{array}\right. \]

where

\[ \tau_1=n+2s_1+2\alpha-p(n-2s_2)\quad\text{and}\quad \tau_2=n+2s_2+2\beta -q(n-2s_1). \]

Note that both $\tau _1$ and $\tau _2$ are nonnegative and they will not be zero simultaneously, since $(p,q)\in \mathcal {R}_{sub}$.

Denote

\[ U_\lambda(x)=u_\lambda(x)-u(x)\quad \text{and}\quad V_\lambda(x)=v_\lambda(x)-v(x). \]

By elementary calculations and the mean value theorem, for $x\in B_\lambda ^+$, there holds

(3.1)\begin{align} (-\Delta)^{s_1}U_\lambda(x)& =|x|^{\alpha}\left(\frac{\lambda}{|x|}\right)^{\tau_1}v_\lambda^p(x)-|x|^{\alpha}v^p(x) \nonumber\\ & =|x|^\alpha\left[(v_\lambda^p(x)-v^p(x))+\left(\left(\frac{\lambda}{|x|}\right)^{\tau_1}-1\right)v_\lambda^p(x)\right]\nonumber\\& \geq|x|^\alpha p\xi_{\lambda}^{p-1}(x)V_\lambda(x), \end{align}

(3.2)\begin{align} (-\Delta)^{s_2}V_\lambda(x)& \geq|x|^\beta q\eta_{\lambda}^{q-1}(x)U_\lambda(x), \end{align}

where $\xi _\lambda (x)$ is between $v(x)$ and $v_\lambda (x)$, $\eta _\lambda (x)$ is between $u(x)$ and $u_\lambda (x)$. Note that

(3.3)\begin{equation} U_\lambda(x)={-}\left(\frac{\lambda}{|x|}\right)^{n-2s_1}U_\lambda(x^\lambda)\quad \text{and}\quad V_\lambda(x)={-}\left(\frac{\lambda}{|x|}\right)^{n-2s_2}V_\lambda(x^\lambda). \end{equation}

Next, we will use the method of moving spheres to claim that $U_\lambda (x)\geq 0$ and $V_\lambda (x)\geq 0$ in $B_\lambda ^+$ for any $\lambda >0$.

Step 1. Give a start point. We show that for sufficiently small $\lambda >0$,

(3.4)\begin{equation} U_\lambda(x)\geq 0\ \text{and}\ V_\lambda(x)\geq 0,\quad x\in B_\lambda^+. \end{equation}

Suppose (3.4) is not true, there must exist a point $\bar {x}\in B_\lambda ^+$ such that at least one of $U_\lambda (\bar {x})$ and $V_\lambda (\bar {x})$ is negative at this point. Without loss of generality, we assume

\[ U_\lambda(\bar{x})=\inf_{x\in B_\lambda^+}\{U_\lambda(x),V_\lambda(x)\}<0. \]

We will obtain contradictions for all four possible cases, respectively.

Case 1. $(p,q)\in \mathcal {R}_{sub}$ and $p\geq 1, q\geq 1$. Due to $p\geq 1$, by the convexity of the function $f(t)=t^p$, then we can take $\xi _\lambda (x)=v(x)$ in (3.1). From equation (3.1) and lemma 2.2, we have

(3.5)\begin{align} |\bar{x}|^\alpha pv^{p-1}(\bar{x})U_\lambda(\bar{x})& \leq|\bar{x}|^\alpha pv^{p-1}(\bar{x})V_\lambda(\bar{x})\leq(-\Delta)^{s_1}U_\lambda(\bar{x})\nonumber\\ & \leq C(n,s_1)U_\lambda(\bar{x})\left((\lambda-|\bar{x}|)^{{-}2s_1}+\frac{\delta^n}{\bar{x}_n^{n+2s_1}}\right). \end{align}

Hence,

(3.6)\begin{equation} v^{p-1}(\bar{x})\geq\frac{C(n,s_1)\left((\lambda-|\bar{x}|)^{{-}2s_1}+\frac{\delta^n}{\bar{x}_n^{n+2s_1}}\right)}{p|\bar{x}|^\alpha }. \end{equation}

If $\delta =\min \{\lambda -|\bar {x}|,\bar {x}_n\}=\lambda -|\bar {x}|$, which implies that $\lambda -|\bar {x}|\leq \bar {x}_n\leq |\bar {x}|$, using the fact and (3.6), we obtain

(3.7)\begin{equation} v^{p-1}(\bar{x})\geq\frac{C(\lambda-|\bar{x}|)^{{-}2s_1}}{p|\bar{x}|^\alpha }\geq C|\bar{x}|^{{-}2s_1-\alpha}. \end{equation}

As $\lambda \to 0$, the right-hand side of (3.7) will go to infinity since $\alpha >-2s_1$. This is impossible.

If $\delta =\min \{\lambda -|\bar {x}|,\bar {x}_n\}=\bar {x}_n$, from $\bar {x}_n\leq |\bar {x}|$ and (3.6), we derive

(3.8)\begin{equation} v^{p-1}(\bar{x})\geq\frac{C\bar{x}_n^{{-}2s_1}}{p|\bar{x}|^\alpha }\geq C|\bar{x}|^{{-}2s_1-\alpha}, \end{equation}

which is also impossible.

Case 2. $0< p,q<1$. Due to $p<1$, we can take $\xi _\lambda (x) =v_\lambda (x)$. From equation (3.1) and lemma 2.2, we have

(3.9)\begin{align} |\bar{x}|^\alpha p v_\lambda^{p-1}(\bar{x})U_\lambda(\bar{x})& \leq|\bar{x}|^\alpha p v_\lambda^{p-1}(\bar{x})V_\lambda(\bar{x})\leq (-\Delta)^{s_1}U_\lambda(\bar{x}) \nonumber\\ & \leq C(n,s_1)U_\lambda(\bar{x})\left((\lambda-|\bar{x}|)^{{-}2s_1}+\frac{\delta^n}{\bar{x}_n^{n+2s_1}}\right). \end{align}

Analogous to (3.7) and (3.8), there holds

(3.10)\begin{equation} v_\lambda^{p-1}(\bar{x})\geq\frac{C\bar{x}_n^{{-}2s_1}}{|\bar{x}|^\alpha}, \end{equation}

or

(3.11)\begin{equation} v_\lambda^{p-1}(\bar{x})\geq\frac{C(\lambda-|\bar{x}|)^{{-}2s_1}}{|\bar{x}|^\alpha }. \end{equation}

Applying lemma 2.3 and the mean value theorem, we obtain that for $x\in B_1^+$,

\begin{align*} u(x)& \geq C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right)|y|^{\alpha}v^p(y)\,{\rm d}y \\ & \geq C\displaystyle\int_{B_{1}(2e_n)}\left(\displaystyle\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right){\rm d}y \\ & \geq C\int_{B_{1}(2e_n)}\frac{x_ny_n}{|x^*-y|^{n-2s_1+2}}\,{\rm d}y \\ & \geq Cx_n. \end{align*}

For $x\in (B_1^+)^C$, we derive

\begin{align*} u(x)& \geq C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right)|y|^{\alpha}v^p(y)\,{\rm d}y \\ & \geq C\int_{B_{1}(2e_n)}\frac{x_ny_n}{|x^*-y|^{n-2s_1+2}}\,{\rm d}y \\ & \geq C\frac{x_n}{|x|^{n-2s_1+2}}. \end{align*}

Similarly, we have

\[ v(x)\geq\left\{\begin{array}{@{}ll} Cx_n, & x\in B_1^+, \\ C\dfrac{x_n}{|x|^{n-2s_2+2}}, & x\in (B_1^+)^C. \end{array}\right. \]

Then by the definition of $v_\lambda (x)$, we obtain

(3.12)\begin{equation} v_\lambda(x)\geq \left\{\begin{array}{@{}ll} C\left(\dfrac{\lambda}{|x|}\right)^{n-2s_2+2}x_n, & x^\lambda\in B_1^+, \\ C\dfrac{x_n}{\lambda^{n-2s_2+2}}, & x^\lambda\in (B_1^+)^C. \end{array}\right. \end{equation}

If $\delta =\lambda -|\bar {x}|\leq \bar {x}_n$, that is, $|\bar {x}|\geq \frac {\lambda }{2}$, combining (3.11) and (3.12), we conclude that for $\bar {x}\in B_\lambda ^+$ and sufficiently small $\lambda$,

\[ (\lambda-|\bar{x}|)^{{-}2s_1} \leq C\bar{x}_n^{p-1}|\bar{x}|^\alpha\leq C(\lambda-|\bar{x}|)^{p-1}|\bar{x}|^\alpha, \]

which gives that

(3.13)\begin{equation} \left(\frac{\lambda}{|\bar{x}|}-1\right)^{{-}2s_1-p+1}\leq C|\bar{x}|^{p+2s_1+\alpha-1}. \end{equation}

Due to $\min \{p+2s_1, p+2s_1+\alpha \}>1$ and $\frac {\lambda }{|\bar {x}|}\leq 2$, inequality (3.13) is impossible as $\lambda >0$ sufficiently small.

If $\delta =\bar {x}_n$, it follows from (3.10) and (3.12) that

\[ \frac{C\bar{x}_n^{{-}2s_1}}{|\bar{x}|^\alpha }\leq v_\lambda^{p-1}(\bar{x})\leq C \bar{x}_n^{p-1}, \]

which implies that

\[ |\bar{x}|^{{-}p-2s_1-\alpha+1}\leq C\quad \text{if}\ \alpha\geq 0,\quad \text{and}\quad \bar{x}_n^{{-}p-2s_1-\alpha+1}\leq C\quad \text{if}\ \alpha< 0. \]

Either one of the two inequalities will yield a contradiction since the left terms go to infinity as $\min \{p+2s_1,p+2s_1+\alpha \}>1$ and $\lambda >0$ small enough.

For case 3: $(p,q)\in \mathcal {R}_{sub}, p\geq 1, 0< q<1$ and case 4: $(p,q)\in \mathcal {R}_{sub}, 0< p<1, q\geq 1$, similar argument as that of cases 1 and 2 can show that $U_\lambda (x)\geq 0$ and $V_\lambda (x)\geq 0$ in $B_\lambda ^+$ for sufficiently small $\lambda >0$. Therefore, (3.4) holds.

Step 2. Now we move the sphere $S_\lambda$ outwards as long as (3.4) holds. Define

\[ \lambda_0=\sup\{\lambda|\,U_\mu(x)\geq 0, V_\mu(x)\geq 0,\ x\in B_\mu^+,\ \forall\ 0<\mu<\lambda\}. \]

We will show that $\lambda _0=+\infty$. Suppose on the contrary that $0<\lambda _0<+\infty$. We want to show that there exists some small $\varepsilon >0$ such that for any $\lambda \in (\lambda _0,\lambda _0+\varepsilon )$,

(3.14)\begin{equation} U_\lambda(x)\geq 0\ \text{and}\ V_\lambda(x)\geq 0,\quad x\in B_\lambda^+. \end{equation}

This implies that the plane $S_{\lambda _0}$ will be moved outwards a little bit further, which contradicts with the definition of $\lambda _0$.

Firstly, we claim that

(3.15)\begin{equation} U_{\lambda_0}(x)> 0\ \text{and}\ V_{\lambda_0}(x)> 0,\quad x\in B_{\lambda_0}^+. \end{equation}

Indeed, if there exists some point $x^0\in B_{\lambda _0}^+$ such that $U_{\lambda _0}(x^0)= 0$, we have

(3.16)\begin{equation} (-\Delta)^{s_1}U_{\lambda_0}(x^0)=C\int_{\mathbb{R}^n}\frac{-U_{\lambda_0}(y)}{|x^0-y|^{n+2s_1}}\,{\rm d}y\leq 0. \end{equation}

On the other hand, it is easy to get that

(3.17)\begin{align} (-\Delta)^{s_1}U_{\lambda_0}(x^0)& =|x^0|^{\alpha}\left(\frac{\lambda_0}{|x^0|}\right)^{\tau_1}v_{\lambda_0}^p(x^0)-|x^0|^{\alpha}v^p(x^0) \nonumber\\ & =|x^0|^\alpha \left(\left(\frac{\lambda_0}{|x^0|}\right)^{\tau_1}-1\right)v_\lambda^p(x^0)+p|x^0|^\alpha \xi_\lambda^{p-1}(x^0)V_{\lambda_0}(x^0). \end{align}

If $\tau _1>0$, then $(-\Delta )^{s_1}U_{\lambda _0}(x^0)>0$, where we use the facts that $V_{\lambda _0}\geq 0$ and $v>0$ in $\mathbb {R}^n_+$. If $\tau _1=0$, then we have that $\tau _2>0$. Moreover, $V_{\lambda _0}(x^0)=0$ follows from (3.16) and (3.17). Using an argument similar to (3.16) and (3.17), we derive

\begin{align*} 0\geq(-\Delta)^{s_2}V_{\lambda_0}(x^0)& = |x^0|^\beta \left(\left(\frac{\lambda_0}{|x^0|}\right)^{\tau_2}-1\right)u_\lambda^q(x^0)+|x^0|^\beta \eta_\lambda^{q-1}(x^0)U_{\lambda_0}(x^0) \\ & = |x^0|^\beta \left(\left(\frac{\lambda_0}{|x^0|}\right)^{\tau_2}-1\right)u_\lambda^q(x^0)>0, \end{align*}

which is absurd. Thus, $U_{\lambda _0}(x)> 0$ is proved. Similarly, we derive that $V_{\lambda _0}(x)> 0$. Hence, (3.15) holds.

Next, we will show that the sphere can be moved further outwards. The continuity of $u(x)$ and (3.15) yield that there exists some sufficiently small $l\in (0,\frac {\lambda _0}{2})$ and $\varepsilon _1\in (0,\frac {\lambda _0}{2})$ such that for $\lambda \in (\lambda _0,\lambda _0+\varepsilon _1)$,

(3.18)\begin{equation} U_\lambda(x)\geq 0, \quad x\in B_{\lambda_0-l}^+. \end{equation}

For $x\in B_\lambda ^+\setminus B_{\lambda _0-l}^+$, using the similar proof of (3.4), we can deduce that

(3.19)\begin{equation} U_\lambda(x)\geq 0,\quad x\in B_\lambda^+{\setminus} B_{\lambda_0-l}^+ . \end{equation}

Note that the distance between $\bar {x}$ and $S_\lambda$, i.e. $\lambda -|\bar {x}|$, plays an important role in this process.

Hence, it follows from (3.18) and (3.19) that for all $\lambda \in (\lambda _0,\lambda _0+\varepsilon _1)$,

\[ U_\lambda(x)\geq 0,\quad x\in B_\lambda^+. \]

Similarly, we can also prove that there exists $\varepsilon _2>0$ such that for all $\lambda \in (\lambda _0, \lambda _0+\varepsilon _2)$,

\[ V_\lambda(x)\geq 0,\quad x\in B_\lambda^+. \]

Let $\varepsilon =\min \{\varepsilon _1,\varepsilon _2\}$, therefore, (3.14) can be completely concluded. This contradicts with the definition of $\lambda _0$. So $\lambda _0=+\infty$.

Then, we have for every $\lambda >0$,

\[ U_\lambda(x)\geq 0,\quad V_\lambda(x)\geq 0,\quad x\in B_\lambda^+, \]

which gives that,

(3.20)\begin{equation} u(x)\geq\left(\frac{\lambda}{|x|}\right)^{n-2s_1}u\left(\frac{\lambda^2x}{|x|^2}\right),\quad \forall \,|x|\geq\lambda, \ \ x\in\mathbb{R}_+^n,\ \forall\, 0<\lambda<{+}\infty. \end{equation}

(3.21)\begin{equation} v(x)\geq\left(\frac{\lambda}{|x|}\right)^{n-2s_2}v\left(\frac{\lambda^2x}{|x|^2}\right),\quad \forall \,|x|\geq\lambda,\ x\in\mathbb{R}_+^n, \ \forall\, 0<\lambda<{+}\infty. \end{equation}

For any given $|x|\geq 1$, let $\lambda =\sqrt {|x|}$, then it follows from (3.20) that

(3.22)\begin{equation} u(x)\geq\left(\min_{x\in S_1^+}u(x)\right)\frac{1}{|x|^{\frac{n-2s_1}{2}}}:=\frac{C}{|x|^{\frac{n-2s_1}{2}}}\geq\frac{Cx_n}{|x|^{\frac{n-2s_1}{2}+1}}, \end{equation}

and similarly from (3.21), we obtain

(3.23)\begin{equation} v(x)\geq\frac{Cx_n}{|x|^{\frac{n-2s_2}{2}+1}}. \end{equation}

Now we make full use of the above properties to derive some lower-bound estimates of solutions to (1.1) through iteration technique.

Let $\theta _0=\frac {n-2s_1}{2}+1$, $\sigma _0=\frac {n-2s_2}{2}+1$. From lemma 2.3, inequality (3.23) and the mean value theorem, we have for $x_n>1$,

(3.24)\begin{align} u(x)& \geq C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right)|y|^{\alpha}v^p(y)\,{\rm d}y\nonumber\\ & \geq C\int_{2|x|}^{4|x|}\int_{2|x|\leq |y'|\leq 4|x|}\frac{x_ny_n}{|x^*-y|^{n-2s_1+2}}\frac{y_n^p}{|y|^{p\sigma_0-\alpha}}\,{\rm d}y'\,{\rm d}y_n\nonumber\\ & \geq C\frac{x_n}{|x|^{(n-2s_1+2)+(p\sigma_0-\alpha)}}\int_{2|x|}^{4|x|}\int_{2|x|\leq |y'|\leq 4|x|}y_n^{p+1}\,{\rm d}y'\,{\rm d}y_n\nonumber\\ & \geq C\frac{x_n}{|x|^{p\sigma_0-\alpha-2s_1+3-(p+2)}}. \end{align}

Similarly, we have

(3.25)\begin{equation} v(x) \geq C\frac{x_n}{|x|^{q(\theta_0-1)-(\beta+2s_2)+1}} \end{equation}

Denote $\theta _1=p(\sigma _0-1)-(\alpha +2 s_1)+1$ and $\sigma _1=q(\theta _0-1)-(\beta +2s_2)+1$. Repeat the above process replacing (3.23) by (3.25), then we have

\[ u(x)\geq C\int_{2|x|}^{4|x|}\int_{2|x|\leq |y'|\leq 4|x| }\frac{x_ny_n}{|x^*-y|^{n-2s_1+2}}\frac{y_n^p}{|y|^{p\sigma_1-\alpha}}\,{\rm d}y'\,{\rm d}y_n\geq C\frac{x_n}{|x|^{\theta_2}}, \]

and analogously,

(3.26)\begin{equation} v(x)\geq C\frac{x_n}{|x|^{\sigma_2}}, \end{equation}

where $\theta _2=p(\sigma _1-1)-\alpha -2s_1+1$ and $\sigma _2=q(\theta _1-1)-\beta -2s_2+1$.

After such $k$ iteration steps, we derive

(3.27)\begin{equation} u(x)\geq C\frac{x_n}{|x|^{\theta_{k+1}}},\quad v(x)\geq C\frac{x_n}{|x|^{\sigma_{k+1}}}, \end{equation}

where $\theta _{k+1}=p(\sigma _{k}-1)-\alpha -2s_1+1$ and $\sigma _{k+1}=q(\theta _{k}-1)-\beta -2s_2+1$. Elementary calculations give that

(3.28)\begin{align} \theta_{2m}& =\frac{n-2s_1}{2}(pq)^{m}-\left[\left(p(2s_2+\beta)+(2s_1+\alpha)\right)\frac{1-(pq)^m}{1-pq}\right]+1,\nonumber\\ \theta_{2m+1}& =\left(\frac{p(n-2s_2)}{2}-2s_1-\alpha \right)(pq)^m\nonumber\\ & \quad-\left[\left(p(2s_2+\beta)+(2s_1+\alpha)\right)\frac{1-(pq)^m}{1-pq}\right]+1,\nonumber\\ \sigma_{2m}& =\frac{n-2s_2}{2}(pq)^{m}-\left[\left(q(2s_1+\alpha)+(2s_2+\beta)\right)\frac{1-(pq)^m}{1-pq}\right]+1, \nonumber\\ \sigma_{2m+1}& =\left(\frac{q(n-2s_1)}{2}-2s_2-\beta \right)(pq)^m\nonumber\\ & \quad-\left[\left(q(2s_1+\alpha)+(2s_2+\beta)\right)\frac{1-(pq)^m}{1-pq}\right]+1, \end{align}

where $m=0,1,2,\ldots$.

For the case $pq\geq 1$, we claim that both $\{\theta _k\}$ and $\{\sigma _k\}$ are decreasing sequences and unbounded from below. Denote

\begin{align*} & A_e=p\frac{n-2s_2}{2}-\frac{n-2s_1}{2}-2s_1-\alpha,\\ & A_o=q\frac{n-2s_1}{2}-\frac{n-2s_2}{2}-2s_2-\beta. \end{align*}

Note that $A_e,A_o\leq 0$ and they will not be zero simultaneously, since $(p,q)\in \mathcal {R}_{sub}$. By elementary calculations, we have

(3.29)\begin{equation} \theta_{k+1}-\theta_{k}=\left\{\begin{array}{@{}ll} p^{\left[\dfrac{k+1}{2}\right]}q^{\left[\dfrac{k}{2}\right]}A_e\leq 0, & k \text{ is even},\\ p^{\left[\dfrac{k+1}{2}\right]}q^{\left[\dfrac{k}{2}\right]}A_o\leq 0, & k \text{ is odd}. \end{array}\right. \end{equation}

Hence, the sequence $\{\theta _k\}$ is decreasing.

Combining the above properties of $A_e, A_o$, the assumption $pq\geq 1$ and (3.29), we deduce that $\theta _k\to -\infty$ as $k\to +\infty$. Similarly, we can show that $\{\sigma _k\}$ is also decreasing and $\sigma _k\to -\infty$. These and (3.27) indicate that $u(x)$ and $v(x)$ do not belong to any $L_{2s}$. Hence, the solutions $(u,v)\equiv (0,0)$ when $pq\geq 1$.

Next, we consider the case $pq<1$. From (3.28), we can conclude that

(3.30)\begin{align} \theta_k\to1-\frac{p(2s_2+\beta)+(2s_1+\alpha)}{1-pq},\quad \sigma_k\to1-\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}\quad \text{as }k\to \infty. \end{align}

Since $p+2s_1+\alpha >1$ and $q+2s_2+\beta >1$, we have

\[ \frac{p(2s_2+\beta)+(2s_1+\alpha)}{1-pq}-1>0,\quad \frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}-1>0. \]

Hence, from (3.30) and (3.27), we have for $x_n>1$,

(3.31)\begin{equation} u(x)\geq cx_n^{\frac{p(2s_2+\beta)+(2s_1+\alpha)}{1-pq}+o(1)}, \quad v(x)\geq cx_n^{\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}+o(1)}. \end{equation}

Combining this with lemma 2.3, we have

\begin{align*} u(x)& \geq C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right)|y|^{\alpha}v^p(y)\,{\rm d}y \\ & \geq C \int_{2x_n}^{+\infty}\int_{\mathbb{R}^{n-1}}\left(\frac{1}{|x-y|^{n-2s_1}}\right.\\& \quad \left.-\frac{1}{|x^*-y|^{n-2s_1}}\right) y_n^{\left(\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}+o(1)\right)p+\alpha}\,{\rm d}y'\,{\rm d}y_n \\ & \geq C\int_{2x_n}^{+\infty}y_n^{\left(\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}+o(1)\right)p+\alpha}\,{\rm d}y_n\\ & \quad \times \int_{\mathbb{R}^{n-1}}\frac{x_ny_n}{\left(|x'-y'|^2+|x_n+y_n|^2\right)^{\frac{n-2s_1+2}{2}}}\,{\rm d}y'. \end{align*}

Let $x'-y'=(x_n+y_n)z'$, we have

\begin{align*} u(x)& \geq C\int_{2x_n}^{+\infty}\frac{y_n^{\left(\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}+o(1)\right)p+\alpha+1}x_n}{(x_n+y_n)^{3-2s_1}}\,{\rm d}y_n\int_{\mathbb{R}^{n-1}}\frac{1}{\left(|z'|^2+1\right)^{\frac{n-2s_1+2}{2}}}\,{\rm d}z' \\ & \geq C\int_{2x_n}^{+\infty}\frac{y_n^{\left(\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}+o(1)\right)p+\alpha+1}x_n}{(x_n+y_n)^{3-2s_1}}\,{\rm d}y_n. \end{align*}

Let $y_n=x_nz_n$, one has

(3.32)\begin{align} u(x)& \geq Cx_n^{\left(\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}+o(1)\right)p+2s_1+\alpha}\int_{2}^{+\infty}\frac{z_n^{\frac{q(2s_1+\alpha) +(2s_2+\beta)}{1-pq}p+\alpha+1}}{(1+z_n)^{3-2s_1}}\,{\rm d}z_n\nonumber\\ & \cong\int_{2}^{+\infty}\frac{1}{(z_n)^{3-2s_1-\left(\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}p+\alpha+1\right)}}\,{\rm d}z_n. \end{align}

Due to $p+2s_1+\alpha >1$ and $q+2s_2+\beta >1$, we have $\frac {q(2s_1+\alpha )+(2s_2+\beta )}{1-pq}>1$, which implies that

\[ 3-2s_1-\left(\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}p+\alpha+1\right)<1. \]

This yields that the right-hand side of (3.32) is infinity, which is impossible. Hence, for $pq<1$ we also obtain $(u,v)\equiv (0,0).$

In sum, we conclude that there is no nontrivial classical solutions to system (1.1) for the case that $(p,q)\in \mathcal {R}_{sub}$, $\min \{p+2s_1, p+2s_1+\alpha \}>1$, $\min \{q+2s_2,q+2s_2+\beta \}>1$.

3.2 Proof of (ii) of theorem 1.1

Proof. Assume that $(u,v) \not \equiv (0,0)$, from the proof of (i), we know that $u>0$ and $v>0$ in $\mathbb {R}_+^n$. Applying lemma 2.3, for $x_n>\min \{2,\frac {|x'|}{10}\}$, we have

(3.33)\begin{align} u(x)& \geq C\int_{\mathbb{R}_+^n}\left(\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right)|y|^{\alpha}v^p(y)\,{\rm d}y \nonumber\\ & \geq C\int_{B_1(2e_n)}\left(\frac{x_ny_n}{|x^*-y|^{n-2s_1+2}}\right)|y|^{\alpha}v^p(y)\,{\rm d}y \nonumber\\ & \geq C\frac{1+|x|}{(1+|x|)^{n-2s_1+2}}\int_{B_1(2e_n)}y_n|y|^{\alpha}v^p(y)\,{\rm d}y\nonumber\\ & \geq C\frac{1}{(1+|x|)^{n-2s_1+1}}. \end{align}

Similarly, for $x_n>\min \{2,\frac {|x'|}{10}\}$, we have

(3.34)\begin{equation} v(x)\geq \frac{C}{(1+|x|)^{n-2s_2+1}}. \end{equation}

Iterating (3.33) with (3.34), for $x_n>\min \{2,\frac {|x'|}{10}\}$, we get

(3.35)\begin{align} u(x)& \geq C\int_{B_{|x|}(0,4|x|)}\left(\frac{1}{|x-y|^{n-2s_1}}-\frac{1}{|x^*-y|^{n-2s_1}}\right)\frac{|y|^\alpha}{(1+|y|)^{p(n-2s_2+1)}}\,{\rm d}y\nonumber\\ & \geq C\int_{B_{3|x|}(0,4|x|)\setminus B_{2|x|}(0,4|x|)}\frac{x_ny_n}{|x^*-y|^{n-2s_1+2}}\frac{|y|^\alpha}{(1+|y|)^{p(n-2s_2+1)}}\,{\rm d}y\nonumber\\ & \geq C\frac{(1+|x|)^{2+\alpha}}{(1+|x|)^{n-2s_1+2+(n-2s_2+1)p}}\int_{B_{3|x|}(0,4|x|)\setminus B_{2|x|}(0,4|x|)}\,{\rm d}y\nonumber\\ & \geq C\frac{1}{(1+|x|)^{(n-2s_2+1)p-2s_1-\alpha}}. \end{align}

Using the same argument as that of (3.35), for $x_n>\min \left \{2,\frac {|x'|}{10}\right \}$, we obtain

\[ v(x)\geq C\frac{1}{(1+|x|)^{(n-2s_1+1)q-2s_2-\beta}}. \]

After $k$ iteration steps, it is easy to see that for $|x|$ large and $x_n>\min \left \{2,\frac {|x'|}{10}\right \}$,

\[ u(x)\geq \frac{C}{(1+|x|)^{\gamma_k}}, \quad v(x)\geq \frac{C}{(1+|x|)^{\delta_k}}. \]

Here,

\[ \gamma_k=\delta_{k-1}p-2s_1-\alpha,\quad \delta_k=\gamma_{k-1}q-2s_2-\beta, \]

where

\[ \gamma_1=(n-2s_2+1)p-2s_1-\alpha,\quad \delta_1=(n-2s_1+1)q-2s_2-\beta. \]

Simple calculations imply that

\begin{align*} \gamma_{2m}& =(n-2s_1+1)(pq)^m-\left[\left(p(2s_2+\beta)+(2s_1+\alpha)\right)\frac{1-(pq)^m}{1-pq}\right], \\ \gamma_{2m+1}& =\left[p(n-2s_2+1)-(2s_1+\alpha)\right](pq)^m\\& \quad -\left[\left(p(2s_2+\beta)+(2s_1+\alpha)\right)\frac{1-(pq)^m}{1-pq}\right],\\ \delta_{2m}& =(n-2s_2+1)(pq)^m-\left[\left(q(2s_1+\alpha)+(2s_2+\beta)\right)\frac{1-(pq)^m}{1-pq}\right], \\ \delta_{2m}& =\left[q(n-2s_1+1)-(2s_2+\beta)\right](pq)^m\\& \quad -\left[\left(q(2s_1+\alpha)+(2s_2+\beta)\right)\frac{1-(pq)^m}{1-pq}\right], \end{align*}

where $m=0,1,2,\ldots.$

From $0< pq<1$, we obtain

\[ \gamma_k\to-\displaystyle\frac{p(2s_2+\beta)+(2s_1+\alpha)}{1-pq},\quad \delta_k\to-\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq},\quad \text{as } k\to\infty. \]

This yields that for $|x|$ large,

\[ u(x)\geq C(1+|x|)^{\frac{p(2s_2+\beta)+(2s_1+\alpha)}{1-pq}-o(1)}, \quad v(x)\geq C(1+|x|)^{\frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}-o(1)}, \]

as $k\to +\infty$. Due to $0< pq<1$ and $\alpha \geq -2s_1pq$, $\beta \geq -2s_2pq$, we have

\[ \frac{p(2s_2+\beta)+(2s_1+\alpha)}{1-pq}>2s_1,\quad \frac{q(2s_1+\alpha)+(2s_2+\beta)}{1-pq}>2s_2 \]

This contradicts with the assumptions that $u(x)\in L_{2s_1}$ and $v(x)\in L_{2s_2}$. Hence, $(u,v)\equiv (0,0)$.

4. Proof of theorem 1.3

In this section, we give the proof of theorem 1.3 by using the method of moving planes.

Proof. Suppose on the contrary that $(u,v) \not \equiv (0,0)$, we know that $u>0$ and $v>0$ in $\mathbb {R}_+^n$. Let $\bar {u}(x)$ and $\bar {v}(x)$ be the Kelvin transform of $u(x)$ and $v(x)$ centred at origin, respectively

\begin{align*} & \bar{u}(x)=\left(\frac{1}{|x|}\right)^{n-2s_1}u\left(\frac{x}{|x|^2}\right),\\ & \bar{v}(x)=\left(\frac{1}{|x|}\right)^{n-2s_2}v\left(\frac{x}{|x|^2}\right) \end{align*}

for arbitrary $x\in \mathbb {R}^n\setminus \{0\}$. Then, $\bar {u}(x)$ and $\bar {v}(x)$ satisfy the following system:

\[ \left\{\begin{array}{@{}ll} (-\Delta)^{s_1}\bar{u}(x)=\left(\dfrac{1}{|x|}\right)^{\bar{\tau}_1}\bar{v}^p(x), & x\in \mathbb{R}_+^n, \\ (-\Delta)^{s_2}\bar{v}(x)=\left(\dfrac{1}{|x|}\right)^{\bar{\tau}_2}\bar{u}^q(x), & x\in \mathbb{R}_+^n, \\ \bar{u}(x)> 0,\ \bar{v}(x)> 0, & x\in \mathbb{R}_+^n, \\ \bar{u}(x',x_n)={-}\bar{u}(x',-x_n),\ \bar{v}(x',x_n)={-}\bar{v}(x',-x_n), & x=(x',x_n)\in\mathbb{R}^n, \end{array}\right. \]

where

\[ \bar{\tau}_1=n+2s_1+\alpha-p(n-2s_2)\ \text{and}\ \bar{\tau}_2=n+2s_2+\beta -q(n-2s_1). \]

Note that $\bar {\tau }_1\leq 0$ and $\bar {\tau }_2\leq 0$ due to $p\geq \frac {n+2s_1+\alpha }{n-2s_2}$ and $q\geq \frac {n+2s_2+\beta }{n-2s_1}$. Obviously, for $|x|$ large enough,

(4.1)\begin{equation} \bar{u}(x)=O\left(\frac{1}{|x|^{n-2s_1}}\right)\quad \text{and}\quad \bar{v}(x)=O\left(\frac{1}{|x|^{n-2s_2}}\right). \end{equation}

For any real number $\rho >0$ and $x\in \Sigma _\rho$, define

\[ \bar{U}_\rho(x)=\bar{u}(x^-_\rho)-\bar{u}(x),\quad \bar{V}_\rho(x)=\bar{v}(x^-_\rho)-\bar{v}(x), \]

where $\Sigma _\rho$, $T_\rho$ and $x^-_\rho$ are defined as in proposition 2.1. Then, for $x\in \Sigma _\rho \cap \mathbb {R}^n_+$, we have

(4.2)\begin{align} (-\Delta)^{s_1}\bar{U}_\rho(x)& =|x^-_\rho|^{-\bar{\tau}_1} |\bar{v}(x^-_\rho)|^{p-1}\bar v(x^-_\rho)-|x|^{-\bar{\tau}_1} |\bar v(x)|^{p-1}\bar v(x)\nonumber\\ & =|x|^{-\bar{\tau}_1} \left(|\bar{v}(x^-_\rho)|^{p-1}\bar v(x^-_\rho)-|\bar v(x)|^{p-1}\bar v(x)\right)\nonumber\\ & \quad+|\bar{v}(x^-_\rho)|^{p-1}\bar v(x^-_\rho)\left(|x^-_\rho|^{-\bar{\tau}_1}-|x|^{-\bar{\tau}_1}\right)\nonumber\\ & \geq |x|^{-\bar{\tau}_1} \left(|\bar{v}(x^-_\rho)|^{p-1}\bar v(x^-_\rho)-|\bar v(x)|^{p-1}\bar v(x)\right)\nonumber\\ & \geq |x|^{-\bar{\tau}_1} p|\bar{v}|^{p-1}(x)\bar{V}_\rho(x), \end{align}

where we use the fact $p>1$ in the last inequality. Similarly,

(4.3)\begin{equation} (-\Delta)^{s_2}\bar{V}_\rho(x)\geq |x|^{-\bar{\tau}_2} q|\bar{u}|^{q-1}(x)\bar{U}_\rho(x). \end{equation}

Step 1. We claim that for $\rho >0$ sufficiently small,

(4.4)\begin{equation} \bar{U}_\rho(x)\geq 0\ \text{and}\ \bar{V}_\rho(x)\geq 0,\quad x\in\Sigma_\rho. \end{equation}

Otherwise, from (4.1), there exists some $\bar {x}\in \Sigma _\rho \cap \mathbb {R}_+^n$ such that at least one of $\bar {U}_\rho (x)$, $\bar {V}_\rho (x)$ is negative. Without loss of generality, we may assume that

\[ \bar{U}_\rho(\bar{x})=\inf_{\Sigma_\rho}\{\bar{U}_\rho(x),\bar{V}_\rho(x)\}<0. \]

Combining equation (4.2) and lemma 2.1, we deduce

(4.5)\begin{align} |\bar{x}|^{-\bar{\tau}_1} p\bar{v}^{p-1}(\bar{x})\bar{U}_\rho(\bar{x})& \leq|\bar{x}|^{-\bar{\tau}_1} p\bar{v}^{p-1}(\bar{x})\bar{V}_\rho(\bar{x})\leq(-\Delta)^{s_1}\bar{U}_\rho(\bar{x})\nonumber\\ & \leq C(n,s_1)\bar{U}_\rho(\bar{x})(\rho-\bar{x}_n)^{{-}2s_1}. \end{align}

This yields that

(4.6)\begin{equation} |\bar{x}|^{-\bar{\tau}_1} p\bar{v}^{p-1}(\bar{x})\geq C(\rho-\bar{x}_n)^{{-}2s_1}\geq C\rho^{{-}2s_1}. \end{equation}

Observe that (4.1) and the decay conditions of $u$ and $v$ in theorem 1.3 ensure that

\[ \mathop{\lim}\limits_{ x \to \infty}|x|^{-\bar{\tau}_1} p\bar{v}^{p-1}(x)=0, \quad\mathop{\overline{\lim}}\limits_{x\to 0}|x|^{-\bar{\tau}_1} p\bar{v}^{p-1}(x)\leq C, \]

where we used the assumption $\alpha >-2s_2$. Hence, inequality (4.6) is impossible as $\rho >0$ is sufficiently small. Therefore, (4.4) holds.

Step 2. Move the plane $T_\rho$ upwards along the $x_n$-axis as long as (4.4) holds. Let

\[ \rho_0=\sup\{\rho \,|\, \bar{U}_\mu(x)\geq 0, \bar{V}_\mu(x)\geq 0, x\in \Sigma_\mu,\mu\leq \rho,\rho>0\}. \]

We will show that $\rho _0=+\infty$ by contradiction arguments.

Suppose on the contrary that $0<\rho _0<+\infty$. We will verify that

(4.7)\begin{equation} \bar{U}_{\rho_0}(x)\equiv0,\ \text{and}\ \bar{V}_{\rho_0}(x)\equiv 0,\quad x\in\Sigma_{\rho_0}. \end{equation}

Then using the above equalities (4.7), we immediately obtain

\[ 0<\bar{u}(x^-_{\rho_0})=u(x)=0,\quad 0<\bar{v}(x^-_{\rho_0})=v(x)=0, \quad x\in \partial\mathbb{R}_+^n, \]

which is impossible. Thus $\rho _0=+\infty$ must hold.

Therefore, our goal is to prove (4.7). Suppose that (4.7) does not hold, then we deduce that

(4.8)\begin{equation} \bar{U}_{\rho_0}(x)>0,\ \text{and}\ \bar{V}_{\rho_0}(x)>0,\quad x\in\Sigma_{\rho_0}. \end{equation}

Otherwise, there exists some point $\tilde {x}\in \Sigma _{\rho _0}\cap \mathbb {R}^n_+$ such that $\bar {U}_{\rho _0}(\tilde {x})=0$. We have

\[ (-\Delta)^s\bar{U}_{\rho_0}(\tilde{x})=C\int_{\mathbb{R}^n}\frac{-\bar{U}_{\rho_0}(y)}{|\tilde{x}-y|^{n+2s}}\,{\rm d}y<0. \]

On the contrary, it is easy to get that

\begin{align*} (-\Delta)^{s}\bar{U}_{\rho_0}(x)& =|\tilde{x}^-_{\rho_0}|^{-\bar{\tau}_1} |\bar{v}(\tilde{x}^-_{\rho_0})|^{p-1}\bar{v}(\tilde{x}^-_{\rho_0})\\& \quad -|\tilde{x}|^{-\bar{\tau}_1} |\bar{v}(\tilde{x})|^{p-1}\bar{v}(\tilde{x})\geq |\tilde{x}|^{-\bar{\tau}_1} p|\bar{v}|^{p-1}(\tilde{x})\bar{V}{\rho_0}(\tilde{x})\geq 0, \end{align*}

where we use the fact $\bar {V}_{\rho _0}\geq 0$. This leads to a contradiction. Hence, (4.8) holds.

Now we show that the plane $T_{\rho _0}$ can be moved upwards a little bit further and hence obtain a contradiction with the definition of $\rho _0$. Precisely, we will verify that there exists some small $\varepsilon >0$ such that for any $\rho \in (\rho _0,\rho _0+\varepsilon )$,

(4.9)\begin{equation} \bar{U}_\rho(x)\geq 0\ \text{and}\ \bar{V}_\rho(x)\geq 0,\quad x\in\Sigma_\rho, \end{equation}

where $\varepsilon$ is determined later.

If (4.9) is not true, then for any $\varepsilon _k\to 0$ as $k\to +\infty$, there exists $\rho _k\in (\rho _0,\rho _0+\varepsilon _k)$ and $x_k\in \mathbb {R}_+^n\cap \Sigma _{\rho _k}$ such that

(4.10)\begin{equation} \bar{U}_{\rho_k}(x_k)=\inf_{\Sigma_{\rho_k}}\{\bar{U}_{\rho_k}(x),\bar{V}_{\rho_k}(x)\}<0. \end{equation}

Similar argument as that of (4.5) gives that

(4.11)\begin{equation} (-\Delta)^{s_1}\bar{U}_{\rho_k}(x_k)+c(x_k)\bar{U}_{\rho_k}(x_k)\geq 0, \end{equation}

where $c(x)=-|x|^{-\bar {\tau }_1} p\bar {v}^{p-1}(x)$. From (4.1) and the decay conditions of $u$ and $v$, we deduce that

(4.12)\begin{equation} \mathop{\lim}\limits_{ x \to \infty}|x|^{2s_1}c(x)=0\ \text{and}\ c(x) \text{ is bounded below in } \Sigma_{\rho_k}, \end{equation}

where we used the assumption $\alpha >-2s_2$. Then from proposition 2.1 we know that there exists $\ell _k>0$ and $R_0>0$ such that

(4.13)\begin{equation} x_k\in B_{R_0}(0)\cap\Sigma_{\rho_k-\ell_k}. \end{equation}

Denote $\ell _0 (>0)$ as the constant given in proposition 2.1 corresponding to the half space $\Sigma _{\rho _0+1}$. Combining the remark about the monotonicity of $\ell$ with respect to $\lambda$ below proposition 2.1, (4.13) and the fact that $\varepsilon _k\to 0$, we have that

(4.14)\begin{equation} x_k\in B_{R_0}(0)\cap \Sigma_{\rho_0-\frac{\ell_0}{2}}. \end{equation}

If $\rho _0-\frac {\ell _0}{2}\leq 0$, then (4.14) contradicts with the fact that $x_k\in \mathbb {R}^n_+$. If $\rho _0-\frac {\ell _0}{2}>0$, due to (4.8) and continuity of $\bar {u}$, we know that there exists $\varepsilon '\in (0,\frac {\ell _0}{2})$ such that for any $\varepsilon _k\leq \varepsilon '$ and $\rho \in (\rho _0,\rho _0+\varepsilon _k)$,

\[ \bar{U}_{\rho}(x)\geq 0,\quad x\in \overline{B_{R_0}(0)\cap \Sigma_{\rho_0-\frac{\ell_0}{2}}}. \]

This contradicts with (4.14) and (4.10). Hence, we derive that for any $\rho \in (\rho _0, \rho _0+\varepsilon ')$ with $\varepsilon '>0$ small enough,

\[ \bar{U}_\rho(x)\geq 0,\quad x\in \Sigma_\rho. \]

Similarly, we may verify that there exists $\varepsilon ''>0$ such that for any $\rho \in (\rho _0, \rho _0+\varepsilon '')$ the inequality holds

\[ \bar{V}_\rho(x)\geq 0,\quad x\in \Sigma_\rho. \]

Let $\varepsilon =\min \{\varepsilon ',\varepsilon ''\}$, then (4.9) follows immediately and hence (4.7) holds, which yields that $\rho _0=+\infty$.

The result $\rho _0=+\infty$ indicates that both $\bar {u}(x)$ and $\bar {v}(x)$ are monotone increasing along the $x_n$-axis. This contradicts with the asymptotic behaviours (4.1). Therefore, $(\bar {u},\bar {v})=(0,0)$, which yields that $(u,v)=(0,0)$. We complete the proof of theorem 1.3.