On a critical time-harmonic Maxwell equation in nonlocal media

Abstract

In this paper, we study the existence of solutions for a critical time–harmonic Maxwell equation in nonlocal media

\[ \begin{cases} \nabla\times(\nabla\times u)+\lambda u=\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)|u|^{2^{{\ast}}_{\alpha}-2}u & \mathrm{in}\ \Omega,\\ \nu\times u=0 & \mathrm{on}\ \partial\Omega, \end{cases} \]

where $\Omega \subset \mathbb {R}^{3}$ is a bounded domain, either convex or with $\mathcal {C}^{1,1}$ boundary, $\nu$ is the exterior normal, $\lambda <0$ is a real parameter, $2^{\ast }_{\alpha }=3+\alpha$ with $0<\alpha <3$ is the upper critical exponent due to the Hardy–Littlewood–Sobolev inequality. By introducing some suitable Coulomb spaces involving curl operator $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$, we are able to obtain the ground state solutions of the curl–curl equation via the method of constraining Nehari–Pankov manifold. Correspondingly, some sharp constants of the Sobolev-like inequalities with curl operator are obtained by a nonlocal version of the concentration–compactness principle.

1. Introduction and main results

1.1. Introduction

Let $\Omega \subset \mathbb {R}^{3}$ be a bounded domain, we are concerned with the curl–curl equation

(1.1)\begin{equation} \begin{cases} \nabla\times(\nabla\times u)+\lambda u=f(x,u) & \mathrm{in}\ \Omega,\\ \nu\times u=0 & \mathrm{on}\ \partial\Omega, \end{cases} \end{equation}

where $\lambda <0$ is a real parameter, $\nu :\partial \Omega \longrightarrow \mathbb {R}^{3}$ is the exterior normal. Equation (1.1) can be derived from the first order Maxwell equation [35]

(1.2)\begin{equation} \begin{cases} \nabla\times \mathcal{H}=\mathcal{J}+\partial_{t}\mathcal{D}, & (\mathrm{Ampere's\ circuital\ law})\\ div(\mathcal{D})=\varrho, & (\mathrm{Gauss's\ law})\\ \partial_{t}\mathcal{B}+\nabla\times \mathcal{E}=0, & (\mathrm{Faraday's\ law\ of\ induction})\\ div(\mathcal{B})=0, & (\mathrm{Gauss's\ law\ for\ magnetism}) \end{cases} \end{equation}

where $\mathcal {E},\mathcal {H},\mathcal {D},\mathcal {B}$ are corresponded to the electric field, magnetic induction, electric displacement and magnetic filed, respectively. $\mathcal {J}$ is the electric current intensity, and $\varrho$ is the electric charge density. Generally, these physical quantities satisfy the following constitutive equations (see [14, § 1.1.3]):

(1.3)\begin{equation} \mathcal{\mathcal{J}}=\sigma \mathcal{E},~\mathcal{D}=\varepsilon \mathcal{E}+\mathcal{P}_{NL}, \mathcal{H}=\frac{1}{\mu}\mathcal{B}-\mathcal{M}, \end{equation}

where $\mathcal {P}_{NL}, \mathcal {M}$ denote the polarization field and magnetization filed respectively, $\varepsilon,\mu,\sigma$ are the electric permittivity, magnetic permeability and the electric conductivity . Taking the special case with the absence of charges, currents and magnetization, namely, $\mathcal {J}=\mathcal {M}=0$, $\rho =0$, equation (1.2) becomes the second curl-curl equation

(1.4)\begin{equation} \nabla\times(\frac{1}{\mu}\nabla\times\mathcal{E})+\varepsilon\partial_{t}^{2}\mathcal{E}={-}\partial_{t}^{2}\mathcal{P}_{NL}. \end{equation}

As the electric field and polarization field are time harmonic with the ansatz $\mathcal {E}(x,t)=E(x)\,{\rm e}^{i\omega t}$, $\mathcal {P}_{NL}(x,t)=P(x)\,{\rm e}^{i\omega t}$, equation (1.4) turns into the time-harmonic Maxwell equation

\[ \nabla\times(\frac{1}{\mu}\nabla\times E)-\varepsilon \omega^{2}E=\omega^{2} P. \]

In some Kerr-like medias, the polarization field function $\mathcal {P}_{NL}$ is usually chosen to be $\mathcal {P}_{NL}=\alpha (x)|\mathcal {E}|^{p-2}\mathcal {E}$ with $2\leq p\leq 6$ for the purpose of simplifying the model. Then by setting

\[ f(x,E)=\partial_{E}F(x,E)=\mu\omega^{2}\alpha(x)|E|^{p-2}E, \]

one can deduce the main equation (1.1)

\[ \nabla\times(\nabla\times E)+\lambda E=f(x,E), \]

where $\lambda =-\mu \omega ^{2}\varepsilon$. The boundary condition holds when $\Omega$ is surrounded by a perfect conductor, see [14].

Apparently, equation (1.1) has a variational structure and the solutions are the critical points of the functional

(1.5)\begin{equation} J_{\lambda}(u)=\int_{\Omega}|\nabla\times u|^{2}\,{\rm d}x+\frac{\lambda}{2}\int_{\Omega}|u|^{2}\,{\rm d}x-\int_{\Omega}F(x,u)\,{\rm d}x, \end{equation}

which is well defined on the natural space

\[ X=W^{p}_{0}(\mathrm{curl};\Omega)=\overline{\mathcal{C}_{0}^{\infty}(\Omega,\mathbb{R}^{3})}^{||\cdot||_{W^{p}(\mathrm{curl};\Omega)}}, \]

where

\[ W^{p}(\mathrm{curl};\Omega)=\{u\in L^{p}(\Omega,\mathbb{R}^{3}):\nabla\times u\in L^{2}(\Omega,\mathbb{R}^{3})\} \]

is a Banach space, see [5]. By introducing the Helmholtz decomposition

\[ W^{p}_{0}(\mathrm{curl};\Omega)=X_{\Omega}\oplus X^c_{\Omega}, \]

where

\begin{align*} X_{\Omega}:& =\{v\in W_{0}^{p}(\mathrm{curl};\Omega):\int_{\Omega}\langle v,\varphi\rangle\,{\rm d}x=0\ \mathrm{for\ every}\ \varphi\\ & \quad \in C^{\infty}_{0}(\Omega,\mathbb{R}^{3})\ \mathrm{with}\ \nabla\times\varphi=0\}\\ & =\{v\in W_{0}^{p}(\mathrm{curl};\Omega):\mathrm{div}(v)=0\ \mathrm{in\ the\ sense\ of\ distributions}\}, \end{align*}

and

\[ X^c_{\Omega}:=\{w\in W^{p}_{0}(\mathrm{curl};\Omega):\int_{\Omega}\langle w,\nabla\times\varphi\rangle\,{\rm d}x=0\ \mathrm{for all}\ \varphi\in C^{\infty}_{0}(\Omega,\mathbb{R}^{3})\}, \]

functional (1.5) can be rewritten as

\begin{align*} J_{\lambda}(u)=J_{\lambda}(v+w)& =\frac{1}{2}\int_{\Omega}|\nabla\times v|^{2}\,{\rm d}x+\frac{\lambda}{2}\int_{ \Omega}|v+w|^{2}\,{\rm d}x-\int_{\Omega}F(x,v+w)\,{\rm d}x\\ & =\frac{1}{2}\int_{\Omega}|\nabla v|^{2}\,{\rm d}x+\frac{\lambda}{2}\int_{ \Omega}|v+w|^{2}\,{\rm d}x-\int_{\Omega}F(x,v+w)\,{\rm d}x, \end{align*}

where $\nabla \times (\nabla \times v)=\nabla (\nabla \cdot v)-\nabla \cdot (\nabla v)=-\Delta v$ for $\mathrm {div}(v)=0$. Since the operator $\nabla \times (\nabla \times \cdot )$ has an infinite dimension kernel, i.e. $\nabla \times (\nabla \varphi )=0$ for $\varphi \in \mathcal {C}^{\infty }_{0}(\Omega )$, one can easily check that $J_{\lambda }$ has the strongly indefinite nature. Particularly, set

\[ \widetilde{\lambda}=\mu(\Omega)^{-\frac{p-2}{p}}p^{-\frac{2}{p}}\mathop{\mathrm{inf}}\limits_{v\in X_{\Omega}:|v|^{p}=1}\int_{\Omega}|\nabla\times v|^{2}\,{\rm d}x>0, \]

then $J_{\lambda }$ has a linking geometry as $\lambda \leq \widetilde {\lambda }$, see [31].

To overcome the difficulty of the strong indefiniteness, by assuming that the additional condition $\nabla \cdot u=0$, then the curl–curl operator become the classical Laplace operator. Moreover, if consider the Dirichlet boundary condition, it becomes the classical elliptic equation

(1.6)\begin{equation} \begin{cases} -\Delta u+\lambda u=f(x,u) & \mathrm{in}\ \Omega,\\ u=0 & \mathrm{on}\ \partial\Omega. \end{cases} \end{equation}

This elliptic equation has been widely studied in different dimensional spaces and topology regions, see the pioneering work of Brezis and Nirenberg [10] and the more references [11, 12, 19]. Essentially, the non-divergence condition is a Coulomb gauge condition, which holds in the gauge invariant field. This requires that the polarization field $\mathcal {P}_{NL}$ does not happen or linearly depends on $\mathcal {E}$, otherwise, it destroys the gauge invariance of the curl–curl equation. If the polarization field $\mathcal {P}_{NL}=0$, then the curl–curl equation becomes a linear time harmonic Maxwell equation, which has been extensively considered in [14, 35, 39]. Physically there indeed exist some special cylindrically symmetric transverse electric and transverse magnetic which satisfy the non-divergence condition, and they have been studied by Stuart and Zhou in [43, 44].

For the general case with $\nabla \cdot u\neq 0$, the study of the curl–curl equation becomes much more challenging. The first attempt goes back to the pioneering work of Benci [7]. Under some nonlinear assumptions on $W(t)$, the authors investigated the Born–Infeld static magnetic model

(1.7)\begin{equation} \nabla\times(\nabla\times A)=W'(|A|^{2})A,\quad\mathrm{in}\ \mathbb{R}^{3}, \end{equation}

where $A=\nabla \times B$ is a magnetic potential. In a suitable subspace, Azzollini et al. [2] obtained the cylindrically symmetric solutions of (1.7) by the Palais principle of symmetric criticality. By using the Hodge decomposition, the cylindrically symmetric solutions with a second form have also been constructed by D'Aprile and Siciliano in [13]. In fact, in some bounded domains with cylindrically symmetric, the similar solutions were obtained in [5, 6, 30]. Bartsch, Dohnal and et al. [4] also analysed the spectrum of the curl–curl operator with cylindrically symmetric periodic potential $V(x)=V(r,x_{3})$, and considered the following time-harmonic Maxwell equation

(1.8)\begin{equation} \nabla\times(\nabla\times E)+V(x)E=\Gamma(x)|E|^{p-2}E,\quad \mathrm{in}\ \mathbb{R}^{3}, \end{equation}

where $\Gamma (x)$ is a period function with respect $x_{3}$. By the method of constraining symmetric sub-manifold, the cylindrically symmetric ground state solutions of (1.8) were obtained, one may see [46] for other extended results.

If the problem was set in some non-symmetric bounded domains or some cases with non-symmetric potential, the methods mentioned above do not work well. Moreover, due to the lack of weak–weak$^{\ast }$ continuity of $J^{'}_{\lambda }(u)$ , the abstract linking theorems established in [8, 20, 29] do not work any longer, and so we fail to look for the suitable $(PS)$ sequences. Even if we can obtain the bounded $(PS)$ sequence, we still do not know whether the weak limit is a critical point of the functional. Inspired by the work of Szulkin and Weth in [45], Bartsch and Mederski[5] constructed a Nehari–Pankov manifold, which is homeomorphism with the upper unit ball of the subspace $X_{\Omega }$, and in where, the $(PS)$ sequence is obtained by the Ekeland variational principle. On the other hand, by the compact embedding

(1.9)\begin{equation} X_{\Omega}\hookrightarrow L^{p}(\Omega,\mathbb{R}^{3}),\quad 2\leq p<6, \end{equation}

they succeeded in verifying the $(PS)_{c}^{\tau }$ condition, see definition 2.20, which implies the weak-weak$^{\ast }$ continuity of $J^{'}_{\lambda }(u)$. We would also like to mention that the convexity assumption of the nonlinearity $f(x,u)$ plays a key role in finding the bounded $(PS)$ sequence, see also [6] for the weakened version. For other related results, we may turn to [40] for the asymptotically linear case and [32] for the case with supercritical growth at $0$ and subcritical growth at infinity.

For the critical case $p=6$, the embedding (1.9) above is not compact any more, then it is rather difficult to verify the $(\mathrm {PS})^{\tau }_{c}$ condition. Mederski [30] proposed a compactly perturbed method and proved that the $(\mathrm {PS})$ sequence contains a weakly convergent subsequence with a nontrivial limit point. Later, Mederski and Szulkin [33] established a general concentration-compactness lemma in $\mathbb {R}^{N}$ and obtained the sharp constant in the curl inequality. As an application, the authors dealt with the Brezis-Nirenberg type problem by an extend skill. In the entire space $\mathbb {R}^{3}$, the embedding above is also not compact, then a new critical point theory related to a new topological manifold has been established by Mederski et al. in [32], there the compactness was recovered and the existence of multiply solutions was obtained. In a direct way, Mederski [29] established a global compactness lemma that accounts for the lack of weak-weak$^{\ast }$ continuity. Moreover, a Pohozaev identity has been established, which gives a criterion for the nonexistence of classical solution. In an earlier work, Bartsch [4] showed that no interesting solution can be leaded under the fully radial symmetry assumption on the potential $V(x)$.

However, for some Kerr-type nonlinear mediums, the material law (1.3) between the electric field $\mathcal {E}$ and the displacement field $\mathcal {D}$ becomes more delicate, see [4, (1.8)],

\[ \mathcal{D}=\epsilon_{0}(n(x)^{2}\mathcal{E}+\mathcal{P}_{NL}(x,\mathcal{E}))\ \mathrm{with}\ \mathcal{P}_{NL}(x,\mathcal{E})=\chi^{(3)}(\mathcal{E}\cdot\mathcal{E})\mathcal{E}, \]

where $n^{2}(x)=1+\chi ^{(1)}(x)$ is the square of the refractive index and $\chi ^{(1)}$, $\chi ^{(3)}$ denote the linear and cubic susceptibilities of the medium respectively. Particularly, in some nonlocal optic materials, the refractive index $n(x)$ is quite dependent on the electric field $\mathcal {E}$ in a small neighbourhood, and the refractive index change $\triangle n$ can be represented in general form as

\[ \triangle n(E)=s\int_{-\infty}^{+\infty}K(x-y)E(x)\,{\rm d}x, \]

see [22, (1)]. This phenomenological model is of great significance in the research of laser beams and solitary waves in nonlocal nematic liquid crystals, see [22, 41] and the reference therein. However, these articles are based on the nonlinear Schrödinger equation, which is an asymptotic approximation of Maxwell's equations. To investigate more information about the electromagnetic waves in the nonlocal optic mediums, one need to deal with the full three-dimensional Maxwell problem. Recently, Mandel [27] investigated the curl–curl equation with nonlocal nonlinearity

\[ \nabla\times(\nabla\times E)+E=(K(x)\ast|E|^{p})|E|^{p-2}E\quad \mathrm{in}\ \mathbb{R}^{3}, \]

and the author proved that nonlocal media may admit ground states even though the corresponding local models do not admit. In there, the parameter ${\lambda =-\mu \omega ^{2}\varepsilon =1}$ with $\varepsilon <0$ is corresponded to the new artificially produced metamaterials with negative reflexive, see [38], and the kernel $K(x)={\rm e}^{-|x|^{2}}$ is a exponent type responding function which expresses the nonlocal polarization of the nonlocal optical media, see [22, 37] for more cases with oscillatory kernel functions. What's more, nonlocality appears naturally in optical systems with a thermal [26] and it is known to influence the propagation of electromagnetic waves in plasmas [9]. Nonlocality also has attracted considerable interest as a means of eliminating collapse and stabilizing multidimensional solitary waves [3] and it plays an important role in the theory of Bose–Einstein condensation [15] where it accounts for the finite-range many-body interactions.

1.2. Main results

In the present paper, we are interested in the curl–curl equation with critical convolution part, namely, we consider the curl–curl equation with Riesz potential interaction part

\[ \nabla\times(\nabla\times E)+\lambda E=(I_{\alpha}(x)\ast|E|^{p})|E|^{p-2}E\quad\mathrm{in}\ \mathbb{R}^{3}, \]

where $I_{\alpha }:\mathbb {R}^{3}\longrightarrow \mathbb {R}$ is the Riesz potential of order $\alpha \in (0,3)$ defined for $x\in \mathbb {R}^{3}\setminus \{0\}$ as

\[ I_{\alpha}(x)=\frac{A_{\alpha}}{|x|^{3-\alpha}},\quad A^{\alpha}=\frac{\Gamma\left(\frac{3-\alpha}{2}\right)}{\Gamma\left(\frac{\alpha}{2}\right)\pi^{\frac{N}{2}}2^{\alpha}}. \]

The choice of normalization constant $A^{\alpha }$ ensures that the kernel $I_{\alpha }$ enjoys the semigroup property

\[ I_{\alpha+\beta}=I_{\alpha}\ast I_{\beta}\ \mathrm{for\ each}\ \alpha,\beta\in(0,3)\ \mathrm{such\ that}\ \alpha+\beta<3, \]

see for example [16, pp. 73-74]. Indeed, the classical elliptic equation with Riesz potential has been widely studied, and it also has a rich physical background and mathematical research value, see [17–19, 34, 36] and the reference therein.

We are going to consider the following Brezis–Nirenberg type problem for the curl–curl equation

(1.10)\begin{equation} \begin{cases} \nabla\times(\nabla\times u)+\lambda u=\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)|u|^{2^{{\ast}}_{\alpha}-2}u & \mathrm{in}\ \Omega,\\ \nu\times u=0 & \mathrm{on}\ \partial\Omega, \end{cases} \end{equation}

where $\Omega \subset \mathbb {R}^{3}$ is a bounded domain, either convex or with $\mathcal {C}^{1,1}$ boundary, $\nu$ is the exterior normal, $\lambda <0$ is a real parameter, $2^{\ast }_{\alpha }=3+\alpha$ with $0<\alpha <3$ is the upper critical exponent in the sense of the following Hardy–Littlewood–Sobolev (HLS for short) inequality, see [24].

Proposition 1.1 Let $t,r\in (1,\infty )$, $\alpha \in (0,N)$ with $\frac {1}{t}+\frac {N-\alpha }{N}+\frac {1}{r}=2$. For $h\in L^{r}(\mathbb {R}^{N},\mathbb {R}^{N}), g\in L^{t}(\mathbb {R}^{N},\mathbb {R}^{N})$, there exists a sharp constant $C(r,t,N,\alpha )$ independent $g$ and $h$ such that

(1.11)\begin{equation} \int_{\mathbb{R}^{N}}(I_{\alpha}\ast |h|)|g|dx\leq C(r,t,N,\alpha)||h||_{L^{r}(\mathbb{R}^{N},\mathbb{R}^{N})}||g||_{L^{t}(\mathbb{R}^{N},\mathbb{R}^{N})}. \end{equation}

If $t=r=\frac {2N}{N+\alpha }$, then there is a equality in (1.11) if and only if $h(x)=Cg(x)$ and

(1.12)\begin{equation} h(x)=A(\gamma^{2}+|x-a|^{2})^{-\frac{N+\alpha}{2}} \end{equation}

for some $A\in \mathbb {C}$, $0\neq \gamma \in \mathbb {R}$ and $a\in \mathbb {R}^{N}$.

In view of the HLS inequality, the functional corresponds to the nonlocal curl–curl equation

(1.13)\begin{equation} J_{\lambda}(u)=\frac{1}{2}\int_{\Omega}|\nabla\times u|^{2}\,{\rm d}x+\frac{\lambda}{2}\int_{ \Omega}|u|^{2}\,{\rm d}x-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x \end{equation}

is well defined on the natural space $W^{2^{\ast }}_{0}(\mathrm {curl};\Omega )$. However, due to the appearance of the convolution part, this space is not good enough for us to prove the coercive property of the functional. Therefore, it is necessary to introduce the Coulomb space

\[ Q^{\alpha,2^{{\ast}}_{\alpha}}(\Omega,\mathbb{R}^{3})=\{u:\Omega\longrightarrow \mathbb{R}^{3}\ \Big|\ \int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x<\infty\}, \]

see definition 2.1 below. Then, we may define the Coulomb space involve curl operator as

\[ W^{\alpha,2^{{\ast}}_{\alpha}}(\mathrm{curl};\Omega)=\{u\in Q^{\alpha,2^{{\ast}}_{\alpha}}(\Omega,\mathbb{R}^{3}):\nabla\times u\in L^{2}(\Omega,\mathbb{R}^{3})\}, \]

which is a Banach space (see lemma 2.5) if provided with the norm

\[ ||u||_{W^{\alpha,2^{{\ast}}_{\alpha}}(\mathrm{curl};\Omega)}:=(||u||_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2}+|\nabla\times u|^{2}_{2})^{1/2}. \]

We also need the following space

\[ W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)=\overline{\mathcal{C}_{0}^{\infty}(\Omega,\mathbb{R}^{3})}^{||\cdot||_{W^{\alpha,2^{{\ast}}_{\alpha}}(\mathrm{curl};\Omega)}}. \]

In this way, we can easily check that the functional (1.13) is well defined on $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$, see lemma 2.7. In order to obtain the Brezis–Lieb lemma in the dual space, we extend the linear functionals of the Coulomb space to a mix-norm space, see proposition 2.2. Correspondingly, to establish the Helmholtz decomposition on the work space $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$ and $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$, see (2.34) and lemma 2.15, we introduce the following subspace

(1.14)\begin{equation} \begin{aligned} \mathcal{V}_{\Omega}: & =\{v\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega):\int_{\Omega}\langle v,\varphi\rangle\,{\rm d}x\\ & \quad =0\ \mathrm{for\ every}\ \varphi\in C^{\infty}_{0}(\Omega,\mathbb{R}^{3})\ \mathrm{with}\ \nabla\times\varphi=0\},\\ \mathcal{W}_{\Omega}: & =\{w\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega):\int_{\Omega}\langle w,\nabla\times\varphi\rangle {\rm d}x=0\ \mathrm{for\ all}\ \varphi\in C^{\infty}_{0}(\Omega,\mathbb{R}^{3})\}\\ & =\{w\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega):\nabla\times w =0\ \mathrm{in\ the\ sense\ of\ distributions}\}. \end{aligned} \end{equation}

Here and below $\langle \cdot,\cdot \rangle$ denote the inner product. Without misunderstanding, we shall write $\mathcal {V}_{\mathbb {R}^{3}}, \mathcal {W}_{\mathbb {R}^{3}}$ if $\Omega =\mathbb {R}^{3}$. Basing on this new decomposition, we can adapt the classical concentration–compactness lemma to suit the new situation. Owing to the concentration-compactness lemma, we obtain the weak–weak$^{\ast }$ continuity of $J'_{\lambda }(u)$ on the Nehari–Pankov manifold [see (4.3)]

\[ \mathcal{N}_{\lambda}:=\{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)\setminus(\widetilde{\mathcal{V}}_{\Omega}\oplus\mathcal{W}_{\Omega}):J'_{\lambda}(u)|_{\mathbb{R}u\oplus\widetilde{\mathcal{V}}_{\Omega}\oplus\mathcal{W}_{\Omega}}=0\}, \]

where $\widetilde {\mathcal {V}}_{\Omega }$ is a subspace of $\mathcal {V}_{\Omega }$ on which the quadratic part of $J_{\lambda }$ (see 4.1) is negative semi-definite. Meanwhile, the concentrated compactness lemma implies the $L^{2}(\Omega,\mathbb {R}^{3})$ convergence for the bounded sequence. This allows us to choose the compactly perturbed functional $J_{cp}=J_{0}=J_{\lambda =0}$ that satisfies the condition $(C1)$ in lemma 2.21. By setting another Nehari–Pankov manifold [see (4.4)]

\[ \mathcal{N}_{cp} = \{E\in (\mathcal{V}_{\Omega}\oplus\mathcal{W}_{\Omega})\setminus\mathcal{W}_{\Omega}: J'_{cp}(u)|_{\mathbb{R}u\oplus\mathcal{W}_{\Omega}}=0\}, \]

and controlling the ground state energy of $J_{\lambda }$ lower than the ground state energy of the perturbed functional $J_{cp}$, i.e.

\[ c_{\lambda}=\mathop{\mathrm{inf}}\limits_{\mathcal{N}_{\lambda}}J_{\lambda}< \mathop{\mathrm{inf}}\limits_{\mathcal{N}_{cp}}J_{cp}=c_{0}, \]

we can obtain the ground state solutions of the curl–curl equation (1.10).

It remains to prove the achievement of $c_{0}$. Actually, for the classical elliptic equation (1.6), the sharp constants corresponded to the infimums of the energy level are only attained provided $\Omega =\mathbb {R}^{3}$, and they are independent on the shape of domain. Moreover, one can use the extremal functions to prove that the Mountain-Pass level is below the level where the compactness holds. Inspired by the local case in [33], we are motivated to investigate the sharp constant of the Sobolev type inequality involving the curl operator on the entire space $\mathbb {R}^{3}$. Let $S_{\mathrm {curl,HL}}=S_{\mathrm {curl,HL}}(\mathbb {R}^{3})$ be the largest constant such that the inequality

(1.15)\begin{equation} \int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x\geq S_{\mathrm{curl,HL}}\mathop{\mathrm{inf}}\limits_{w\in \mathcal{W}_{\mathbb{R}^{3}}}\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u+w|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}. \end{equation}

holds for any $u\in W^{\alpha,2^{\ast }_{\alpha }}(\mathrm {curl};\mathbb {R}^{3})\setminus \mathcal {W}_{\mathbb {R}^{3}}$. Then the achievement of the sharp constant $S_{\mathrm {curl},HL}$ is related to a certain least energy solution of the limiting problem

(1.16)\begin{equation} \nabla\times(\nabla\times u)=\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)|u|^{2^{{\ast}}_{\alpha}-2}u,\ \mathrm{in}\ \mathbb{R}^{3}, \end{equation}

where $u\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$. By setting the functional

(1.17)\begin{equation} J(u)=\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x-\frac{1}{2\cdot 2^{{\ast}}_{\alpha}}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x, \end{equation}

and introducing the following Nehari–Pankov manifold [see (3.1)]

\[ \mathcal{N}:=\left\{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\mathbb{R}^{3})\setminus \mathcal{W}_{\mathbb{R}^{3}}: J'(u)u=0\ \mathrm{and}\ J'(u)|_{\mathcal{W}_{\mathbb{R}^{3}}}=0\right\}, \]

then we have

Theorem 1.2 We have the following two conclusions:

1. (a). $\mathop {\mathrm {inf}}\limits _{\mathcal {N}} J=\frac {2^{\ast }_{\alpha }-1}{2\cdot 2^{\ast }_{\alpha }} S_{\mathrm {curl},HL}^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}$ and is attained. Moreover, if $u\in \mathcal {N}$ and $J(u)=\mathop {\mathrm {inf}}\limits _{\mathcal {N}}J$, then $u$ is a ground state solution to equation (1.16) and equality holds in (1.15) for this $u$. If $u$ satisfies equality (1.15), then there are unique $t>0$ and $w\in \mathcal {W}_{\mathbb {R}^{3}}$ such that $t(u+w)\in \mathcal {N}$ and $J(t(u+w))=\mathop {\mathrm {inf}}\limits _{\mathcal {N}}J$.

2. (b). $S_{\mathrm {curl},HL}> S_{HL}$, where

   (1.18)\begin{equation} S_{HL}:=\mathop{\mathrm{inf}}\limits_{u\in\mathcal{D}^{1,2}(\mathbb{R}^{3},\mathbb{R}^{3})\setminus\{0\}}\frac{\int_{\mathbb{R}^{3}}|\nabla u|^{2}\,{\rm d}x}{\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}}. \end{equation}

Note that $S_{HL}$ is the best constant of the combination of the HLS inequality and the Sobolev inequality, see [17, lemma 1.2] for example. It is not clear that whether the sharp constant $S_{\mathrm {curl},HL}$ is independent on shape of the domain $\Omega$ or not. Therefore, we may define another two constants $S_{\mathrm {curl},HL}(\Omega )$ and $\overline {S}_{\mathrm {curl},HL}(\Omega )$. $S_{\mathrm {curl},HL}(\Omega )$ is the largest possible constant such that the inequality

(1.19)\begin{equation} \int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x\geq S_{\mathrm{curl},HL}(\Omega)\mathop{\mathrm{inf}}\limits_{w\in \mathcal{W}_{\mathbb{R}^{3}}}\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u+w|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}} \end{equation}

holds for any $u\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )\setminus \mathcal {W}_{\mathbb {R}^{3}}$ with a zero extending; $\overline {S}_{\mathrm {curl},HL}(\Omega )$ is another constant such that the inequality

(1.20)\begin{equation} \int_{\Omega}|\nabla\times u|^{2}\,{\rm d}x\geq \overline{S}_{\mathrm{curl},HL}(\Omega)\mathop{\mathrm{inf}}\limits_{w\in \mathcal{W}_{\Omega}}\left(\int_{\Omega}|I_{\alpha/2}\ast|u+w|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}. \end{equation}

holds for any $u\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )\setminus \mathcal {W}_{\Omega }$, and $\overline {S}_{\mathrm {curl},HL}(\Omega )$ is largest with this property. We compare the four constants as follow.

Theorem 1.3 Let $\Omega$ be a bounded domain, either convex or with $\mathcal {C}^{1,1}$ boundary. Then

\[ S_{\mathrm{curl},HL}=S_{\mathrm{curl},HL}(\Omega)\geq \overline{S}_{\mathrm{curl},HL}(\Omega). \]

Unfortunately, we don't have any information about the shape of the solutions of (1.16). Hence, the method of taking cut-off functions and comparing the energy levels does not work well any longer. Inspired by the idea in [30], we are going to investigate the energy levels of the ground states. From [5] we know that the spectrum of the curl–curl operator in $W^{2}_{0}(\mathrm {curl};\Omega )$ consists of the eigenvalue $\lambda _{0}=0$ with infinite multiplicity and of a sequence of eigenvalues

\[ 0<\lambda_{1}\leq\lambda_{1}\leq\lambda_{2}\leq{\cdot}\cdot{\cdot}\leq\lambda_{k}\longrightarrow\infty \]

with finite multiplicity $m(\lambda _{k})\in \mathbb {N}$.

The main results for the existence are as follow:

Theorem 1.4 Suppose $\Omega$ is a bounded domain, either convex or with $\mathcal {C}^{1,1}$ boundary. Let $\lambda \in (-\lambda _{\nu },-\lambda _{\nu -1}]$ for some $\nu \geq 1$. Then $c_{\lambda }>0$ and the following statements hold$:$

1. (a) If $c_{\lambda }< c_{0}$, then there is ground state solution to (1.10) , i.e. $c_{\lambda }$ is attained by a critical point of $J_{\lambda }$. A sufficient condition for this inequality to hold is $\lambda \in (-\lambda _{\nu },-\lambda _{\nu }+\bar {S}_{\mathrm {curl},HL}(\Omega )|\mathrm {diam}\Omega |^{-\frac {3\cdot 2^{\ast }_{\alpha }-\alpha -3}{2^{\ast }_{\alpha }}})$, where $|\mathrm {diam}\Omega |=\mathop {max}\limits _{x,y\in \Omega }|x-y|$.

2. (b) There exists $\varepsilon _{\nu }\geq \bar {S}_{\mathrm {curl},HL}(\Omega )|\ \mathrm {diam}\ \Omega |^{-\frac {3\cdot 2^{\ast }_{\alpha }-\alpha -3}{2^{\ast }_{\alpha }}}$ such that $c_{\lambda }$ is not attained for $\lambda \in (-\lambda _{\nu }+\varepsilon _{\nu },-\lambda _{\nu -1}]$, and $c_{\lambda }=c_{0}$ for $\lambda \in (-\lambda _{\nu }+\varepsilon _{\nu },-\lambda _{\nu -1}]$. We do not exclude that $\varepsilon >\lambda _{\nu }-\lambda _{\nu -1}$, so these intervals may be empty.

3. (c) $c_{\lambda }\longrightarrow 0$ as $\lambda \longrightarrow -\lambda ^{-}_{\nu }$, and the function

   \[ (-\lambda_{\nu},-\lambda_{\nu}+\varepsilon_{\nu}]\cap (-\lambda_{\nu},-\lambda_{\nu-1}]\ni\lambda\mapsto c_{\lambda}\in(0,\infty) \]
   is continuous and strictly increasing.

4. (d) There exist at least $\sharp \{k:-\lambda _{k}\!<\!\lambda \!<\!-\lambda _{k}\!+\!\frac {2^{\ast }_{\alpha }-1}{2\cdot 2^{\ast }_{\alpha }}\bar {S}_{\mathrm {curl},HL}(\Omega )|\ \mathrm {diam}\ \Omega |^{-\frac {3\cdot 2^{\ast }_{\alpha }-\alpha -3}{2^{\ast }_{\alpha }}}\}$ pairs of solutions $\pm u$ to (1.10).

The paper is organized as follow. In §2 we introduce some work spaces on bounded domains and entire space $\mathbb {R}^{3}$, and we adapt the concentration compactness lemma for the curl–curl problem with nonlocal nonlinearities. And an abstract critical point theorem is also recalled in this part for the readers’ convenience. In §3, we show that the sharp constant $S_{\mathrm {curl},HL}$ is attained provided $\Omega =\mathbb {R}^{3}$, and we compare the four constants as we introduced. In the last Section, we are devoted to the proof of theorem 1.4.

2. Preliminaries and variational setting

2.1. Preliminaries

Throughout this paper we assume that $\Omega \subset \mathbb {R}^{3}$ is a bounded domain, either convex or with $\mathcal {C}^{1,1}$ boundary. In some cases $\Omega$ is only required to be a Lipschitz domain, see [33] for more details. We shall look for solutions to problem (1.10) and (1.16) in $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$ and $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$ respectively. Now we are ready to introdcue the definitions of the working spaces.

2.1.1. Coulomb space involving curl operator.

Definition 2.1 [34, definition 1] Let $N\in \mathbb {N}$, $\alpha \in (0,N)$ and $p\geq 1$. We define the Coulomb space $Q^{\alpha,p}(\mathbb {R}^{N},\mathbb {R}^{N})$ as the vector space of measurable functions $u: \mathbb {R}^{N}\longrightarrow \mathbb {R}^{N}$ such that

\[ ||u||_{Q^{\alpha,p}(\mathbb{R}^{N},\mathbb{R}^{N})}=\left(\int_{\mathbb{R}^{N}}|I_{\alpha/2}\ast|u|^{p}|^{2}dx\right)^{\frac{1}{2p}}<{+}\infty. \]

It is not difficult to see that $||\cdot ||_{Q^{\alpha,p}(\mathbb {R}^{N},\mathbb {R}^{N})}$ defines a norm, see proposition 2.1 in [34], and the Coulomb space is complete with respect to this norm. By the same way, we also define $Q^{\alpha,p}(\Omega,\mathbb {R}^{N})$ as the Coulomb space on the bounded domain. For the the dual space of $Q^{\alpha,p}(\mathbb {R}^{N},\mathbb {R}^{N})$, it can be characterized by the following proposition, and it is also adopted in $Q^{\alpha,p}(\Omega,\mathbb {R}^{N})$.

Proposition 2.2 [34, proposition 2.11] Let $T$ be a distribution, then $T\in (Q^{\alpha,p}(\mathbb {R}^{N},\mathbb {R}^{N}))'$ if and only if there exists $G(x,y)\in L^{\frac {2p}{2p-1}}(\mathbb {R}^{N},L^{\frac {p}{p-1}}(\mathbb {R}^{N}))$ such that for every $\varphi \in \mathcal {C}_{0}^{\infty }(\mathbb {R}^{N},\mathbb {R}^{N})$,

\[ \langle T,\varphi\rangle=\int_{\mathbb{R}^{N}}\left(\int_{\mathbb{R}^{N}}G(x,y)I_{\alpha/2}(x-y)^{\frac{1}{p}}\,{\rm d}y\right)\varphi(x)\,{\rm d}x. \]

Proof. By the definition of the Coulomb space $Q^{\alpha,p}(\mathbb {R}^{N},\mathbb {R}^{N})$, one can observe that the map

\[ \mathcal{L}:Q^{\alpha,p}(\mathbb{R}^{N},\mathbb{R}^{N})\longrightarrow L^{2p}(\mathbb{R}^{N},L^{p}(\mathbb{R}^{N})) \]

defined by

\[ \mathcal{L}u(x,y)=(I_{\alpha/2}(x-y))^{\frac{1}{p}}u(y) \]

is a linear isometry from $Q^{\alpha,p}(\mathbb {R}^{N},\mathbb {R}^{N})$ into $L^{2p}(\mathbb {R}^{N},L^{p}(\mathbb {R}^{N}))$. Then any linear functional on $Q^{\alpha,p}(\mathbb {R}^{N})$ can be extended to a linear functional on $L^{2p}(\mathbb {R}^{N},L^{p}(\mathbb {R}^{N}))$. Namely, there exists $G(x,y)\in L^{\frac {2p}{2p-1}}\left (\mathbb {R}^{N},L^{\frac {p}{p-1}}(\mathbb {R}^{N})\right )$ such that

\[ \langle T,\varphi\rangle=\langle G(x,y),\mathcal{L}\varphi\rangle. \]

□

For the Coulomb space involving curl operator, we have the following definition.

Definition 2.3 Let $N=3$, $\alpha \in (0,3)$ and $p\geq 1$. We define the Coulomb space involving curl operator $W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$ as the vector space of functions $u\in Q^{\alpha,p}(\mathbb {R}^{3},\mathbb {R}^{3})$ such that u is weakly differentiable in $\mathbb {R}^{3}$, $\nabla \times u\in L^{2}(\mathbb {R}^{3},\mathbb {R}^{3})$ and

\[ ||u||_{W^{\alpha,p}(\mathrm{curl};\mathbb{R}^{3})}=\left(\int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x+\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u|^{p}|^{2}\,{\rm d}x\right)^{\frac{1}{p}}\right)^{\frac{1}{2}} <{+}\infty. \]

The function $||\cdot ||_{W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})}$ defines a norm in view of the proposition 2.1 in [34]. By the same way, we also define $W^{\alpha,p}(\mathrm {curl};\Omega )$ as the Coulomb space involve curl operator on the bounded spaces, namely,

(2.1)\begin{equation} W^{\alpha,p}(\mathrm{curl};\Omega):=\{u\in Q^{\alpha,p}(\Omega,\mathbb{R}^{3}):\nabla\times u\in L^{2}(\Omega,\mathbb{R}^{3})\}. \end{equation}

We are going to prove that $W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$ and $W^{\alpha,p}(\mathrm {curl};\Omega )$ are Banach spaces. The proof of completeness follows by the same arguments as in the proof of theorem 4.3 in [21] and proposition 2.2 in [34]. The first ingredient is the following Fatou property for locally converging sequences.

Lemma 2.4 Let $N=3$, $\alpha \in (0,3)$ and $p\geq 1$. If $(u_{n})_{n\in \mathbb {N}}$ is a bounded sequence in $W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$ that converges to a function $u:\mathbb {R}^{3}\longrightarrow \mathbb {R}^{3}$ in $L^{1}_{loc}(\mathbb {R}^{3},\mathbb {R}^{3})$, then $u\in W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$,

(2.2)\begin{equation} \int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast| u|^{p}|^{2}\,{\rm d}x\leq\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast| u_{n}|^{p}|^{2}\,{\rm d}x, \end{equation}

and

(2.3)\begin{equation} \int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x\leq\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}\int_{\mathbb{R}^{3}}|\nabla\times u_{n}|^{2}\,{\rm d}x. \end{equation}

Proof. Since $(u_{n})_{n\in \mathbb {N}}\longrightarrow u$ is bounded in $W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$, we have

(2.4)\begin{equation} \int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u_{n}|^{p}|^{2}\,{\rm d}x\leq \infty, \end{equation}

then by the Fatou lemma, we have

(2.5)\begin{equation} \int_{\mathbb{R}^{3}}\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}|I_{\alpha/2}\ast| u_{n}|^{p}|^{2}dx\leq\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast| u_{n}|^{p}|^{2}\,{\rm d}x. \end{equation}

By the Fatou lemma again, we have

(2.6)\begin{equation} I_{\alpha}\ast(\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty} |u_{n}|^{p}) \leq\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty} I_{\alpha}\ast(|u_{n}|^{p}). \end{equation}

Since $(u_{n})_{n\in \mathbb {N}}\longrightarrow u$ in $L^{1}_{loc}(\mathbb {R}^{3},\mathbb {R}^{3})$, for almost every $x\in \mathbb {R}^{3}$, we have

(2.7)\begin{equation} I_{\alpha}\ast(\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty} |u_{n}|^{p})(x) \longrightarrow I_{\alpha}\ast(|u|^{p})(x). \end{equation}

Then (2.2) follows (2.5), (2.6) and (2.7).

We are going to prove (2.3). Define $f$ on $\mathcal {D}(\mathbb {R}^{3},\mathbb {R}^{3})$ by

(2.8)\begin{equation} \langle f,v\rangle=\int_{\mathbb{R}^{3}}u\cdot (\nabla\times v)\,{\rm d}x, \end{equation}

since $u_{n}\longrightarrow u$ in $L^{1}_{loc}(\mathbb {R}^{3},\mathbb {R}^{3})$, we have

(2.9)\begin{align} |\langle f,v\rangle|& =|\int_{\mathbb{R}^{3}}u\cdot (\nabla\times v)\mathrm{d}x|=\mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}|\int_{\mathbb{R}^{3}}u_{n}\cdot (\nabla\times v)\,{\rm d}x|\nonumber\\ & =\mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}|\int_{\mathbb{R}^{3}}(\nabla\times u_{n})\cdot v\,{\rm d}x|\leq \mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}||\nabla\times u_{n}||_{2}\left(\int_{\mathbb{R}^{3}}|v|^{2}\,{\rm d}x\right)^{\frac{1}{2}}, \end{align}

where we use the Cauchy–Schwarz inequality. Since $\mathcal {D}(\mathbb {R}^{3},\mathbb {R}^{3})$ is dense in $L^{2}(\mathbb {R}^{3},\mathbb {R}^{3})$, by the the Hahn-Banach theorem, the distribution $f$ can be continuously extend to a linear functional on $L^{2}(\mathbb {R}^{3},\mathbb {R}^{3})$. Therefore, by the Riesz representation theorem, there exists $F\in L^{2}(\mathbb {R}^{3},\mathbb {R}^{3})$ such that for every $v\in \mathcal {D}(\mathbb {R}^{3},\mathbb {R}^{3})$

(2.10)\begin{equation} \int_{\mathbb{R}^{3}}F\cdot v\,{\rm d}x=\langle f,v\rangle=\int_{\mathbb{R}^{3}}u\cdot (\nabla\times v)\,{\rm d}x. \end{equation}

Setting $\nabla \times u$ as the curl of $u$ in the following distribute sense

(2.11)\begin{equation} \int_{\mathbb{R}^{3}}u\cdot(\nabla\times v)\,{\rm d}x=\int_{\mathbb{R}^{3}}(\nabla\times u)\cdot v\,{\rm d}x, \end{equation}

we can see $F=\nabla \times u\in L^{2}(\mathbb {R}^{3},\mathbb {R}^{3})$ in the weak sense. Choosing $v=\nabla \times u$ we find that

(2.12)\begin{align} \int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x& \leq \mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}||\nabla\times u_{n}||_{2}\left(\int_{\mathbb{R}^{3}}|v|^{2}\mathrm{d}x\right)^{\frac{1}{2}}\nonumber\\ & \leq \mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}||\nabla\times u_{n}||_{2}\left(\int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\mathrm{d}x\right)^{\frac{1}{2}}. \end{align}

Therefore we have

(2.13)\begin{equation} \int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x\leq\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}\int_{\mathbb{R}^{3}}|\nabla\times u_{n}|^{2}\,{\rm d}x. \end{equation}

Lemma 2.5 Let N=3, $\alpha \in (0,3)$ and $p\geq 1$. The normed spaces $W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$ and $W^{\alpha,p}(\mathrm {curl};\Omega )$ are complete.

Proof. Let $(u_{n})_{n\in \mathbb {N}}$ be a Cauchy sequence in $W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$. By the local estimate of the Coulomb energy, $(u_{n})_{n\in \mathbb {N}}$ is also a Cauchy sequence in $L_{loc}^{p}(\mathbb {R}^{3},\mathbb {R}^{3})$. Hence there exists $u\in L_{loc}^{p}(\mathbb {R}^{3},\mathbb {R}^{3})$ such that $(u_{n})_{n\in \mathbb {N}}\longrightarrow u$ in $L_{loc}^{p}(\mathbb {R}^{3},\mathbb {R}^{3})$. In light of lemma 2.4, we conclude that $u\in W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$. Moreover, for every $n\in \mathbb {N}$ the sequence $(u_{n}-u_{m})_{m\in \mathbb {N}}$ converges to $(u_{n}-u)$ in $L_{loc}^{p}(\mathbb {R}^{3},\mathbb {R}^{3})$. Hence, by lemma 2.4 again, we have

\begin{align*} & \mathop{\mathrm{lim\ sup}}\limits_{n\longrightarrow\infty}\left(\int_{\mathbb{R}^{3}}|\nabla\times u_{n}-\nabla\times u|^{2}\,{\rm d}x+\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u_{n}-u|^{p}|^{2}\,{\rm d}x\right)\\ & \leq\mathop{\mathrm{lim\ sup}}\limits_{n\longrightarrow\infty}\mathop{\mathrm{lim\ sup}}\limits_{m\longrightarrow\infty} \left(\int_{\mathbb{R}^{3}}|\nabla\times u_{n}-\nabla\times u_{m}|^{2}\,{\rm d}x+\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u_{n}-u_{m}|^{p}|^{2}\,{\rm d}x\right)\\ & \leq\mathop{\mathrm{lim\ sup}}\limits_{m,n\longrightarrow\infty}\left(\int_{\mathbb{R}^{3}}|\nabla\times u_{n}-\nabla\times u_{m}|^{2}\,{\rm d}x+\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u_{n}-u_{m}|^{p}|^{2}\,{\rm d}x\right)\\ & \leq0. \end{align*}

This implies $W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})$ is complete. The completeness of $W^{\alpha,p}(\mathrm {curl};\Omega )$ can be proved in the same way.

We also define

\[ W^{\alpha,p}_{0}(\mathrm{curl};\Omega)=\mathrm{closure}\ \mathrm{of}\ \mathcal{C}^{\infty}_{0}(\Omega;\mathbb{R}^{3})\ \mathrm{in}\ W^{\alpha,p}(\mathrm{curl};\Omega). \]

If $p$ lies in some suitable range, then the two Coulomb spaces are the same for the case $\Omega =\mathbb {R}^{3}$.

Lemma 2.6 Let $\alpha \in (0,3)$, $\frac {3+\alpha }{3}\leq p\leq 3+\alpha$, then $W^{\alpha,p}(\mathrm {curl};\mathbb {R}^{3})=W^{\alpha,p}_{0}(\mathrm {curl};\mathbb {R}^{3})$.

Proof. Let $\eta _{R}\in \mathcal {C}^{\infty }_{0}(\mathbb {R}^{3})$ be such that $|\nabla \eta _{R}|\leq \frac {2}{R}$ for $R\leq |x|\leq 2R$, $\eta _{R}=1$ for $|x|\leq R$ and $\eta _{R}=0$ for $|x|\geq 2R$. Then for $u=(u_{1},u_{2},u_{3})\in W^{\alpha,p}(\rm {curl},\mathbb {R}^{3})$, we have $\eta _{R}u\longrightarrow u$ in $Q^{\alpha,p}(\mathbb {R}^{3},\mathbb {R}^{3})$ as $R\longrightarrow \infty$. Note that

(2.14)\begin{equation} \nabla\times(\eta_{R}u_{i}) =(\partial_{i}\eta_{R})u_{j}-(\partial_{j}\eta_{R})u_{i}+\eta_{R}(\partial_{i}u_{j}-\partial_{j}u_{i}),\quad i\neq j. \end{equation}

If $p=2$, we have $(\partial _{i}\eta _{R})u_{j}\longrightarrow 0$ in $L^{2}(\mathbb {R}^{3},\mathbb {R}^{3})$ as $\alpha \in (0,3)$. Indeed, we have

\begin{align*} \int_{\mathbb{R}^{3}}(\partial\eta_{R})^{2}u_{j}^{2}\,{\rm d}x& =\int_{R\leq|x|\leq 2R}(\partial\eta_{R})^{2}u_{j}^{2}\,{\rm d}x\leq\left(\frac{2}{R}\right)^{2}\int_{R\leq|x|\leq 2R}u_{j}^{2}\,{\rm d}x\\ & \quad\leq\left(\frac{2}{R}\right)^{2}\int_{|x|\leq 2R}u_{j}^{2}\,{\rm d}x\\ & \leq CR^{{-}2}(2R)^{\frac{3-\alpha}{2}}\left(\int_{|x|\leq 2R}|I_{\alpha/2}\ast|u|^{2}|^{2}\mathrm{d}x\right)^{\frac{1}{2}}. \end{align*}

If $p\neq 2$, let $q$ be such that $\frac {1}{p}+\frac {1}{q}=\frac {1}{2}$, then applying the Hölder inequality we have

\begin{align*} \int_{\mathbb{R}^{3}}(\partial\eta_{R})^{2}u_{j}^{2}\,{\rm d}x& \leq\left(\int_{R\leq|x|\leq 2R}|\partial_{i}\eta_{R}|^{q}\,{\rm d}x\right)^{\frac{2}{q}}\left(\int_{R\leq|x|\leq 2R}|u_{j}|^{p}\,{\rm d}x\right)^{\frac{2}{p}}\\ & \leq C_{1}(R^{3-q})^{\frac{2}{q}}(2R^{\frac{3-\alpha}{2}})^{\frac{2}{p}}\left(\int_{|x|\leq 2R}|I_{\frac{\alpha}{2}}\ast|u|^{p}|^{2}\,{\rm d}x\right)^{\frac{1}{p}}\\ & \leq C_{2}R^{\frac{p-(3+\alpha)}{p}}\left(\int_{|x|\leq 2R}|I_{\alpha/2}\ast|u|^{p}|^{2}\,{\rm d}x\right)^{\frac{1}{p}}. \end{align*}

Then, for $p\leq 3+\alpha$, we have $(\partial _{i}\eta _{R})u_{j}\longrightarrow 0$ in $L^{2}(\mathbb {R}^{3},\mathbb {R}^{3})$ as $R\longrightarrow \infty$. As $\partial _{i}u_{j}-\partial _{j}u_{i}\in L^{2}(\mathbb {R}^{3})$, it follows that the left-hand side in (2.14) tends to $\partial _{i}u_{j}-\partial _{j}u_{i}$ in $L^{2}(\mathbb {R}^{3})$ as $R\longrightarrow \infty$. Hence $\eta _{R}u\longrightarrow u$ in $W^{\alpha,p}(\rm {curl};\mathbb {R}^{3})$ and functions of compact support are dense in $W^{\alpha,p}(\rm {curl};\mathbb {R}^{3})$. The rest of the proof is similar to the [33, lemma 2.1].

Lemma 2.7

1. (i) $J_{\lambda }(u)$ and $J(u)$ are well defined on $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$ and $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$ respectively.

2. (ii) Let

   \[ G(u)(x,y)=\left(\frac{1}{|x-y|^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}N-\frac{2\cdot2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}\alpha}}\right)|u(y)|^{2^{{\ast}}_{\alpha}}|u(x)|^{2^{{\ast}}_{\alpha}-1}. \]
   Then, $G(u)(x,y)\in L^{\frac {2\cdot 2^{\ast }_{\alpha }}{2\cdot 2^{\ast }_{\alpha }-1}}(\mathbb {R}^{3};L^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}(\mathbb {R}^{3}))$.

3. (iii) $J_{\lambda }(u)$ and $J(u)$ are of class $\mathcal {C}^{1}$.

Proof.

1. (i) From the definition of $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$ and $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$ , we know that the functionals $J_{\lambda }(u)$ and $J(u)$ are well defined.

2. (ii) Set

   \[ I(u)=\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x. \]
   We claim that $I'(u)\in (Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3}))'$. Indeed, for any $\varphi \in Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$, we have
   (2.15)\begin{align} \langle I'(u),\varphi\rangle& =\int_{\mathbb{R}^{3}}(I_{\alpha} \ast|u|^{2^{{\ast}}_{\alpha}})|u|^{2^{{\ast}}_{\alpha}-2}u\cdot\varphi\,{\rm d}x\leq\left(\int_{\mathbb{R}^{3}}(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}})|u|^{2^{{\ast}}_{\alpha}}\mathrm{d}x\right)^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}} \nonumber\\ & \qquad \cdot\left(\int_{\mathbb{R}^{3}}(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}})|\varphi|^{2^{{\ast}}_{\alpha}}\mathrm{d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & =||u||^{\frac{2^{{\ast}}_{\alpha}-1}{2}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\cdot\left(\int_{\mathbb{R}^{3}}(I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}})(I_{\alpha/2}\ast|\varphi|^{2^{{\ast}}_{\alpha}}\mathrm{d}x)\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \leq||u||^{\frac{2^{{\ast}}_{\alpha}-1}{2}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\cdot\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2\cdot2^{{\ast}}_{\alpha}}} \nonumber\\ & \qquad \cdot\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|\varphi|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2\cdot2^{{\ast}}_{\alpha}}}\nonumber\\ & =||u||^{\frac{2^{{\ast}}_{\alpha}+1}{2}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\cdot ||\varphi||_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}. \end{align}
   Then, by the definition of the functional space on Coulomb space, we have $I'(u)\in (Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3}))'$.

   On the other hand, $G(u)(x,y)$ obviously satisfies that

   (2.16)\begin{equation} \langle I'(u),\varphi\rangle=\int_{\mathbb{R}^{3}}\left(\int_{\mathbb{R}^{3}}G(u)(x,y)(I_{\alpha/2}(x-y))^{\frac{1}{2^{{\ast}}_{\alpha}}}\,{\rm d}y\right)\varphi(x)\,{\rm d}x, \end{equation}
   Therefore, by proposition 2.2, we have $G(u)(x,y)\in L^{\frac {2\cdot 2^{\ast }_{\alpha }}{2\cdot 2^{\ast }_{\alpha }-1}}(\mathbb {R}^{3};L^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}(\mathbb {R}^{3}))$.

3. (iii) We are going to show that $I'(u)$ is continuous. For any sequences $u_{n}$, $u$ and $\varphi \in Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$, we have

   (2.17)\begin{align} & \langle(I'(u_{n})-I'(u)),\varphi\rangle\nonumber\\ & =\int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast|u_{n}|^{2^{{\ast}}_{\alpha}}\right)|u_{n}|^{2^{{\ast}}_{\alpha}-2}u_{n}\cdot\varphi\,{\rm d}x-\int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)|u|^{2^{{\ast}}_{\alpha}-2}u\cdot\varphi\,{\rm d}x\nonumber\\ & =\int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast(|u_{n}|^{2^{{\ast}}_{\alpha}}-|u|^{2^{{\ast}}_{\alpha}})\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\left(I_{\alpha}\ast(|u_{n}|^{2^{{\ast}}_{\alpha}}-|u|^{2^{{\ast}}_{\alpha}})\right)^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}|u_{n}|^{2^{{\ast}}_{\alpha}-2}u_{n}\cdot\varphi\,{\rm d}x\nonumber\\ & \quad+\int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}(|u_{n}|^{2^{{\ast}}_{\alpha}-2}u_{n}-|u|^{2^{{\ast}}_{\alpha}-2}u)\cdot\varphi\,{\rm d}x\nonumber\\ & \leq\left(\int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast(|u_{n}|^{2^{{\ast}}_{\alpha}}-|u|^{2^{{\ast}}_{\alpha}}) \right)|\varphi|^{2^{{\ast}}_{\alpha}}dx\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \qquad\cdot \left(\int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast(|u_{n}|^{2^{{\ast}}_{\alpha}}-|u|^{2^{{\ast}}_{\alpha}}) \right)|u_{n}|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \quad+\left(\int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}} \right)|\varphi|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \qquad\cdot \left(\int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}} \right)(|u_{n}|^{2^{{\ast}}_{\alpha}-1}-|u|^{2^{{\ast}}_{\alpha}-1})^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}\,{\rm d}x\right)^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & =A_{1}\cdot A_{2}+A_{3}\cdot A_{4}. \end{align}
   By the semi-group property and Hölder inequality, we have
   (2.18)\begin{align} A_{1}& =\left(\int_{\mathbb{R}^{3}}\left(I_{\alpha/2}\ast(|u_{n}|^{2^{{\ast}}_{\alpha}}-|u|^{2^{{\ast}}_{\alpha}})\right)\left(I_{\alpha/2}\ast|\varphi|^{2^{{\ast}}_{\alpha}}\right) {\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \leq\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast(|u_{n}|^{2^{{\ast}}_{\alpha}}-|u|^{2^{{\ast}}_{\alpha}})|^{2}\,{\rm d}x\right)^{\frac{1}{2\cdot2^{{\ast}}_{\alpha}}}\cdot \left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|\varphi|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2\cdot2^{{\ast}}_{\alpha}}}\nonumber\\ & =B_{1}^{\frac{1}{2\cdot2^{{\ast}}_{\alpha}}}\cdot||\varphi||_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}, \end{align}
   where $B_{1}=\int _{\mathbb {R}^{3}}|I_{\alpha /2}\ast (|u_{n}|^{2^{\ast }_{\alpha }}-|u|^{2^{\ast }_{\alpha }})|^{2}dx$. Recalling the mean value theorem, and noting that $|\xi |=|u|+\theta |u_{n}-u|$ for some $\theta \in (0,1)$, we have
   (2.19)\begin{align} & ||u_{n}|^{2^{{\ast}}_{\alpha}}-|u|^{2^{{\ast}}_{\alpha}}|=C(2^{{\ast}}_{\alpha})(|u|+\theta|u_{n}-u|)^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|\nonumber\\ & \quad=|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|\quad\mathrm{for}\ 0\leq\theta\leq1. \end{align}
   Therefore, by linearity of the convolution and by positivity of the Riesz–kernel, we deduce that
   (2.20)\begin{align} B_{1}& =\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast(|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)|^{2}dx\nonumber\\ & =\int_{\mathbb{R}^{3}}I_{\alpha}\ast(|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)\cdot (|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)\,{\rm d}x\nonumber\\ & \leq \left( \int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast(|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)\right) |\xi|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \qquad \cdot\left( \int_{\mathbb{R}^{3}}\left(I_{\alpha}\ast(|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)\right) |u_{n}-u|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & =\left( \int_{\mathbb{R}^{3}}\left(I_{\alpha/2}\ast(|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)\right) \cdot \left(I_{\alpha/2}\ast|\xi|^{2^{{\ast}}_{\alpha}}\right){\rm d}x\right)^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \quad\cdot\left( \int_{\mathbb{R}^{3}}\left(I_{\alpha/2}\ast(|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)\right) \cdot \left(I_{\alpha/2}\ast|u_{n}-u|^{2^{{\ast}}_{\alpha}}\right){\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \leq\left( \int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast(|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)|^{2}\,{\rm d}x\right)^{\frac{1}{2}\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \qquad \cdot \left( \int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|\xi|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2}\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \quad\cdot\left( \int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast(|\xi|^{2^{{\ast}}_{\alpha}-1}|u_{n}-u|)|^{2}\,{\rm d}x\right)^{\frac{1}{2}\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \qquad\cdot \left( \int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast(|u_{n}-u|^{2^{{\ast}}_{\alpha}})|^{2}\,{\rm d}x\right)^{\frac{1}{2}\frac{1}{2^{{\ast}}_{\alpha}}}\nonumber\\ & \leq B_{1}^{\frac{1}{2}}\left(||u_{n}||^{2^{{\ast}}_{\alpha}-1}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}+ ||u||^{2^{{\ast}}_{\alpha}-1}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\right) \cdot||u_{n}-u||_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}. \end{align}
   This implies
   (2.21)\begin{equation} B_{1}\leq\left(||u_{n}||^{2(2^{{\ast}}_{\alpha}-1)}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}+ ||u||^{2(2^{{\ast}}_{\alpha}-1)}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\right) \cdot||u_{n}-u||^{2}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}. \end{equation}
   Thus,
   (2.22)\begin{align} A_{1}& \leq \left(||u_{n}||^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}+||u||^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\right)\cdot||u_{n}-u||^{\frac{1}{2^{{\ast}}_{\alpha}}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\nonumber\\ & \qquad \cdot||\varphi||_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}. \end{align}
   Similarly, we have
   (2.23)\begin{equation} \begin{aligned} A_{2} & \leq \left(||u_{n}||^{\frac{(2^{{\ast}}_{\alpha}-1)^{2}}{2^{{\ast}}_{\alpha}}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}+||u||^{\frac{(2^{{\ast}}_{\alpha}-1)^{2}}{2^{{\ast}}_{\alpha}}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\right)\nonumber\\ & \quad \cdot||u_{n}-u||^{\frac{2^{{\ast}}_{\alpha}-1}{2^{{\ast}}_{\alpha}}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\cdot||u_{n}||^{2^{{\ast}}_{\alpha}-1}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\\ A_{3} & \leq||u||_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}||\varphi||_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\\ A_{4} & \leq \left(||u_{n}||^{2^{{\ast}}_{\alpha}-2}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}+||u||^{2^{{\ast}}_{\alpha}-2}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\right)\cdot||u_{n}-u||_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}\nonumber\\ & \quad \cdot||u||^{2^{{\ast}}_{\alpha}-1}_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}. \end{aligned} \end{equation}
   Therefore, for any $u_{n}\longrightarrow u$ in $Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$, we have $\langle (I'(u_{n})-I'(u)),\varphi \rangle \longrightarrow 0$. This implies that $I(u)$ is $\mathcal {C}^{1}$. Therefore, $J_{\lambda }(u)$ and $J(u)$ are of class $\mathcal {C}^{1}$.

To apply the concentration compactness arguments, we need to introduce the following Coulomb–Sobolev space.

Definition 2.8 Let $\Omega \subset \mathbb {R}^{N}$, $\alpha \in (0,N)$ and $p\geq 1$. We define $W^{1,\alpha,p}(\Omega )$ as the scalar space of measurable functions $u:\Omega \longrightarrow \mathbb {R}$ such that $u \in Q^{\alpha,p}(\Omega )$ and $u$ is weakly differentiable in $\Omega$, $Du\in Q^{\alpha,p}(\Omega,\mathbb {R}^{N})$ and

\[ ||u||_{W^{1,\alpha,p}(\Omega)}=\left( \left(\int_{\Omega}|I_{\alpha/2}\ast|u|^{p}|^{2}\,{\rm d}x\right)^{\frac{1}{2}}+\left(\int_{\Omega}|I_{\alpha/2}\ast|Du|^{p}|^{2}\,{\rm d}x\right)^{\frac{1}{2}}\right)^{\frac{1}{p}}<{+}\infty. \]

We are going to prove that the Coulomb–Sobolev space $W^{1,\alpha,p}(\Omega )$ is a Banach space. Firstly, We have the following Fatou property for locally converging sequence.

Lemma 2.9 Let $N\in \mathbb {N}$, $\alpha \in (0,N)$ and $p\geq 1$. If $(u_{n})_{n\in \mathbb {N}}$ is a bounded sequence in $W^{1,\alpha,p}(\Omega )$ such that $u_{n}\longrightarrow u$ in $L^{1}_{loc}(\Omega )$ and $Du_{n}\longrightarrow g$ in $L^{1}_{loc}(\Omega,\mathbb {R}^{N})$, then $g=Du$ and $u\in W^{1,\alpha,p}(\Omega )$,

(2.24)\begin{equation} \int_{\Omega}|I_{\alpha/2}\ast| u|^{p}|^{2}\,{\rm d}x\leq\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}\int_{\Omega}|I_{\alpha/2}\ast| u_{n}|^{p}|^{2}\,{\rm d}x, \end{equation}

and

(2.25)\begin{equation} \int_{\Omega}|I_{\alpha/2}\ast|Du|^{p}|^{2}\,{\rm d}x\leq\mathop{\mathrm{lim\ inf}}\limits_{n\longrightarrow\infty}\int_{\Omega}|I_{\alpha/2}\ast|Du_{n}|^{p}|^{2}\,{\rm d}x. \end{equation}

Proof. The proof of (2.24) follows the same argument in the proof of (2.2). We are going to prove (2.25). For $v\in \mathcal {C}_{0}^{\infty }(\Omega )$ we conclude that

(2.26)\begin{equation} \int_{\Omega}u_{n}\ \mathrm{div}(v)\,{\rm d}x={-}\int_{\Omega}\nabla u_{n}\cdot v \,{\rm d}x. \end{equation}

Since $u_{n}\longrightarrow u$ in $L^{1}_{loc}(\Omega )$, we have

(2.27)\begin{equation} \int_{\Omega}u_{n}\ \mathrm{div}(v)\,{\rm d}x\longrightarrow\int_{\Omega}u\ \mathrm{div}(v)\,{\rm d}x. \end{equation}

Since $Du_{n}\longrightarrow g$ in $L^{1}_{loc}(\Omega,\mathbb {R}^{N})$, we have

(2.28)\begin{equation} -\int_{\Omega}\nabla u_{n}\cdot v\,{\rm d}x\longrightarrow -\int_{\Omega}g\cdot v\,{\rm d}x. \end{equation}

Setting $Du$ as the weak derivation of $u$ in the following distribute sense

(2.29)\begin{equation} \int_{\Omega}u\ \mathrm{div}(v)\,{\rm d}x={-}\int_{\Omega}Du\cdot v\,{\rm d}x, \end{equation}

we can see $g=Du\in L^{p}(\Omega,\mathbb {R}^{N})$ in the weak sense, and $Du_{n}\longrightarrow Du$ in $L^{1}_{loc}(\Omega,\mathbb {R}^{N})$. Based on this fact, we can obtain (2.25) by the same analysis in the proof of (2.2).

Lemma 2.10 Let $N\in \mathbb {N}$, $\alpha \in (0,N)$ and $p\geq 1$. The normed space $W^{1,\alpha,p}(\Omega )$ is complete.

Proof. Let $(u_{n})_{n\in \mathbb {N}}$ be a Cauchy sequence in $W^{1,\alpha,p}(\Omega )$. By the local estimate of the Coulomb energy, $(u_{n})_{n\in \mathbb {N}}$ and $(Du_{n})_{n\in \mathbb {N}}$ are also the Cauchy sequences in $L_{loc}^{p}(\Omega )$. Hence there exists $u\in L_{loc}^{p}(\Omega )$ such that $(u_{n})_{n\in \mathbb {N}}\longrightarrow u$ in $L_{loc}^{p}(\Omega )$ and $g\in L_{loc}^{p}(\Omega,\mathbb {R}^{N})$ such that $(Du_{n})_{n\in \mathbb {N}}\longrightarrow g$ in $L_{loc}^{p}(\Omega,\mathbb {R}^{N})$. In light of lemma 2.9, we conclude that $u\in W^{1,\alpha,p}(\Omega )$. Moreover, for every $n\in \mathbb {N}$ the sequence $(u_{n}-u_{m})_{m\in \mathbb {N}}$ converges to $(u_{n}-u)$ in $L_{loc}^{p}(\Omega )$. Hence, by lemma 2.9 again, we have

\begin{align*} & \mathop{\mathrm{lim\ sup}}\limits_{n\longrightarrow\infty}\left(\int_{\Omega}|I_{\alpha/2}\ast|Du_{n}-Du|^{p}|^{2}\,{\rm d}x+\int_{\Omega}|I_{\alpha/2}\ast|u_{n}-u|^{p}|^{2}\,{\rm d}x\right)\\ & \leq\mathop{\mathrm{lim\ sup}}\limits_{n\longrightarrow\infty}\mathop{\mathrm{lim\ sup}}\limits_{m\longrightarrow\infty}\left(\int_{\Omega}|I_{\alpha/2}\ast|Du_{n}-Du_{m}|^{p}|^{2}\,{\rm d}x+\int_{\Omega}|I_{\alpha/2}\ast|u_{n}-u_{m}|^{p}|^{2}\,{\rm d}x\right)\\ & \leq\mathop{\mathrm{lim\ sup}}\limits_{m,n\longrightarrow\infty}\left(\int_{\Omega}|I_{\alpha/2}\ast|Du_{n}-Du_{m}|^{p}|^{2}\,{\rm d}x+\int_{\Omega}|I_{\alpha/2}\ast|u_{n}-u_{m}|^{p}|^{2}\,{\rm d}x\right)\\ & \leq0. \end{align*}

This implies $W^{1,\alpha,p}(\Omega )$ is complete.

We show that the Coulomb–Sobolev space $W^{1,\alpha,p}(\Omega )$ can be naturally identified with the completion of the set of the test functions $\mathcal {C}_{0}^{\infty }(\Omega )$ under the norm $||\cdot ||_{W^{1,\alpha,p}}$.

Lemma 2.11 Let $N\in \mathbb {N}$, $\alpha \in (0,N)$ and $p\geq 1$. The space of test function $\mathcal {C}_{0}^{\infty }(\Omega )$ is dense in $W^{1,\alpha,p}(\Omega )$.

Proof. Since the test function $\mathcal {C}_{0}^{\infty }(\Omega )$ is dense in $Q^{\alpha,p}(\Omega )$, see proposition 2.6 in [34], then, by lemma 2.9 the conclusion also holds in $W^{1,\alpha,p}(\Omega )$.

Similar to the Poincaré inequality for the local case, we have the following Poincaré inequality for the nonlocal case.

Lemma 2.12 For all $N\in \mathbb {N}$ and $\alpha \in (0,N)$, there exist $p\in (\frac {N-\alpha }{2},\infty )$ if $\alpha \in (0,N-2)$, while $p\in [N,\infty )$ if $\alpha \in [N-2,N)$, such that for every $a\in \Omega$ and $\rho >0$

\[ \int_{B_{\rho}(a)}|I_{\alpha/2}\ast|u|^{p}|^{2}\,{\rm d}x\leq C\rho^{\frac{N-\alpha}{2}}\left(\int_{B_{\rho}(a)}|I_{\alpha/2}\ast|Du|^{p}|^{2}\,{\rm d}x\right)^{\frac{1}{p}}. \]

Proof. By the HLS inequality (1.11), we have

(2.30)\begin{equation} \int_{B_{\rho}(a)}|I_{\alpha/2}\ast|u|^{p}|^{2}\,{\rm d}x\leq C_{1}(\alpha,\rho,p,N)\left(\int_{B_{\rho}(a)}|u(x)|^{\frac{2Np}{N+\alpha}}\,{\rm d}x\right)^{\frac{N+\alpha}{N}}. \end{equation}

If $\alpha \in (0,N-2)$ and $p\in (\frac {N-\alpha }{2},N)\subset (1,N)$, then we have

(2.31)\begin{align} \left(\int_{B_{\rho}(a)}|u(x)|^{\frac{2Np}{N+\alpha}}\,{\rm d}x\right)^{\frac{N+\alpha}{N}}& \leq C_{2}(\alpha,\rho,p,N)\left(\int_{B_{\rho}(a)}|u(x)|^{\frac{Np}{N-p}}\,{\rm d}x\right)^{\frac{N-p}{Np}}\nonumber\\ & \leq C_{3}(\alpha,\rho,p,N)\left(\int_{B_{\rho}(a)}|Du|^{p}\,{\rm d}x\right)^{\frac{2}{p}}. \end{align}

On the other hand, if $\alpha \in (0,N-2)$ and $p\in [N,\infty )$, we know there exists $h\in [\frac {2Np}{N+\alpha +2p},N)$ such that

(2.32)\begin{align} \left(\int_{B_{\rho}(a)}|u(x)|^{\frac{2Np}{N+\alpha}}\,{\rm d}x\right)^{\frac{N+\alpha}{N}}& \leq C_{4}(\alpha,\rho,p,N)\left(\int_{B_{\rho}(a)}|Du|^{h}\,{\rm d}x\right)^{\frac{2}{h}}\nonumber\\ & \leq C_{5}(\alpha,\rho,p,N)\left(\int_{B_{\rho}(a)}|Du|^{p}\,{\rm d}x\right)^{\frac{2}{p}}, \end{align}

where the Hölder inequality was applied. Consequently, for $\alpha \in [N-2,N)$, there also exist $p\in [N,\infty )$ and $h\in [\frac {2Np}{N+\alpha +2p},N)$ such that (2.32) holds.

Then the conclusion follows from (2.30), (2.31), (2.32) and the local estimate of Coulomb energy [34, proposition 2.3], which says that

\[ \left(\int_{B_{\rho}(a)}|Du|^{p}\,{\rm d}x\right)^{\frac{2}{p}}\leq C\rho^{\frac{N-\alpha}{2}}\left(\int_{B_{\rho}(a)}|I_{\alpha/2}\ast|Du|^{p}|^{2}\,{\rm d}x\right)^{\frac{1}{p}}. \]

To establish the Helmholtz decomposition, we also define the following Coulomb–Sobolev space.

Definition 2.13 Let $\Omega \subset \mathbb {R}^{3}$, $\alpha \in (0,3)$ and $p\in (1,\infty )$. We define $W^{1,\alpha,p}_{0}(\mathbb {R}^{3})$ and $W^{1,\alpha,p}_{0}(\Omega )$ as the completion of $C_{0}^{\infty }(\mathbb {R}^{3})$ and $C_{0}^{\infty }(\Omega )$ with respect to the norm

\[ ||w||_{W^{1,\alpha,p}_{0}(\mathbb{R}^{3})}=|\nabla w|_{Q^{\alpha,p}(\mathbb{R}^{3})},\quad ||w||_{W^{1,\alpha,p}_{0}(\Omega)}=|\nabla w|_{Q^{\alpha,p}(\Omega)}. \]

Proposition 2.14 $W^{1,\alpha,p}_{0}(\mathbb {R}^{3})$ is linearly isometric to

\[ \nabla W^{1,\alpha,p}_{0}(\mathbb{R}^{3}):=\{\nabla w\in Q^{\alpha,p}(\mathbb{R}^{3},\mathbb{R}^{3}): w\in W^{1,\alpha,p}_{0}(\mathbb{R}^{3})\}, \]

and $W^{1,\alpha,p}_{0}(\Omega )$ is linearly isometric to

\[ \nabla W^{1,\alpha,p}_{0}(\Omega):=\{\nabla w\in Q^{\alpha,p}(\Omega,\mathbb{R}^{3}): w\in W^{1,\alpha,p}_{0}(\Omega)\}. \]

Proof. Set the map $\nabla :W^{1,\alpha,p}_{0}(\mathbb {R}^{3})\longrightarrow \nabla W^{1,\alpha,p}_{0}(\mathbb {R}^{3})$. Since the Coulomb space is complete, the map is obviously injective and surjective. We also easily check that the map is isometric by the definition of $W^{1,\alpha,p}_{0}(\mathbb {R}^{3})$, this implies our conclusion.

2.1.2 Helmholtz decomposition.

Let $\mathcal {D}^{1,2}(\mathbb {R}^{3},\mathbb {R}^{3})$ denote the completion of $\mathcal {C}^{\infty }_{0}(\mathbb {R}^{3},\mathbb {R}^{3})$ with respect to the norm $|\nabla \cdot |_{2}$. Recall the subspace $\mathcal {V}_{\mathbb {R}^{3}}$ and $\mathcal {W}_{\mathbb {R}^{3}}$ of $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$ in the introduction, we have the following Helmholtz decomposition on $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$.

Lemma 2.15 $\mathcal {V}_{\mathbb {R}^{3}}$ and $\mathcal {W}_{\mathbb {R}^{3}}$ are closed subspaces of $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$ and

(2.33)\begin{equation} W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\mathbb{R}^{3})=\mathcal{V}_{\mathbb{R}^{3}}\oplus\nabla W^{1,\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathbb{R}^{3})=\mathcal{V}_{\mathbb{R}^{3}}\oplus\mathcal{W}_{\mathbb{R}^{3}}.\quad(direct\ sum) \end{equation}

Moreover, $\mathcal {V}_{\mathbb {R}^{3}}\subset \mathcal {D}^{1,2}(\mathbb {R}^{3},\mathbb {R}^{3})$ and the norms $|\nabla \cdot |_{2}$ and $||\cdot ||_{W^{\alpha,2^{\ast }_{\alpha }}(\mathrm {curl};\mathbb {R}^{3})}$ are equivalent in $\mathcal {V}_{\mathbb {R}^{3}}$.

Proof. By the HLS inequality in proposition 1.1, there is a continuous embedding

\[ L^{2^{{\ast}}}(\mathbb{R}^{3},\mathbb{R}^{3})\hookrightarrow Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3}). \]

Then the conclusion follows from the argument in [28, lemma 3.2]. Indeed, Since $W^{1,\alpha,2^{\ast }_{\alpha }}_{0}(\mathbb {R}^{3})$ is a complete space, then $\nabla W^{1,\alpha,2^{\ast }_{\alpha }}_{0}(\mathbb {R}^{3})$ is a closed subspace of $Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$. Moreover $\mathrm {cl}\mathcal {V}_{\mathbb {R}^{3}}\cap \nabla W^{1,\alpha,2^{\ast }_{\alpha }}_{0}(\mathbb {R}^{3})=\{0\}$ in $Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$, hence $\mathcal {V}_{\mathbb {R}^{3}}\cap \nabla W^{1,\alpha,2^{\ast }_{\alpha }}_{0}(\mathbb {R}^{3})=\{0\}$ in $W^{\alpha,2^{\ast }_{\alpha }}(\mathrm {curl};\mathbb {R}^{3})$. In view of the Helmholtz decomposition, and smooth function $\varphi \in \mathcal {C}_{0}^{\infty }(\mathbb {R}^{3},\mathbb {R}^{3})$ can be written as

\[ \varphi=\varphi_{1}+\nabla\varphi_{2} \]

such that $\varphi _{1}\in \mathcal {D}^{1,2}(\mathbb {R}^{3},\mathbb {R}^{3})\cap \mathcal {C}^{\infty }(\mathbb {R}^{3},\mathbb {R}^{3})$, $\mathrm {div}(\varphi _{1})=0$ and $\varphi _{2}\in \mathcal {C}^{\infty }(\mathbb {R}^{3})$ is the Newton potential of $\mathrm {div}(\varphi )$. Since $\varphi$ has compact support, then $\nabla \varphi _{2}\in L^{6}(\mathbb {R}^{3},\mathbb {R}^{3})\subset Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$ and $\varphi _{1}=\varphi -\nabla \varphi _{2}\in \mathcal {V}_{\mathbb {R}^{3}}$. Observe that $\nabla \times \nabla \varphi _{1}=-\triangle \varphi _{1}$, hence

\[ |\nabla\times u|_{2}=|\nabla u|_{2}=||u||_{\mathcal{D}^{1,2}(\mathbb{R}^{3},\mathbb{R}^{3})} \]

for any $u\in \mathcal {V}_{\mathbb {R}^{3}}$. By the Sobolev embedding we have $\mathcal {V}_{\mathbb {R}^{3}}$ is continuously embedded in $L^{6}(\mathbb {R}^{3},\mathbb {R}^{3})$ and by the HLS inequality also in $Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$. Therefore the norms $||\cdot ||_{\mathcal {D}^{1,2}(\mathbb {R}^{3},\mathbb {R}^{3})}$ and $||\cdot ||_{W^{\alpha,2^{\ast }_{\alpha }}}(\mathrm {curl};\mathbb {R}^{3})$ are equivalent on $\mathcal {V}_{\mathbb {R}^{3}}$ and by the density argument we get the decomposition (2.33).

For the bounded domains case, we recall the definition of $\mathcal {V}'_{\Omega }$ in [5], that is

\[ \mathcal{V}'_{\Omega}=\{v\in W^{2}_{0}(\mathrm{curl};\Omega):\int_{\Omega}\langle v,\varphi\rangle\,{\rm d}x=0\ \mathrm{for\ every}\ \varphi\in C^{\infty}_{0}(\Omega,\mathbb{R}^{3})\ \mathrm{with}\ \nabla\times\varphi=0\}. \]

Indeed, if $\varphi$ is supported in a ball, we have $\varphi =\nabla \psi$ for some $\psi \in \mathcal {C}_{0}^{\infty }(\Omega )$, hence we have $\mathrm {div}(v)=0$ for $v\in \mathcal {V}'_{\Omega }$. This implies that

\begin{align*} \mathcal{V}'_{\Omega}& =\{v\in W_{0}^{2}(\mathrm{curl};\Omega):\mathrm{div}(v)=0\ \mathrm{in\ the\ sense\ of\ distributions}\}\\ & \subset \{v\in W^{2}_{0}(\mathrm{curl};\Omega):\mathrm{div}(v)\in L^{2}(\Omega,\mathbb{R}^{3})\}=:X_{N}(\Omega). \end{align*}

Furthermore, since $\Omega$ is a bounded domain, either convex or with $\mathcal {C}^{1,1}$ boundary, $X_{N}(\Omega )$ is continuously embedded in $H^{1}(\Omega,\mathbb {R}^{3})$, see [1]. Therefore in view of the Rellich's theorem $\mathcal {V}'_{\Omega }$ is compactly embedded in $L^{2}(\Omega,\mathbb {R}^{3})$ and continuously in $L^{6}(\Omega,\mathbb {R}^{3})$, so is $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$. This implies in particular that $\mathcal {V}'_{\Omega }\subset \mathcal {V}_{\Omega }$. On the other hand, since $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )\subset W^{2}_{0}(\mathrm {curl};\Omega )$, we have $\mathcal {V}_{\Omega }\subset \mathcal {V}'_{\Omega }$. Therefore, we can see that $\mathcal {V}_{\Omega }=\mathcal {V}'_{\Omega }$ is a Hilbert space with inner product

\[ (v,z)=\int_{\Omega}\langle \nabla\times v,\nabla\times z\rangle\,dx=\int_{\Omega}\langle\nabla v,\nabla z\rangle\,dx. \]

Also, one can easily observe that $\mathcal {V}_{\Omega }$ is a closed linear subspace of $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$. Therefore, by theorem 4.21 (c) in [21], we have the following Helmholtz decomposition

(2.34)\begin{equation} W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)=\mathcal{V}_{\Omega}\oplus\mathcal{W}_{\Omega}.\quad (\mathrm{direct\ sum}) \end{equation}

and that

(2.35)\begin{equation} \int_{\Omega}\langle v,w\rangle\,{\rm d}x=0\ \mathrm{if}\ v\in\mathcal{V}_{\Omega},\ w\in \mathcal{W}_{\Omega}, \end{equation}

which means that $\mathcal {V}_{\Omega }$ and $\mathcal {W}_{\Omega }$ are orthogonal in $L^{2}(\Omega,\mathbb {R}^{3})$. Then the norm

\[ ||v+w||:=((v,v)+|w|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2})^{\frac{1}{2}},~v\in\mathcal{V}_{\Omega},\ w\in \mathcal{W}_{\Omega} \]

is equivalent to $||\cdot ||_{W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )}$.

For the setting of boundary condition, according to [35, theorem 3.33], there is a continuous tangential trace operator $\gamma _{t}:W^{2}(\mathrm {curl};\Omega )\longrightarrow H^{-1/2}(\partial \Omega )$ such that

\[ \gamma_{t}(u)=\nu\times u|_{\partial\Omega}\quad\mathrm{for}\ \mathrm{any}\ u\in C^{\infty}(\overline{\Omega},\mathbb{R}^{3}) \]

and

\[ W^{2}_{0}(\mathrm{curl};\Omega)=\{u\in W^{2}(\mathrm{curl};\Omega):\gamma_{t}(u)=0\}. \]

Hence the vector field $u\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )=\mathcal {V}_{\Omega }\oplus \mathcal {W}_{\Omega }\subset W^{2}_{0}(\mathrm {curl};\Omega )$ satisfies the boundary condition in (1.10).

On the other hand, $\mathcal {W}_{\Omega }$ contains all gradient vectors fields, i.e. $\nabla W^{1,\alpha,2^{\ast }_{\alpha }}_{0}(\Omega )\subset \mathcal {W}_{\Omega }$. However, for some general domains, $\{w\in \mathcal {W}_{\Omega }:\mathrm {div}(w)=0\}$ may be nontrivial (harmonic field) and hence $\nabla W^{1,\alpha,2^{\ast }_{\alpha }}_{0}(\Omega )\subsetneq \mathcal {W}_{\Omega }$, see [6, pp. 4314–4315]. While in the topology domains as we supposed, we have the following conclusion, which is a trivial extended from Lemma 2.3 in [33].

Lemma 2.16 There holds $\mathcal {W}_{\Omega }\!=\!W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )\cap \mathcal {W}_{\mathbb {R}^{3}}\!=\!W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )\cap \nabla W^{1,\alpha,2^{\ast }_{\alpha }} (\Omega )$. If $\partial \Omega$ is connected, then $\mathcal {W}_{\Omega }=\nabla W^{1,\alpha,2^{\ast }_{\alpha }}_{0}(\Omega )$. If $\Omega$ is unbounded, $\mathcal {W}_{\Omega }=W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )\cap \mathcal {W}_{\mathbb {R}^{3}}$ still holds.

2.2. Concentration–compactness lemma

In view of the Helmholtz decomposition, the work space is decomposed into a Hilbert space $\mathcal {V}_{\Omega }$ and a Banach space $\mathcal {W}_{\Omega }$. For a bounded sequence in the work space, one can obtain the a.e convergenc in $\mathcal {V}_{\Omega }$ by the Rellich compactness theorem, which is important to the weak-weak$^{\ast }$ continuity of $J'(u)$. While in the subspace $\mathcal {W}_{\Omega }$, $w_{n}=\nabla p_{n}\rightharpoonup \nabla p=w$ can not deduce the a.e convergenc. By setting the convex nonlinearity satisfied the coercive condition, Merderski [33] connected the subspaces $\mathcal {V}_{\Omega }$ and $\mathcal {W}_{\Omega }$ by the global minimum argument, then the a.e. convergenc on $\mathcal {W}_{\Omega }$ can be recovered by the second concentration–compactness lemma, see Lions [25]. Since the nonlinearity becomes a nonlocal term, we make some minor modifications to the concentration–compactness lemma.

In this subsection, We work in some subspaces of $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$ and $Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$. Let $Z\subset \mathcal {V}_{\Omega }$ be a finite-dimension subspace of $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{N})$ such that $Z\cap \mathcal {W}_{\Omega }=\{0\}$ and put

\[ \widetilde{\mathcal{W}}:=\mathcal{W}_{\Omega}\oplus Z. \]

Correspondingly, in $\mathbb {R}^{3}$, we put $Z=\{0\}$ and $\widetilde {\mathcal {W}}=\mathcal {W}_{\mathbb {R}^{3}}$. For simplicity, we only show the discussion on bounded domains $\Omega$, and the case in the entire space $\mathbb {R}^{3}$ is similar. Note that we always assume that $v\in \mathcal {V}_{\mathbb {R}^{3}}\subset \mathcal {D}^{1,2}(\mathbb {R}^{3},\mathbb {R}^{3})$ but not $\mathcal {V}_{\Omega }$, we then have

Lemma 2.17 Assume $F(u)=(I_{\alpha }\ast |u|^{2^{\ast }_{\alpha }})|u|^{2^{\ast }_{\alpha }}$ and $f(u)=\partial _{u}F(u)$, then $F(u)$ is uniformly strictly convex with respect to $u\in \mathbb {R}^{N}$, i.e. for any compact $A\subset (\mathbb {R}^{3}\times \mathbb {R}^{3})\setminus \{(u,u):u\in \mathbb {R}^{3}\}$

(2.36)\begin{equation} \mathop{\mathrm{inf}}\limits_{x\in\Omega,(u_{1},u_{2})\in A}\left(\frac{1}{2}\left(F(u_{1})+F(u_{2})\right)-F\left(\frac{u_{1}+u_{2}}{2}\right)\right)>0; \end{equation}

Moreover, for any $v\in \mathcal {V}_{\mathbb {R}^{3}}$ we find a unique $\widetilde {w}_{\Omega }(v)\in \widetilde {\mathcal {W}}$ such that

(2.37)\begin{equation} \int_{\Omega}F(v+\widetilde{w}_{\Omega}(v))\,{\rm d}x\leq\int_{\Omega}F(v+\widetilde{w})\,{\rm d}x \quad\mathrm{for}\ \mathrm{all}\ \widetilde{w}\in\widetilde{\mathcal{W}}. \end{equation}

In other word,

(2.38)\begin{equation} \int_{\Omega}\langle f(v+\widetilde{w}),\zeta\rangle\,{\rm d}x=0\ \mathrm{for}\ \mathrm{all}\ \zeta\in\widetilde{\mathcal{W}}\ \mathrm{if}\ \mathrm{and}\ \mathrm{only}\ \mathrm{if}\ \widetilde{w}=\widetilde{w}_{\Omega}(v). \end{equation}

Proof. The uniformly convexity of $F(u)$ follows from the proposition 2.8 in [34]. Now, we prove that $F(u)$ is strictly convex. Set $I(u)=\int _{\mathbb {R}^{3}}F(u)\,{\rm d}x$ and $u(x)=(u_{1},u_{2},u_{3})$, then for any $(s_{1},s_{2},s_{3})\in \mathbb {R}^{3}$ we have

\begin{align*} I(u)& =\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x=\int_{\Omega}\left[I_{\alpha/2}\ast\left(\mathop{\sum}\limits_{i=1}^{3}s_{i}|u_{i}|^{2^{{\ast}}_{\alpha}}\right)\right]^{2}dx \nonumber\\ & =\int_{\Omega}\left[\mathop{\sum}\limits_{i=1}^{3}s_{j}\left(I_{\alpha/2}\ast |u_{i}|^{2^{{\ast}}_{\alpha}}\right)\right]^{2}\,{\rm d}x \end{align*}

Set

\[ g(s_{1},s_{2},s_{3})=\left[\mathop{\sum}\limits_{i=1}^{3}s_{j}\left(I_{\alpha/2}\ast |u_{i}|^{2^{{\ast}}_{\alpha}}\right)\right]^{2}=h(L(s_{1},s_{2},s_{3})), \]

where $h(t)=t^{2}$ ia a strict convex function and

\[ L(s_{1},s_{2},s_{3})=\mathop{\sum}\limits_{i=1}^{3}s_{j}\left(I_{\alpha/2}\ast |u_{i}|^{2^{{\ast}}_{\alpha}}\right). \]

is a linear functional. Then, for each $x\in \mathbb {R}^{3}$, $g(s_{1},s_{2},s_{3})$ is convex.

Indeed, fix $\lambda \in (0,1)$ and $(s_{1},s_{2},s_{3}),(r_{1},r_{2},r_{3})\in \mathbb {R}^{3}$, we have

\begin{align*} & g((1-\lambda)(s_{1},s_{2},s_{3})+\lambda(r_{1},r_{2},r_{3})) =h(L((1-\lambda)(s_{1},s_{2},s_{3})+\lambda(r_{1},r_{2},r_{3})))\\ & =h((1-\lambda)L(s_{1},s_{2},s_{3})\!+\!\lambda L(r_{1},r_{2},r_{3}))\leq(1\!-\!\lambda)h(L(s_{1},s_{2},s_{3}))\!+\!\lambda h(L(r_{1},r_{2},r_{3}))\\ & =(1-\lambda)g(s_{1},s_{2},s_{3})+\lambda g(r_{1},r_{2},r_{3}). \end{align*}

Moreover, since $L$ is an injective function, we deduce that $g$ is strictly convex. Hence, $I(u)$ is strictly convex, so is $F(u)$. On the other hand, $I(u)$ is coercive in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$. Then, by the global minimum theorem, we have (2.37) and (2.38).

Denote the space of finite measures in $\mathbb {R}^{3}$ by $\mathcal {M}(\mathbb {R}^{3})$. Then we have the following concentration–compactness lemma, see [33, lemma 3.1] for the local case.

Lemma 2.18 Assume $F(u)=(I_{\alpha }\ast |u|^{2^{\ast }_{\alpha }})|u|^{2^{\ast }_{\alpha }}$. Suppose $(v_{n})\subset \mathcal {V}_{\mathbb {R}^{3}}$, $v_{n}\rightharpoonup v_{0}$ in $\mathcal {V}_{\mathbb {R}^{3}}$, $v_{n}\longrightarrow v_{0}$ a.e. in $\mathbb {R}^{3}$, $|\nabla v_{n}|^{2}\rightharpoonup \mu$ and $(I_{\alpha }\ast |v_{0}|^{2^{\ast }_{\alpha }})|v_{0}|^{2^{\ast }_{\alpha }}\rightharpoonup \rho$ in $\mathcal {M}(\mathbb {R}^{3})$. Then there exists an at most countable set $I\subset \mathbb {R}^{3}$ and nonnegative weights $\{\mu _{x}\}_{x\in I}$, $\{\rho _{x}\}_{x\in I}$ such that

\[ \mu\geq |\nabla v_{0}|^{2}+\mathop{\Sigma}\limits_{x\in I}\mu_{x}\delta_{x},\quad\rho=\left(I_{\alpha}\ast|v_{0}|^{2^{{\ast}}_{\alpha}}\right)|v_{0}|^{2^{{\ast}}_{\alpha}}+\mathop{\Sigma}\limits_{x\in I}\rho_{x}\delta_{x}, \]

and passing to a subsequence, $\widetilde {w}_{\Omega }(v_{n})\rightharpoonup \widetilde {w}_{\Omega }(v_{0})$ in $\widetilde {\mathcal {W}}$, $\widetilde {w}_{\Omega }(v_{n})\longrightarrow \widetilde {w}_{\Omega }(v_{0})$ a.e. in $\Omega$ and in $L^{p}_{loc}(\Omega )$ for any $1\leq p\leq 2^{\ast }_{\alpha }$.

Remark 2.19 If $\Omega =\mathbb {R}^{3}$, $\widetilde {\mathcal {W}}=\mathcal {W}_{\mathbb {R}^{3}}$, we have the same conclusion, that is $\widetilde {w}_{\mathbb {R}^{3}}(v_{n})\rightharpoonup \widetilde {w}_{\mathbb {R}^{3}}(v_{0})$ in $\widetilde {\mathcal {W}}$, $\widetilde {w}_{\mathbb {R}^{3}}(v_{n})\longrightarrow \widetilde {w}_{\mathbb {R}^{3}}(v_{0})$ a.e. in $\mathbb {R}^{3}$.

Proof. Step 1. Let $\varphi \in \mathcal {C}^{\infty }_{0}(\mathbb {R}^{3})$, then by the definition of $S_{HL}$ in (1.18), we have

\[ \left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|\varphi(v_{n}-v_{0})|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}} S_{HL}\leq \int_{\mathbb{R}^{3}}|\nabla(\varphi (v_{n}-v_{0}))|^{2}\,{\rm d}x. \]

This means that

\begin{align*} & \left(\int_{\mathbb{R}^{3}}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\left(I_{\alpha}\ast|v_{n}-v_{0}|^{2^{{\ast}}_{\alpha}}\right)|(v_{n}-v_{0})|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}} S_{HL}\nonumber\\ & \quad\leq \int_{\mathbb{R}^{3}}|\varphi|^{2}|\nabla(v_{n}-v_{0})|^{2}\,{\rm d}x+o(1). \end{align*}

Using the Brezis-Lieb lemma for the nonlocal case on the left-hand side, see [19, pp. 1226], we then obtain

(2.39)\begin{equation} \left(\int_{\mathbb{R}^{3}}|\varphi|^{2\cdot 2^{{\ast}}_{\alpha}}\,{\rm d}\overline{\rho}\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}S_{HL}\leq \left(\int_{\mathbb{R}^{3}}|\varphi|^{2}\,{\rm d}\overline{\mu}\right)^{1/2}, \end{equation}

where $\overline {\mu }:=\mu -|\nabla v_{0}|^{2}$ and $\overline {\rho }=\rho -(I_{\alpha }\ast |v_{0}|^{2^{\ast }_{\alpha }})|v_{0}|^{2^{\ast }_{\alpha }}$. Set $I=\{x\in \mathbb {R}^{3}:\mu (\{x\}) >0\}$. Since $\mu$ is finite and $\mu,\overline {\mu }$ have the same singular set, $I$ is at most countable and $\mu \geq |\nabla v_{0}|^{2}+\Sigma _{x\in I}\mu _{x}\delta _{x}$. As in the proof of lemma 2.5 in [18] it follows from (2.39) that $\overline {\rho }=\Sigma _{x\in I}\rho _{x}\delta _{x}$. So $\mu$ and $\rho$ are as claimed.

Step 2. To recover the a.e. convergence of the sequence on $\mathcal {W}_{\Omega }$, we consider the global minimum argument which connects $\mathcal {V}_{\mathbb {R}^{3}}$ and $\mathcal {W}_{\Omega }$. Using (2.37) we infer that

(2.40)\begin{equation} |v_{n}+\widetilde{w}_{\Omega}(v_{n})|^{2\cdot2^{{\ast}}_{\alpha}}_{Q^{\alpha,2^{{\ast}}_{\alpha}}}\leq\int_{\Omega}F(v_{n}+\widetilde{w}_{\Omega}(v_{n}))\mathrm{d}x\leq\int_{\Omega}F(v_{n})\,{\rm d}x\leq |v_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2\cdot2^{{\ast}}_{\alpha}}. \end{equation}

Since the right-hand side above is bounded, so is $(|\widetilde {w}_{\Omega }(v_{n})|_{Q^{\alpha,2^{\ast }_{\alpha }}})$. Hence, by the uniform convexity and reflexivity of Coulomb space, see [34, § 2.4.1], up to a subsequence, $\widetilde {w}_{\Omega }(v_{n})\rightharpoonup \widetilde {w}_{0}$ for some $\widetilde {w}_{0}$.

In the following we are going to prove that $\widetilde {w}_{\Omega }(v_{n})\longrightarrow \widetilde {w}_{0}$ a.e. in $\Omega$ after taking subsequence. The convexity of $F$ in $u$ implies that

\[ F\left(\frac{u_{1}+u_{2}}{2}\right)\geq F(u_{1})+\left\langle f(u_{1}),\frac{u_{2}-u_{1}}{2}\right\rangle, \]

applying (2.36), we obtain for any $k\geq 1$ and $|u_{1}-u_{2}|\geq \frac {1}{k}$, $|u_{1}|,|u_{2}|\leq k$ that

(2.41)\begin{equation} m_{k}\leq\frac{1}{2}(F(u_{1})+F(u_{2}))-F\left(\frac{u_{1}+u_{2}}{2}\right)\leq\frac{1}{4}\langle f(u_{1})-f(u_{2}),u_{1}-u_{2}\rangle, \end{equation}

where

\begin{align*} m_{k}& :=\mathop{\mathrm{inf}}\limits_{x\in\Omega,u_{1},u_{2}\in\mathbb{R}^{3}}\frac{1}{2}(F(u_{1})+F(u_{2}))-F\left(\frac{u_{1}+u_{2}}{2}\right)>0\quad\mathrm{for}\ \frac{1}{k}\\ & \quad \leq|u_{1}-u_{2}|,~|u_{1}|,|u_{2}|\leq k. \end{align*}

Now we decompose by $\widetilde {w}_{\Omega }(v_{n})=w_{n}+z_{n}$, $\widetilde {w}_{0}=w_{0}+z_{0}$ where $w_{n}$, $w_{0}\in \mathcal {W}_{\Omega }$ and $z_{n}$, $z_{0}\in Z$. Obviously, since $Z$ is a finite dimension space, we may assume $z_{n}\longrightarrow z_{0}$ in $Z$ and a.e. in $\Omega$. Notice that $v_{n}+\widetilde {w}_{\Omega }(v_{n})$ is bounded in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$, we may introduce

(2.42)\begin{align} \Omega_{n,k}& :=\{x\in\Omega:|v_{n}+\widetilde{w}_{\Omega}(v_{n})-v_{0}-w_{0}-z_{0}|\nonumber\\ & \quad\geq\frac{1}{k}~\mathrm{and}~|v_{n}+\widetilde{w}_{\Omega}(v_{n})|,|v_{0}+w_{0}+z_{0}|\leq k\}. \end{align}

Then, by (2.42) and (2.41), we have

(2.43)\begin{align} & 4m_{k}\int_{\Omega_{n,k}}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\,{\rm d}x\nonumber\\ & \leq\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\langle f(v_{n}+\widetilde{w}_{\Omega}(v_{n}))-f(v_{0}+w_{0}+z_{0}),v_{n}+\widetilde{w}_{\Omega}(v_{n})-v_{0}-w_{0}-z_{0} \rangle\,{\rm d}x\nonumber\\ & =\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\langle f(v_{n}+\widetilde{w}_{\Omega}(v_{n}))-f(v_{0}+w_{0}+z_{0}),v_{n}-v_{0} \rangle\,{\rm d}x\nonumber\\ & \quad+\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\langle f(v_{n}+\widetilde{w}_{\Omega}(v_{n}))-f(v_{0}+w_{0}+z_{0}),\widetilde{w}_{\Omega}(v_{n})-w_{0}-z_{0}\rangle {\rm d}x=I_{1}+I_{2}. \end{align}

Since $|v_{n}+\widetilde {w}_{\Omega }(v_{n})|\leq k$ and $|v_{0}+w_{0}+z_{0}|\leq k$ on $\Omega _{n,k}$, we have $|v_{n}+\widetilde {w}_{\Omega }(v_{n})|\leq C_{1}|v_{n}|$ and $|v_{0}+w_{0}+z_{0}|\leq C_{2}|v_{0}|$. Then, by the similar estimation in (iii) of lemma 2.7 we have

(2.44)\begin{align} I_{1}& =\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\left\langle (I_{\alpha}\ast|v_{n}+\widetilde{w}_{\Omega}(v_{n})|^{2^{{\ast}}_{\alpha}})|v_{n}+\widetilde{w}_{\Omega}(v_{n})|^{2^{{\ast}}_{\alpha}-2}(v_{n}+\widetilde{w}_{\Omega}(v_{n}))\right.\nonumber\\ & \quad\left.-(I_{\alpha}\ast|v_{0}+w_{0}+z_{0}|^{2^{{\ast}}_{\alpha}})|v_{0}+w_{0}+z_{0}|^{2^{{\ast}}_{\alpha}-2}(v_{0}+w_{0}+z_{0}),v_{n}-v_{0} \right\rangle {\rm d}x\nonumber\\ & =\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\left\langle (I_{\alpha}\ast|v_{n}+\widetilde{w}_{\Omega}(v_{n})|^{2^{{\ast}}_{\alpha}})|v_{n}+\widetilde{w}_{\Omega}(v_{n})|^{2^{{\ast}}_{\alpha}-2}(v_{n}+z_{n})\right.\nonumber\\ & \quad\left.-(I_{\alpha}\ast|v_{0}+w_{0}+z_{0}|^{2^{{\ast}}_{\alpha}})|v_{0}+w_{0}+z_{0}|^{2^{{\ast}}_{\alpha}-2}(v_{0}+z_{0}),v_{n}-v_{0} \right\rangle {\rm d}x\nonumber\\ & \leq C\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\left\langle (I_{\alpha}\ast|v_{n}|^{2^{{\ast}}_{\alpha}})|v_{n}|^{2^{{\ast}}_{\alpha}-2}v_{n}-(I_{\alpha}\ast|v_{0}|^{2^{{\ast}}_{\alpha}})|v_{0}|^{2^{{\ast}}_{\alpha}-2}v_{0},v_{n}-v_{0} \right\rangle {\rm d}x\nonumber\\ & \leq C\left(\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}(I_{\alpha}\ast|v_{n}-v_{0}|^{2^{{\ast}}_{\alpha}})|v_{n}-v_{0}|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}} =C\left(\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\,{\rm d}\overline{\rho}\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}. \end{align}

where we use the fact that $Z$ is a finite dimension space and $\int _{\Omega }\langle v,w\rangle \,{\rm d}x=0$ see (4.2).

Next, we are going to show that $I_{2}=o(1)$. Fix $l\geq 1$. In view of lemmas 2.11, 2.12 and lemma 1.1 in [23], there exists $\xi _{n}\in W^{1,\alpha,2^{\ast }_{\alpha }}(B_{l})$ such that $w_{n}=\nabla \xi _{n}$ and we may assume without loss of generality that $\int _{B_{l}}\xi _{n}\,{\rm d}x=0$. Then by the Poincaré inequality in lemma 2.12

\[ ||\xi_{n}||_{W^{1,\alpha,2^{{\ast}}_{\alpha}}(B_{l})}\leq C|w_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}(B_{l},\mathbb{R}^{3})}\leq C|w_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3})}, \]

and passing to a subsequence, $\xi _{n}\rightharpoonup \xi$ for some $\xi \in W^{1,\alpha,2^{\ast }_{\alpha }}(B_{l})$. So by the natural compactly embedding, $\xi _{n}\longrightarrow \xi$ in $Q^{\alpha,2^{\ast }_{\alpha }}(B_{l})$. Now take any $\varphi \in C^{\infty }_{0}(B_{l})$. Since $\nabla (|\varphi |^{2\cdot 2^{\ast }_{\alpha }}(\xi _{n}-\xi ))\in \widetilde {\mathcal {W}}$, in view of (2.38) we get

\[ \int_{\Omega}\langle f(v_{n}+\widetilde{w}_{\Omega}(v_{n})),\nabla(|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}(\xi_{n}-\xi))\rangle\,{\rm d}x=0. \]

That is

\begin{align*} & \int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\langle f(v_{n}+\widetilde{w}_{\Omega}(v_{n})),w_{n}-\nabla\xi\rangle\,{\rm d}x\nonumber\\ & \quad=\int_{\Omega}\langle f(v_{n}+\widetilde{w}_{\Omega}(v_{n})),\nabla(|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}) (\xi-\xi_{n})\rangle\,{\rm d}x \end{align*}

where the right-hand side tends to 0 as $n\longrightarrow \infty$. Since $w_{n}\rightharpoonup \nabla \xi$ in $Q^{\alpha,2^{\ast }_{\alpha }}(B_{l})$,

\[ \int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\langle f(v_{0}+\nabla\xi+z_{0}),w_{n}-\nabla\xi\rangle\,{\rm d}x=o(1). \]

Hence, recalling that $\widetilde {w}_{\Omega }(v_{n})=w_{n}+z_{n}$ and $z_{n}\longrightarrow z_{0}$, we obtain

(2.45)\begin{equation} I_{2}=\int_{\Omega}|\varphi|^{2\cdot2^{{\ast}}_{\alpha}}\langle f(v_{n}+\widetilde{w}_{\Omega}(v_{n}))-f(v_{0}+\nabla\xi+z_{0}),\widetilde{w}_{\Omega}(v_{n})-\nabla\xi-z_{0}\rangle\,{\rm d}x=o(1). \end{equation}

Since $\varphi \in C^{\infty }_{0}(B_{l})$ is arbitrary, it follows from (2.43) and (2.45) that

(2.46)\begin{equation} 4m_{k}|\Omega_{n,k}\cap E|\leq(\overline{\rho}(E))^{1/2^{{\ast}}_{\alpha}}+o(1) \end{equation}

for any Borel set $E\subset B_{l}$. On the other hand, we can find an open set $E_{k}\supset I$ such that $|E_{k}|<\frac {1}{2^{k+1}}$. Then, taking $E=B_{l}\setminus E_{k}$ in (2.46), we have $4m_{k}|\Omega _{n,k}\cap (B_{l}\setminus E_{k})|=o(1)$ as $n\longrightarrow \infty$ because $\mathrm {supp}(\overline {\rho })\subset I$; hence we can find a sufficiently large $n_{k}$ such that $|\Omega _{n_{k},k}\cap B_{l}|<\frac {1}{2^{k}}$ and we obtain

\[ |\mathop{\cap}\limits_{j=1}^{\infty}\mathop{\cup}\limits_{k=j}^{\infty}\Omega_{n_{k},k}\cap B_{l}|\leq \mathop{\mathrm{lim}}\limits_{j\longrightarrow\infty}\mathop{\Sigma}\limits_{k=j}^{\infty}|\Omega_{n_{k},k}\cap B_{l}|\leq\mathop{\mathrm{lim}}\limits_{j\longrightarrow\infty}\frac{1}{2^{j-1}}=0. \]

According to the fact that $\widetilde {w}_{\Omega }(v_{n})\rightharpoonup \widetilde {w}_{0}$, one can employ the diagonal procedure and hence find a subsequence of $\widetilde {w}_{\Omega }(v_{n})$ which converges to $\widetilde {w}_{0}$ a.e. in $\Omega =\cup _{l=1}^{\infty }B_{l}$.

Let $p\in [1,2^{\ast }_{\alpha }]$. For $\Omega '\subset \Omega$ such that $|\Omega '|<+\infty$ we have

\begin{align*} & \int_{\Omega'}|v_{n}-v_{0}+\widetilde{w}_{\Omega}(v_{n})-\widetilde{w}_{0}|^{p}\,{\rm d}x\leq |\Omega'|^{1-\frac{p}{2^{{\ast}}_{\alpha}}} \left(\int_{\Omega}|v_{n}-v_{0}+\widetilde{w}_{\Omega}(v_{n})-\widetilde{w}_{0}|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{\frac{p}{2^{{\ast}}_{\alpha}}}\\ & \leq |\Omega'|^{1-\frac{p}{2^{{\ast}}_{\alpha}}}|\mathrm{diam}\Omega|^{\frac{3-\alpha}{2}\cdot\frac{p}{2^{{\ast}}_{\alpha}}}\left(\int_{\Omega}|I_{\alpha/2}\ast|v_{n}-v_{0}+\widetilde{w}_{\Omega}(v_{n})-\widetilde{w}_{0}|^{2^{{\ast}}_{\alpha}}|^{2}\mathrm{d}x\right)^{\frac{1}{2}\cdot\frac{p}{2^{{\ast}}_{\alpha}}}, \end{align*}

where $\mathrm {diam}\Omega =\mathop {max}\limits _{x,y\in \Omega }|x-y|$. Hence by the Vitali convergence theorem, $v_{n}-v_{0}+\widetilde {w}_{\Omega }(v_{n})-\widetilde {w}_{0}\longrightarrow 0$ in $L^{p}_{loc}(\Omega )$ after passing to a subsequence.

Step 3. We show that $\widetilde {w}_{\Omega }(v_{0})=\widetilde {w}_{0}$. Take any $\widetilde {w}\in \widetilde {\mathcal {W}}$ and observe that by the Vitali convergence theorem,

\[ 0=\int_{\Omega}\langle f(v_{n}+\widetilde{w}_{\Omega}(v_{n})),\widetilde{w}\rangle dx\longrightarrow \int_{\Omega}\langle f(v_{0}+\widetilde{w}_{0}),\widetilde{w}\rangle\,{\rm d}x, \]

up to a subsequence. Now (2.38) implies that $\widetilde {w}_{0} =\widetilde {w}_{\Omega }(v_{0})$ which completes the proof.

2.3. Abstract critical point theory

For readers convenient, we end this section with recalling the abstract critical point lemma, see [5, § 4] and [30, § 3] for more details. Let $X$ be a reflexive Banach space with norm $||\cdot ||$ and with a topological direct sum decomposition $X=X^+\oplus \widetilde {X}$, where $X^+$ is a Hilbert space with a scalar product. For $u\in X$ we denote by $u^+\in X^+$ and $\widetilde {u}\in \widetilde {X}$ the corresponding summands so that $u=u^++\widetilde {u}$. We may assume that $\langle u,v\rangle =||u||^{2}$ for any $u\in X^+$ and that $||u||^{2}=||u^+||^{2}+||\widetilde {u}||^{2}$. The topology $\mathcal {T}$ on $X$ is defined as the product of the norm topology in $X^+$ and the weak topology in $\widetilde {X}$. Thus $u_{n}\mathop {\longrightarrow }\limits ^{\mathcal {T}}u$ is equivalent to $u^+_{n}\longrightarrow u^+$ and $\widetilde {u}\rightharpoonup \widetilde {u}$.

Let $J\in \mathcal {C}^{1}(X,\mathbb {R})$ be a functional on $X$ of the form

\[ J(u)=\frac{1}{2}||u^+||-I(u)\ \mathrm{for}\ u=u^+{+}\widetilde{u}\in X^+{\oplus}\widetilde{X} \]

such that the following assumptions hold

1. (A1) $I\in \mathcal {C}^{1}(X,\mathbb {R})$ and $I(u)\geq I(0)=0$ for any $u\in X$.

2. (A2) $I$ is $\mathcal {T}$-sequentially lower semi-continuous: $u_{n}\mathop {\longrightarrow }\limits ^{\mathcal {T}}u\Longrightarrow \mathrm {lim}~\mathrm {inf}~I(u_{n})\geq I(u)$.

3. (A3) If $u_{n}\mathop {\longrightarrow }\limits ^{\mathcal {T}} u$ and $I(u_{n})\longrightarrow I(u)$ then $u_{n}\longrightarrow u$.

4. (A4) There exists $r>0$ such that $a:=\mathop {\mathrm {inf}}\limits _{u\in X^+:||u||=r}J(u)>0$.

5. (B1) $||u^+||+I(u)\longrightarrow \infty$ as $||u||\longrightarrow \infty$.

6. (B2) $I(t_{n}u_{n})/t_{n}^{2}\longrightarrow \infty$ if $t_{n}\longrightarrow \infty$ and $u^+_{n}\longrightarrow u^+$ for some $u^+\neq 0$ as $n\longrightarrow \infty$.

7. (B3) $\frac {t^{2}-1}{2}I'(u)(u)+tI'(u)(v)+I(u)-I(tu+v)<0$ for every $u\in \mathcal {N},t>0,v\in X$ such that $u\neq tu+v$.

We defined the following Nehari–Pankov

\[ \mathcal{N}:=\{u\in X\setminus\widetilde{X}:J'(u)|_{\mathbb{R}u\oplus \widetilde{X}}=0\}. \]

Correspondingly, we defined the (PS)$^{\mathcal {T}}_{c}$ condition for $J$.

Definition 2.20 We say that $J$ satisfies the $(PS)_{c}^{\mathcal {T}}$ condition in $\mathcal {N}$ if every $(PS)_{c}$ sequence $(u_{n})\in \mathcal {N}$ has a subsequence which convergence in the $\mathcal {T}$ topology:

\[ u_{n}\in\mathcal{N},\quad J(u_{n})\longrightarrow 0,\quad J'(u_{n})\longrightarrow c\quad\Longrightarrow\quad u_{n}\mathop{\longrightarrow}\limits^{\mathcal{T}}u\in X~~along~a~subsequence. \]

We also recall the compactly perturbed problem with respect to another decomposition of $X$. Namely, suppose that

(2.47)\begin{equation} \widetilde{X}=X^{0}\oplus X^{1}, \end{equation}

where $X^{0},X^{1}$ are closed in $\widetilde {X}$, and $X^{0}$ is a Hilbert space. For $u\in \widetilde {X}$ we denote $u^{0}\in X^{0}$ and $u^{1}\in X^{1}$ the corresponding summands so that $u=u^{0}+u^{1}$. We use the same notation for the scalar product in $X^+\oplus X^{0}$ and $\langle u,u\rangle =||u||^{2}=||u^+||^{2}+||u^{0}||^{2}$ for any $u=u^++u^{0}\in X^+\oplus X^{0}$, hence $X^+$ and $X^{0}$ are orthogonal. We consider another functional $J_{cp}\in \mathcal {C}^{1}(X,\mathbb {R})$ of the form

\[ J_{cp}=\frac{1}{2}||u^+{+}u^{0}||^{2}-I_{cp}(u)\ \mathrm{for}\ u=u^+{+}u^{0}+u^{1}\in X^+{\oplus} X^{0}\oplus X^{1}. \]

We define the corresponding Nehari–Pankov manifold for $J_{cp}$

\[ \mathcal{N}_{cp}:=\{u\in X\setminus X^{1}:J'_{cp}(u)|_{\mathbb{R}u\oplus X^{1}}=0\}, \]

and assume that $J_{cp}$ satisfies all corresponding assumption (A1)–(A4), (B1)–(B3), where we replace $X^+\oplus X^{0}$, $X^{1}$ and $I_{cp}$ instead of $X^+, X$ and $I$ respectively. Moreover, we enlist new additional conditions:

1. (C1) $J_{cp}(u_{n})-J_{u_{n}}\longrightarrow 0$ if $(u_{n})\subset \mathcal {N}_{cp}$ is bounded and $(u^+_{n}+u^{0}_{n})\rightharpoonup 0$. Moreover there is $M>0$ such that $J_{cp}(u)-J(u)\leq M||u^++u^{0}||^{2}$ for $u\in \mathcal {N}_{cp}$.

2. (C2) $I(t_{n}u_{n})\setminus t^{2}_{n}\longrightarrow \infty$ and $t_{n}\longrightarrow \infty$ and $(I(tu^+_{n}))_{n}$ is bounded away from 0 for any $t>1$.

3. (C3) $J'$ is weak-to-weak$^{\ast }$ continuous on $\mathcal {N}$, i.e. if $(u_{n})_{n}\subset \mathcal {N}$, $u_{n}\rightharpoonup u$, then $J'(u_{n})\mathop {\rightharpoonup }\limits ^{\ast } J'(u)$ in $X^{\ast }$. Moreover $J$ is weakly sequentially lower semi-continuous on $\mathcal {N}$, i.e. if $(u_{n})_{n}\subset \mathcal {N}$, $u_{n}\rightharpoonup u$ and $u\in \mathcal {N}$, then $\mathop {\mathrm {lim}~\mathrm {inf}}\limits _{n\longrightarrow \infty }J(u_{n})\geq J(u)$.

There we present the abstract critical point theorem:

Lemma 2.21 [30, theorem 3.2]: Let $J\in \mathcal {C}^{1}(X,\mathbb {R})$ be coercive on $\mathcal {N}$ and let $J_{cp}\in \mathcal {C}^{1}(X,\mathbb {R})$ be coercive on $\mathcal {N}_{cp}$. Suppose that $J$ and $J_{cp}$ satisfy (A1)-(A4),(B1)-(B3) and set $c=\mathop {\mathrm {inf}}\limits _{\mathcal {N}}J$ and $d=\mathop {\mathrm {inf}}\limits _{\mathcal {N}_{cp}}J_{cp}$. Then the following statements hold$:$

1. (a) If (C1)–(C2) hold and $\beta < d$, then any $(PS)_{\beta }$-sequence in $\mathcal {N}$ contains a weakly convergent subsequence with a nontrivial limit point.

2. (b) If (C1)–(C3) hold and $c< d$, then $c$ is achieved by a critical point (ground state) of $J$.

3. (c) Suppose that $J$ is even and satisfies the $(PS)_{\beta }^{\mathcal {T}}$-condition in $\mathcal {N}$ for any $\beta <\beta _{0}$ for some fixed $\beta _{0}\in (c,\infty ]$. Let

   \[ m(\mathcal{N},\beta_{0})=sup\{\gamma(J^{{-}1}((0,\beta)\cap\mathcal{N}):\beta<\beta_{0}\}\in \mathbb{N}_{0}, \]
   where $\gamma$ stands for the Krasnoselskii genus for closed and symmetric subsets of $X$. Then $J$ has at least $m(\mathcal {N},\beta _{0})$ pairs of critical points $u$ and $-u$ such that $u\neq 0$ and $c\leq J(u)<\beta _{0}$.

3. Sharp constant $S_{\mathrm {curl},HL}(\mathbb {R}^{3})$

3.1. Proof of theorem 1.2

In this subsection, we consider functional (1.17), which is associated to equation (1.16), and we work on the following Nehari–Pankov manifold

\[ \mathcal{N}:=\left\{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\mathbb{R}^{3})\setminus \mathcal{W}_{\mathbb{R}^{3}}: J'(u)u=0\ \mathrm{and}\ J'(u)|_{\mathcal{W}_{\mathbb{R}^{3}}}=0\right\}. \]

Lemma 3.1 There exists a continuous mapping $m:\mathcal {V}_{\mathbb {R}^{3}}\setminus \{0\}\longrightarrow \mathcal {N}$.

Proof. By lemma 2.15, $W_{0}^{\alpha,2^{\ast }_{\alpha }}(\mathrm {curl};\mathbb {R}^{3})=\mathcal {V}_{\mathbb {R}^{3}}\oplus \mathcal {W}_{\mathbb {R}^{3}}$. It follows from (2.37) and (2.38) that if $v\in \mathcal {V}_{\mathbb {R}^{3}}$ and $\widetilde {w}_{\mathbb {R}^{3}}(v)\in \widetilde {\mathcal {W}}=\mathcal {W}_{\mathbb {R}^{3}}$, then we have ${J'(v+\widetilde {w}_{\mathbb {R}^{3}}(v))|_{\mathcal {W}_{\mathbb {R}^{3}}}\,{=}\,0}$. And as

(3.1)\begin{equation} J(t(v+\widetilde{w}(v)))=\frac{t^{2}}{2}\int_{\mathbb{R}^{3}}|\nabla v|^{2}\,{\rm d}x-\frac{t^{2\cdot2^{{\ast}}_{\alpha} }}{2\cdot 2^{{\ast}}_{\alpha}}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v+\widetilde{w}_{\mathbb{R}^{3}}(v)|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x, \end{equation}

there is a unique $t(v)>0$ such that

(3.2)\begin{equation} t(v)(v+\widetilde{w}_{\mathbb{R}^{3}}(v))\in\mathcal{N}\ \mathrm{for}\ v\in\mathcal{V}_{\mathbb{R}^{3}}\setminus\{0\}. \end{equation}

Setting $m(v):=t(v)(v+\widetilde {w}_{\mathbb {R}^{3}}(v))$, we then note that

(3.3)\begin{equation} J(m(v))\geq J(t(v+\widetilde{w}_{\mathbb{R}^{3}}))\ \mathrm{for}\ \mathrm{all}\ t>0\ \mathrm{and}\ \widetilde{w}_{\mathbb{R}^{3}}\in\mathcal{W}_{\mathbb{R}^{3}}. \end{equation}

Since $J(m(v))\geq J(v)$ and there exist $a,r>0$ such that $J(v)\geq a$ if $||v||=r$, this implies that $\mathcal {N}$ is bounded away from $\mathcal {W}_{\mathbb {R}^{3}}$ and hence closed. Therefore, by the similar analysis in [33, lemma 4.4 ], the mapping $m$ is continuous.

Lemma 3.2 Set $\mathcal {S}:=\{v\in \mathcal {V}_{\mathbb {R}^{3}}:||v||=1\}$, there exist a $(PS)_{c}$ sequence $(v_{n})$ for $J\circ m$, and a $(PS)_{c}$ sequence $(m(v_{n}))$ for $J$ on $\mathcal {N}$.

Proof. By the continuity of mapping $m$, we easily observe that $m|_{\mathcal {S}}:\mathcal {S}\longrightarrow \mathcal {N}$ is a homeomorphism with the inverse $u=v+m(v)\mapsto \frac {v}{||v||}$. Recall the argument in [28, proposition 4.4(b)], we know that $J\circ m|_{\mathcal {S}}:\mathcal {S}\longrightarrow \mathcal {R}$ is of class $\mathcal {C}^{1}$ and is bounded from below by the constant $a>0$. By the Ekeland variational principle, there is a $(PS)_{c}$ sequence $(v_{n})\subset \mathcal {S}$ such that

(3.4)\begin{equation} (J\circ m)(v_{n})\longrightarrow \mathop{\mathrm{inf}}\limits_{\mathcal{S}}J\circ m=\mathop{\mathrm{inf}}\limits_{\mathcal{N}}J\geq a>0. \end{equation}

Again, by the argument in [28, proposition 4.4(b)], we have $(m(v_{n}))$ is a $(PS)_{c}$ sequence for $J$ on $\mathcal {N}$.

Proof. Complete of the proof of theorem 1.2

Firstly, we prove part (a). Taking a minimizing sequence $(u_{n})=(m(v_{n}))\subset \mathcal {N}$ and set $u_{n}=t(v_{n})(v_{n}+\widetilde {w}_{\mathbb {R}^{3}}(v_{n}))=v'_{n}+\widetilde {w}_{\mathbb {R}^{3}}(v'_{n})\in \mathcal {V}_{\mathbb {R}^{3}}\oplus \mathcal {W}_{\mathbb {R}^{3}}$. Then we have

\[ J(u_{n})=J(u_{n})-\frac{1}{2\cdot 2^{{\ast}}_{\alpha}}J'(u_{n})u_{n}=\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}|\nabla\times u_{n}|^{2}_{2}=\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}|\nabla v'_{n}|^{2}_{2}. \]

Since the norm $|\nabla \cdot |_{2}$ is an equivalent norm in $\mathcal {V}_{\mathbb {R}^{3}}$, it follows that $J(u_{n})$ is coercive on $\mathcal {N}$, hence $(v'_{n})$ is bounded. On the other hand, we also have

\[ J(u_{n})=J(u_{n})-\frac{1}{2}J'(u_{n})u_{n}=\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u_{n}|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x. \]

By (3.4), $J(u_{n})$ is bounded away from $0$, so is $|u_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}\nrightarrow 0$, and hence by (2.40), we also have $|v'_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}\nrightarrow 0$.

Denote $T_{s,y}(v'):=s^{1/2}v'(s\cdot +y)$, where $s>0$, $y\in \mathbb {R}^{3}$. Then, passing to a subsequence and using the argument in [42, theorem 1], we have $\overline {v}_{n}=T_{s_{n},y_{n}}(v'_{n})\rightharpoonup v_{0}$ for some $v_{0}\neq 0$, where $(s_{n})\subset \mathbb {R}^+$ and $(y_{n})\subset \mathbb {R}^{3}$. Taking subsequence again, we also have $\overline {v}_{n}\longrightarrow v_{0}$ a.e. in $\mathbb {R}^{3}$ and in view of the concentration–compactness lemma 2.18, we deduce $\widetilde {w}_{\mathbb {R}^{3}}(\overline {v}_{n})\rightharpoonup \widetilde {w}_{\mathbb {R}^{3}}(v_{0})$ and $\widetilde {w}_{\mathbb {R}^{3}}(\overline {v}_{n})\longrightarrow \widetilde {w}_{\mathbb {R}^{3}}(v_{0})$ a.e. in $\mathbb {R}^{3}$. Setting $u:=v_{0}+\widetilde {w}_{\mathbb {R}^{3}}(v_{0})$ and assume without loss of generality that $s_{n}=1$ and $y_{n}=0$, then by lemma 4.6 in [33], we have $u_{n}\rightharpoonup u$ and $u_{n}\longrightarrow u$ a.e. in $\mathbb {R}^{3}$. Moreover, by lemma 2.7 we have

\[ (I_{\alpha}\ast|u_{n}|^{2^{{\ast}}_{\alpha}})|u_{n}|^{2^{{\ast}}_{\alpha}-2}u_{n}\rightharpoonup(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}})|u|^{2^{{\ast}}_{\alpha}-2}u\ \mathrm{in}\ (Q^{\alpha,2^{{\ast}}_{\alpha}}(\mathbb{R}^{3},\mathbb{R}^{3}))', \]

Therefore, for any $z\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$, using weak and a.e. convergence, we have

\begin{align*} \langle J'(u_{n}),z\rangle& =\int_{\mathbb{R}^{3}}\langle\nabla\times u_{n},z\rangle\,{\rm d}x-\int_{\mathbb{R}^{3}}\left\langle\left(I_{\alpha}\ast|u_{n}|^{2^{{\ast}}_{\alpha}}\right)|u_{n}(x)|^{2^{{\ast}}_{\alpha}-2}u_{n}(x),z\right\rangle\,{\rm d}x \nonumber\\ & \quad\longrightarrow \langle J'(u),z\rangle. \end{align*}

This implies that $u$ is a solution to (1.16). Using Fatou's lemma, we deduce that

\begin{align*} \mathop{\mathrm{inf}}\limits_{\mathcal{N}}J& =J(u_{n})+o(1)=J(u_{n})-\frac{1}{2}J'(u_{n})u_{n}+o(1)\\ & =\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u_{n}|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x+o(1)\geq \frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x +o(1)\\ & =J(u)-\frac{1}{2}J'(u)u+o(1)=J(u)+o(1). \end{align*}

Hence $J(u)\leq \mathop {\mathrm {inf}}\limits _{\mathcal {N}} J\leq J(u)$ and as a solution, $u\in \mathcal {N}$.

Next, we show $\mathop {\mathrm {inf}}\limits _{\mathcal {N}} J=\frac {2^{\ast }_{\alpha }-1}{2\cdot 2^{\ast }_{\alpha }} S_{\mathrm {curl},HL}^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}$, where $S_{\mathrm {curl},HL}$ is the sharp constant in (1.15), which can be rewritten as follow

(3.5)\begin{equation} S_{\mathrm{curl},HL}=\mathop{\mathrm{inf}}\limits_{\mathop{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\mathbb{R}^{3})}\limits_{\nabla\times u\neq0}}\frac{\int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x}{\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u+\widetilde{w}_{\mathbb{R}^{3}}(u)|^{2^{{\ast}}_{\alpha}}|^{2}dx\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}}. \end{equation}

In fact, by (2.37), it is clear that a minimize $\widetilde {w}_{\mathbb {R}^{3}}(u)$ exists uniquely for any $u\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$, not only $u\in \mathcal {V}_{\mathbb {R}^{3}}$. So by lemma 2.15, $u+\widetilde {w}_{\mathbb {R}^{3}}(u)=v+\widetilde {w}_{\mathbb {R}^{3}}(v)\in \mathcal {V}_{\mathbb {R}^{3}}\oplus \mathcal {W}_{\mathbb {R}^{3}}$ for some $v\in \mathcal {V}_{\mathbb {R}^{3}}$ and therefore

(3.6)\begin{align} \mathop{\mathrm{inf}}\limits_{w\in\mathcal{W}_{\mathbb{R}^{3}}}\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u+w|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x & =\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u+\widetilde{w}_{\mathbb{R}^{3}}(u)|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\nonumber\\ & \quad =\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v+\widetilde{w}_{\mathbb{R}^{3}}(v)|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x. \end{align}

On the other hand, since $u+\widetilde {w}_{\mathbb {R}^{3}}(u)\in \mathcal {N}$, $J'(u)u=0$, i.e.

\[ \int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x=\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u+\widetilde{w}_{\mathbb{R}^{3}}(u)|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x. \]

Then we can easily calculate that

\[ \mathop{\mathrm{inf}}\limits_{\mathcal{N}} J=\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\mathbb{R}^{3}}|\nabla\times u|^{2}dx =\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}} S_{\mathrm{curl},HL}^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}. \]

As we can see, if $u$ satisfies equality (1.15), then $t(u)(u+\widetilde {w}_{\mathbb {R}^{3}}(u))\in \mathcal {N}$ and is a minimizer for $J|_{\mathcal {N}}$ and the corresponding point $v$ in $\mathcal {S}$ is a minimizer for $J\circ m|_{\mathcal {S}}$, see (3.4). Hence $v$ is a critical point of $J\circ m|_{\mathcal {S}}$ and $m(v)=u$ is a critical point of $J$. This completes the proof of (a).

(b) To compare the constants $S_{\mathrm {curl},HL}$ and $S_{HL}$, see (3.5) and (1.18), we firstly claim that $S_{\mathrm {curl},HL}\geq S_{HL}$. In fact, by (3.6) and $\mathrm {div}(v)=0$, we have

(3.7)\begin{equation} S_{\mathrm{curl},HL}=\mathop{\mathrm{inf}}\limits_{v\in\mathcal{V}_{\mathbb{R}^{3}}\setminus\{0\}}\frac{\int_{\mathbb{R}^{3}}|\nabla v|^{2}\,{\rm d}x}{\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v+\widetilde{w}_{\mathbb{R}^{3}}(v)|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}}. \end{equation}

Then given $\varepsilon >0$, we can find $v\neq 0$ such that

\[ \int_{\mathbb{R}^{3}}|\nabla v|^{2}\,{\rm d}x\leq (S_{\mathrm{curl},HL}+\varepsilon)\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v+\widetilde{w}_{\mathbb{R}^{3}}(v)|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}. \]

Since $\widetilde {w}_{\mathbb {R}^{3}}(v)$ is a minimizer, we deduce that

\[ \int_{\mathbb{R}^{3}}|\nabla v|^{2}\,{\rm d}x\leq (S_{\mathrm{curl},HL}+\varepsilon)\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}. \]

On the other hand, let $v=(v_{1},v_{2},v_{3})$, then $|v|=(v_{1}^{2}+v_{2}^{2}+v_{3}^{2})^{\frac {1}{2}}$. Since $\frac {2^{\ast }_{\alpha }}{2}>1$, then by the second inequality in [34, proposition 2.1], we have

\begin{align*} \left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}} & =\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast(v_{1}^{2}+v_{2}^{2}+v_{3}^{2})^{\frac{2^{{\ast}}_{\alpha}}{2}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\\ & \quad \leq \mathop{\sum}\limits_{i}^{3}\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v_{i}|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}. \end{align*}

Moreover, by the definition of $S_{HL}$, see (1.18), we have

\begin{align*} \int_{\mathbb{R}^{3}}|\nabla v|^{2}\,{\rm d}x& \leq\frac{(S_{\mathrm{curl},HL}+\varepsilon)}{S_{HL}}\cdot S_{HL}\cdot\mathop{\sum}\limits_{i}^{3}\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v_{i}|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\\ & \quad\leq\frac{(S_{\mathrm{curl},HL}+\varepsilon)}{S_{HL}}\int_{\mathbb{R}^{3}}|\nabla v|^{2}\,{\rm d}x. \end{align*}

Hence we get our claim by $S_{\mathrm {curl},HL}+\varepsilon \geq S_{HL}$.

Secondly, we exclude the case $S_{\mathrm {curl},HL}=S_{HL}$. Otherwise, all inequalities above become equalities with $\varepsilon =0$. Particularly, $S_{HL}(\int _{\mathbb {R}^{3}}|I_{\alpha /2}\ast |v_{i}|^{2^{\ast }_{\alpha }}|^{2}\,{\rm d}x)^{\frac {1}{2^{\ast }_{\alpha }}} =\int _{\mathbb {R}^{3}}|\nabla v_{i}|^{2}\,{\rm d}x$. This implies that all $v_{i}$ are instantons and $v_{i}=C(\frac {b}{b^{2}+|x-a|^{2}})^{\frac {N-2}{2}}$, up to multiplicative constants, see [19, lemma 1.2] for the optimal function of $S_{HL}$. A simple calculation shows that $\mathrm {div}(v)\neq 0$. However, this is impossible because $v\in \mathcal {V}\setminus \{0\}$. Hence, $S_{\mathrm {curl},HL}\neq S_{HL}$.

3.2 Proof of theorem 1.3

To compare the sharp constants $S_{\mathrm {curl},HL}(\mathbb {R}^{3})$ and $\bar {S}_{\mathrm {curl},HL}(\Omega )$, we have introduced another constant $S_{\mathrm {curl},HL}(\Omega )$. Recall from § 2.1 that we have the following Helmholtz decomposition in entire space $\mathbb {R}^{3}$ and in the bounded domain $\Omega$:

\[ W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\mathbb{R}^{3})=\mathcal{V}_{\mathbb{R}^{3}}\oplus\mathcal{W}_{\mathbb{R}^{3}}\quad \mathrm{and}\quad W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)=\mathcal{V}_{\Omega}\oplus\mathcal{W}_{\Omega}. \]

Then, as (3.5), we note that $S_{\mathrm {curl},HL}(\Omega )$ [see (1.19)] can be characterized as

(3.8)\begin{align} S_{\mathrm{curl},HL}(\Omega)& =\mathop{\mathrm{inf}}\limits_{\mathop{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)}\limits_{\nabla\times u\neq0}}\mathop{\mathrm{sup}}\limits_{w\in\mathcal{W}_{\mathbb{R}^{3}}}\frac{\int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x}{\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u+w|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}}\nonumber\\ & =\mathop{\mathrm{inf}}\limits_{\mathop{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)}\limits_{\nabla\times u\neq0}}\frac{\int_{\mathbb{R}^{3}}|\nabla\times u|^{2}\,{\rm d}x}{\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u+\widetilde{w}_{\mathbb{R}^{3}}(u)|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}}, \end{align}

where $u\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$ is extended by $0$ outside $\Omega$. For constant $\bar {S}_{\mathrm {curl},HL}(\Omega )$ in domains $\Omega \neq \mathbb {R}^{3}$, it also can be characterized as

(3.9)\begin{align} \bar{S}_{\mathrm{curl},HL}(\Omega)& =\mathop{\mathrm{inf}}\limits_{\mathop{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)}\limits_{\nabla\times u\neq0}}\mathop{\mathrm{sup}}\limits_{w\in\mathcal{W}_{\Omega}}\frac{\int_{\Omega}|\nabla\times u|^{2}\,{\rm d}x}{\left(\int_{\Omega}|I_{\alpha/2}\ast|u+w|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}}\nonumber\\ & =\mathop{\mathrm{inf}}\limits_{\mathop{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)}\limits_{\nabla\times u\neq0}}\frac{\int_{\Omega}|\nabla\times u|^{2}\,{\rm d}x}{\left(\int_{\Omega}|I_{\alpha/2}\ast|u+\widetilde{w}_{\Omega}(u)|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}}. \end{align}

To compare these sharp constants, we introduce the following set

(3.10)\begin{equation} \mathcal{N}_{\Omega}:=\{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)\setminus\mathcal{W}_{\Omega} ;J'(u)u=0\ \mathrm{and}\ J'(u)|_{\mathcal{W}_{\Omega}}=0\}. \end{equation}

According to the argument in [33, lemma 4.2], we have $tu+\widetilde {w}_{\mathbb {R}^{3}}(tu)=t(u+\widetilde {w}_{\mathbb {R}^{3}}(u))$ , then we may assume without loss of generality that $u+\widetilde {w}_{\mathbb {R}^{3}}(u)\in \mathcal {N}$ in (3.8). By the maximality and uniqueness of $\widetilde {w}_{\Omega }(u)$, we easily deduce that the mapping $u\mapsto \widetilde {w}_{\Omega }(u)$ is also continuous. Therefore, we may assume that $u+\widetilde {w}_{\Omega }(u)\in \mathcal {N}_{\Omega }$ in (3.9). Then easily calculate that

(3.11)\begin{align} \mathop{\mathrm{inf}}\limits_{\mathcal{N}} J& =\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}} S_{\mathrm{curl},HL}^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}},\quad \mathop{\mathrm{inf}}\limits_{\mathcal{N}} J|_{W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)}\nonumber\\ & \quad =\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}} S^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}_{\mathrm{curl},HL}(\Omega),\quad \mathop{\mathrm{inf}}\limits_{\mathcal{N}_{\Omega}} J=\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}} \bar{S}^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}_{\mathrm{curl},HL}(\Omega). \end{align}

Lemma 3.3 $S_{\mathrm {curl},HL}(\Omega )\geq S_{\mathrm {curl},HL}$, $S_{\mathrm {curl},HL}(\Omega )\geq \bar {S}_{\mathrm {curl},HL}(\Omega )$.

Proof. In view of lemma 2.16, $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )\subset W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\mathbb {R}^{3})$, we can easily observe from (3.8) and (3.5) that $S_{\mathrm {curl},HL}(\Omega )\geq S_{\mathrm {curl},HL}$. Similarly, since $\mathcal {W}_{\Omega }\subset \mathcal {W}_{\mathbb {R}^{3}}$, we can deduce that $S_{\mathrm {curl},HL}(\Omega )\geq \bar {S}_{\mathrm {curl},HL}(\Omega )$ from (3.8) and (3.9).

To complete theorem 1.3, we shall need the following inequality, which corresponds to the condition (B3), and the proof follows a similar argument in [30, lemma 4.1].

Lemma 3.4 If $u\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )\setminus \{0\},u\in \mathcal {W}_{\Omega }$ and $t\geq 0$, then

\[ J(u)\geq J(tu+w)-J'(u)\left[\frac{t^{2}-1}{2}u+tw\right]. \]

Moreover, strict inequality holds provided $t=1$ and $w=0$. $(\Omega =\mathbb {R}^{3}\ admitted.)$

Proof. By an explicit computation and using $\nabla \times w=0$, we show that

\[ J(u)-J(tu+w)+J'(u)\left[\frac{t^{2}-1}{2}u+tw\right]=\int_{\Omega}\varphi(t,x)\,{\rm d}x, \]

where

\begin{align*} \varphi(t,x)& ={-}\left\langle\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)|u(x)|^{2^{{\ast}}_{\alpha}-2}u(x) ,\frac{t^{2}-1}{2}u(x)+tw(x)\right\rangle\\ & \quad-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x \\ & \quad+\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|tu+w|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x. \end{align*}

It is easy to check that $\varphi (0,x)>0$ as $t=0$ and $\varphi (t,x)\longrightarrow \infty$ as $t\longrightarrow \infty$. Therefore, if there exist $t$ such that $\varphi (t,x)\leq 0$, then there exists $t_{0}>0$ such that $\partial _{t}\varphi (t_{0},x)=0$, namely

\begin{align*} \partial_{t}\varphi(t_{0},x)& ={-}\left\langle\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)|u(x)|^{2^{{\ast}}_{\alpha}-2}u(x) ,t_{0}u(x)+w(x)\right\rangle\\ & \quad+\left\langle\left(I_{\alpha}\ast|t_{0}u+w|^{2^{{\ast}}_{\alpha}}\right)|t_{0}u(x)+w(x)|^{2^{{\ast}}_{\alpha}-2}\left(t_{0}u(x)+w(x)\right),u(x)\right\rangle=0, \end{align*}

then either $\langle u,t_{0}u+w\rangle$=0, i.e. $-\langle u,w\rangle =t_{0}\langle u,u\rangle =t_{0}|u|^{2}$, or $|u|=|t_{0}+w|$, i.e., $-t_{0}\langle u,w\rangle =\frac {t_{0}^{2}-1}{2}|u|^{2}+\frac {1}{2}|w|^{2}$. In the first case, we obtain that

\begin{align*} \varphi(t_{0},x)& =\left(\frac{t_{0}^{2}+1}{2}-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\right)\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\\ & \quad +\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\mathbb{R}{3}}|I_{\alpha/2}\ast|t_{0}u+w|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x>0. \end{align*}

And in the second case, we deduce that

\[ \varphi(t_{0},x)=\frac{1}{2}\int_{\Omega}(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}})|u(x)|^{2^{{\ast}}_{\alpha}-2}|w(x)|^{2}\,{\rm d}x\geq0. \]

Hence $\varphi (t,x)\geq 0$ for all $t\geq 0$ and the inequality is strict if $w\neq 0$. If $w=0$, we can see

\[ \varphi(t,x)=\left(\frac{t^{2\cdot2^{{\ast}}_{\alpha}}}{2\cdot2^{{\ast}}_{\alpha}}-\frac{t^{2}}{2}+\frac{1}{2}-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\right)\int_{\Omega}(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}})|u(x)|^{2^{{\ast}}_{\alpha}-2}|w(x)|^{2}dx>0 \]

provided $t\neq 1$.

Lemma 3.5 $S_{\mathrm {curl},HL}(\Omega )\leq S_{\mathrm {curl},HL}$.

Proof. By theorem 1.2(a), $u$ is a minimizer for $J$ on $\mathcal {N}$, then we can find a sequence $(u_{n})\subset \mathcal {C}_{0}^{\infty }(\mathbb {R}^{3},\mathbb {R}^{3})$ such that $u_{n}\longrightarrow u$. By the Helmholtz decomposition, we have $u_{n}=v_{n}+w_{m}, v_{n}\in \mathcal {V}_{\mathbb {R}^{3}},w_{n}\in \mathcal {W}_{\mathbb {R}^{3}}$. Since $u_{n}=v_{n}+w_{n}\longrightarrow u=v_{0}+\widetilde {w}_{\mathbb {R}^{3}}(v_{0})$ and therefore $v_{n}\longrightarrow v_{0}$, $w_{n}\longrightarrow \widetilde {w}_{\mathbb {R}^{3}}(v_{0})$. So $v_{0}\neq 0$ and $v_{n}$ are bounded away from $0$ in $Q^{\alpha,2^{\ast }_{\alpha }}(\mathbb {R}^{3},\mathbb {R}^{3})$ due to $u\in \mathcal {N}$.

Assume without loss of generality that $0\in \Omega$. There exist $\lambda _{n}$ such that $\overline {u}_{n}$ given by $\overline {u}_{n}(x):=\lambda _{n}^{1\setminus 2}u_{n}(\lambda _{n}x)$ are supported in $\Omega$, that is $\overline {u}_{n}(x)\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$. Set $\widetilde {w}_{\mathbb {R}^{3}}(\overline {u}_{n})\in \mathcal {W}_{\mathbb {R}^{3}}$ and choose $t_{n}$ so that $t_{n}(\overline {u}_{n}+\widetilde {w}_{\mathbb {R}^{3}}(\overline {u}_{n}))\in \mathcal {N}$, then

(3.12)\begin{equation} t_{n}^{2}=\frac{\left(\int_{\mathbb{R}^{3}}|\nabla\times \overline{u}_{n}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}-1}}} {\left(\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|\overline{u}_{n}+\widetilde{w}_{\mathbb{R}^{3}}(\overline{u}_{n})|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}-1}}}. \end{equation}

Since the Riesz potential is invariant with respect to translation, we have $||\overline {u}_{n}||=||u_{n}||$ and

\begin{align*} \int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|\overline{u}_{n}+\widetilde{w}_{\mathbb{R}^{3}}(\overline{u}_{n})|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x& =\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|u_{n}+\widetilde{w}_{\mathbb{R}^{3}}(u_{n})|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\\ & =\int_{\mathbb{R}^{3}}|I_{\alpha/2}\ast|v_{n}+\widetilde{w}_{\mathbb{R}^{3}}(v_{n})|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x. \end{align*}

Therefore, as $(u_{n})$ is bounded, we have $(\overline {u}_{n})$ and $(\widetilde {w}_{\mathbb {R}^{3}}(\overline {u}_{n}))$ are bounded away from $0$, so is $||\overline {u}_{n}(x)+\widetilde {w}_{\mathbb {R}^{3}}(\overline {u}_{n})(x)||_{Q^{\alpha,2^{\ast }_{\alpha }}}$. Then we deduce that $(t_{n})$ is bounded, hence so is $(t_{n}^{2})$. Moreover, since $J(\overline {u}_{n})=J(u_{n})\longrightarrow \frac {2^{\ast }_{\alpha }-1}{2\cdot 2^{\ast }_{\alpha }} S_{\mathrm {curl},HL}^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}$ and $||J'(\overline {u}_{n})||=||J'(u_{n})||\longrightarrow 0$, it follows from lemma 3.4 that

\begin{align*} \frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}} S_{\mathrm{curl},HL}^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}} & =\mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}J(\overline{u}_{n}) \geq\mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty} \left(\vphantom{\left. -J'(\overline{u}_{n})\left[\frac{t^{2}_{n}-1}{2}\overline{u}_{n} +t^{2}_{n}\widetilde{w}_{\mathbb{R}^{3}}(\overline{u}_{n}))\right]\right)}J(t_{n}(\overline{u}_{n}+\widetilde{w}_{\mathbb{R}^{3}}(\overline{u}_{n})))\right.\\ & \left.\quad -\,J'(\overline{u}_{n})\left[\frac{t^{2}_{n}-1}{2}\overline{u}_{n} +t^{2}_{n}\widetilde{w}_{\mathbb{R}^{3}}(\overline{u}_{n}))\right]\right)\\ & =\mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}J(t_{n}(\overline{u}_{n}+\widetilde{w}_{\mathbb{R}^{3}}(\overline{u}_{n})))\geq\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}} S^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}_{\mathrm{curl},HL}(\Omega). \end{align*}

The last inequality follows from the fact that $\overline {u}_{n}\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$.

Proof. Complete of the proof of theorem 1.3

Repeating the proof of theorem 1.2 (b) with obvious changes, namely, change the domain $\mathbb {R}^{3}$ into $\Omega$, change $S_{\mathrm {curl},HL}$ into $\bar {S}_{\mathrm {curl},HL}(\Omega )$, we have $\bar {S}_{\mathrm {curl},HL}(\Omega )\geq S_{HL}$. Since the optimal function for $\bar {S}_{\mathrm {curl},HL}(\Omega )$ is not found in our process, we can not exclude the case $\bar {S}_{\mathrm {curl},HL}(\Omega )=S_{HL}$. As a consequently, we complete the proof of theorem 1.3 by lemmas 3.3 and 3.5.

4. Proof of theorem 1.4

According to the spectrum analysis of the curl–curl operator in the introduction, for $\lambda \leq 0$, we find two closed and orthogonal subspaces $\mathcal {V}_{\Omega }^+$ and $\widetilde {\mathcal {V}}_{\Omega }$ of $\mathcal {V}_{\Omega }$ such that the quadratic form $Q:\mathcal {V}_{\Omega }\longrightarrow \mathbb {R}$ given by

(4.1)\begin{equation} Q(v):=\int_{\Omega}(|\nabla\times v|^{2}+\lambda|v|^{2})\,{\rm d}x=\int_{\Omega}(|\nabla v|^{2}+\lambda|v|^{2})\,{\rm d}x \end{equation}

is positive defined on $\mathcal {V}_{\Omega }^+$ and negative semi-definite on $\widetilde {\mathcal {V}}_{\Omega }$ where $\mathrm {dim}\widetilde {\mathcal {V}}_{\Omega }<\infty$. Writing $u=v+w=v^++\widetilde {v}+w\in \mathcal {V}_{\Omega }^+\oplus \widetilde {\mathcal {V}}_{\Omega }\oplus \mathcal {W}_{\Omega }$, the functional $J_{\lambda }$ [see (1.13)] can be expressed as

\begin{align*} J_{\lambda}(u) & =\frac{1}{2}||v^+||^{2}+\frac{1}{2}||\widetilde{v}||^{2}+\frac{\lambda}{2}\int_{\Omega}(|v|^{2}+|w|^{2})\,{\rm d}x-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\\ & =\frac{1}{2}||v^+||^{2}-I_{\lambda}(v+w), \end{align*}

where

\[ I_{\lambda}(v+w)={-}\frac{1}{2}||\widetilde{v}||^{2}-\frac{\lambda}{2}\int_{\Omega}(|v|^{2}+|w|^{2})\,{\rm d}x+\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x. \]

Similarly as in [5], we shall show that $J_{\lambda }$ satisfies the assumptions (A1)–(A4), (B1)–(B3) and (C1)–(C3) from § 2.2.

Lemma 4.1 Conditions (A1)–(A4), (B1)–(B3) and (C2) in lemma 2.21 hold for $J_{\lambda }$.

Proof.

1. (i) By lemma 2.7, we have $I_{\lambda }$ is of class $\mathcal {C}^{1}$. Since $Q(v)$ is negative on $\widetilde {\mathcal {V}}_{\Omega }$, $2^{\ast }_{\alpha }$ is a upper critical index, we have $I_{\lambda }(u)\geq I_{\lambda }(0)=0$ for any $u\in W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$.

2. (ii) Since $I_{\lambda }$ is convex, $I_{\lambda }$ is $\mathcal {T}-$sequentialy lower semicontinuous. Hence, (A2) holds.

3. (iii) We easily check (A3), since $u_{n}\rightharpoonup u_{0}$ in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$, and $I_{\lambda }(u_{n})\longrightarrow I_{\lambda }(u_{0})$ imply $|u_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}\longrightarrow |u_{0}|_{Q^{\alpha,2^{\ast }_{\alpha }}}$, thus $u_{n}\longrightarrow u_{0}$ in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$.

4. (iv) Since $\mathcal {V}_{\Omega }$ is a Hilbert space, the HLS inequality is still valid on there, then for any $u\in \mathcal {V}_{\Omega }^+$, we have

   \begin{align*} J(u)=J(v,0)& =\frac{1}{2}||v||_{\mathcal{V}_{\Omega}}^{2}+\frac{\lambda}{2}|v|^{2}_{2}-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|v|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\\ & \geq\frac{1}{2}||v||_{\mathcal{V}_{\Omega}}^{2}+\frac{\lambda}{2}|v|^{2}_{2}-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\left(\int_{\Omega}|v|^{6}\,{\rm d}x\right)^{\frac{1}{6}\ast2}\\ & \geq\frac{\delta}{2}||v||_{\mathcal{V}_{\Omega}}^{2}-\varepsilon|v|^{2}-c_{\varepsilon}|v|^{3}_{6}\geq\frac{\delta}{4}||v||^{2}_{\mathcal{V}_{\Omega}}-C_{1}||v||_{\mathcal{V}_{\Omega}}^{3} \end{align*}
   for some $\delta, C_{1}>0$.

5. (v) Condition (B1) follows from lemma 5.1 (c) in [5]. Suppose that $(||v^+_{n}||_{\mathcal {V}_{\Omega }})_{n}$ is bounded and $||(v_{n},w_{n})||\longrightarrow \infty$ as $n\longrightarrow \infty$. Since $\mathrm {dim}(\widetilde {\mathcal {V}}_{\Omega })<\infty$ there holds $|v_{n}+w_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}\longrightarrow \infty$. Moreover by the orthogonality $\mathcal {V}_{\Omega }^+\perp \widetilde {\mathcal {V}}_{\Omega }$ in $L^{2}(\Omega,\mathbb {R}^{3})$ and $\mathcal {V}_{\Omega }\perp \mathcal {W}_{\Omega }$ in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$, we have

   (4.2)\begin{equation} ||\widetilde{v}_{n}||_{\mathcal{V}_{\Omega}}^{2}\leq C_{1}|\widetilde{v}_{n}|_{2}^{2}\leq C_{1}|v_{n}|^{2}_{2}\leq C_{1}|v_{n}+ w_{n}|_{2}^{2}\leq C_{2}|v_{n}+ w_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2} \end{equation}
   for some $0< C_{1}< C_{2}$. This implies
   \begin{align*} I(v_{n},w_{n})& ={-}\frac{1}{2}||\widetilde{v}_{n}||-\frac{\lambda}{2}|v_{n}+ w_{n}|^{2}_{2}+\frac{1}{2\cdot2^{{\ast}}_{\alpha}}|v_{n}+ w_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2^{{\ast}}_{\alpha}}\\ & \geq{-}\frac{C_{2}}{2}|v_{n}+ w_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2}+\frac{1}{2\cdot2^{{\ast}}_{\alpha}}|v_{n}+ w_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2^{{\ast}}_{\alpha}}\longrightarrow\infty, \end{align*}
   because $|v_{n}+\nabla w_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}\longrightarrow \infty$.

6. (vi) This part we check condition (B2) and (C2). By (4.2), we have

   \begin{align*} & I(t_{n}(v_{n}+w_{n}))\\ & =\frac{1}{2}||t_{n}\widetilde{v}_{n}||_{\mathcal{V}_{\Omega}}^{2}-\frac{\lambda}{2}|t_{n}(v_{n}+w_{n})|^{2}_{2}+\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|t_{n}(v_{n}+w_{n})|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\\ & \geq{-}\frac{1}{2}t^{2}_{n}||\widetilde{v}_{n}||^{2}_{\mathcal{V}_{\Omega}}-\frac{\lambda}{2}t^{2}_{n}|v_{n}+w_{n}|^{2}_{2}+t_{n}^{2\cdot2^{{\ast}}_{\alpha}}\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|v_{n}+w_{n}|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\\ & \geq{-}\frac{C_{2}}{2}t^{2}_{n}||{v}_{n}+w_{n}||^{2}_{Q_{\alpha,2^{{\ast}}_{\alpha}}}+t_{n}^{2\cdot2^{{\ast}}_{\alpha}}\frac{1}{2\cdot2^{{\ast}}_{\alpha}}||{v}_{n}+w_{n}||^{2\cdot2^{{\ast}}_{\alpha}}_{Q_{\alpha,2^{{\ast}}_{\alpha}}}. \end{align*}
   Then
   \[ I(t_{n}(v_{n}+w_{n}))/ t^{2}_{n}\geq{-}\frac{C_{2}}{2}||\widetilde{v}_{n}||^{2}_{Q_{\alpha,2^{{\ast}}_{\alpha}}}+t_{n}^{2\cdot2^{{\ast}}_{\alpha}-2}\frac{1}{2\cdot2^{{\ast}}_{\alpha}}||\widetilde{v}_{n}||^{2\cdot2^{{\ast}}_{\alpha}}_{Q_{\alpha,2^{{\ast}}_{\alpha}}}. \]
   If $||(v_{n},w_{n})||\longrightarrow \infty$ then $I(t_{n}(v_{n}+w_{n}))/ t^{2}_{n}\longrightarrow \infty$. If $(||(v_{n},w_{n})||)_{n}$ is bounded. Then $(|v_{n}+w_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}})_{n}$ is bounded. If $|v_{n}+w_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}\longrightarrow 0$, then $|v_{n}+w_{n}|_{2}\longrightarrow 0$ and by the orthogonality in $L^{2}(\Omega,\mathbb {R}^{3})$ which contradicts $u_{0}\neq 0$. Therefore $\frac {t_{n}^{2\cdot 2^{\ast }_{\alpha }-2}}{2\cdot 2^{\ast }_{\alpha }}||\widetilde {v}_{n}||^{2\cdot 2^{\ast }_{\alpha }}_{Q_{\alpha,2^{\ast }_{\alpha }}}\longrightarrow \infty$ as $n\longrightarrow \infty$ and again $I(t_{n}(v_{n}+w_{n}))/ t^{2}_{n}\longrightarrow \infty$.

7. (vii) Condition (B3) follows from lemma 3.4 by changing $J(u)$ into $J_{\lambda }(u)$.

To apply the concentration–compactness lemma, we set $\widetilde {\mathcal {W}}:=\widetilde {\mathcal {V}}_{\Omega }\oplus \mathcal {W}_{\Omega }$ with $\widetilde {w}_{\Omega }=\widetilde {v}+w$, where $\widetilde {\mathcal {V}}_{\Omega }=Z$, see §2.2. On the other hand, we shall extend $\mathcal {V}_{\Omega }^+$ into $\mathcal {V}_{\mathbb {R}^{3}}$, which is a closed subspaces of $\mathcal {D}^{1,2}(\mathbb {R}^{3},\mathbb {R}^{3})$. Indeed, let $U$ be a bounded domain in $\mathbb {R}^{3}$, $\bar {\Omega }\subset U$. Since $\mathcal {V}_{\Omega }\subset H^{1}(\Omega,\mathbb {R}^{3})$, then each $v\in \mathcal {V}_{\Omega }$ may be extended to $v'\in H^{1}_{0}(U,\mathbb {R}^{3})$ such that $v'|_{\Omega } =v$. This extension is bounded as a mapping from $\mathcal {V}_{\Omega }$ to $H^{1}_{0}(U,\mathbb {R}^{3})$. Since

\[ \mathcal{V}':=\{v'\in H^{1}_{0}(U,\mathbb{R}^{3}):v'|_{\Omega}\in\mathcal{V}_{\Omega}\} \]

is a closed subspace of $H^{1}_{0}(U,\mathbb {R}^{3})$, and hence of $\mathcal {D}^{1,2}(\mathbb {R}^{3},\mathbb {R}^{3})$, we then can apply lemma 2.18 with $\mathcal {V}_{\Omega }^+$ replacing $\mathcal {V}_{\mathbb {R}^{3}}$. Set the generalized Nehari–Pankov manifold as follow

(4.3)\begin{equation} \mathcal{N}_{\lambda}:=\{u\in W^{\alpha,2^{{\ast}}_{\alpha}}_{0}(\mathrm{curl};\Omega)\setminus(\widetilde{\mathcal{V}}_{\Omega}\oplus\mathcal{W}_{\Omega}) :J'_{\lambda}(u)|_{\mathcal{R}u\oplus\widetilde{\mathcal{V}}_{\Omega}\oplus\widetilde{\mathcal{W}}_{\Omega}}=0\}. \end{equation}

Lemma 4.2 $J'$ is weak-to-weak$^{\ast }$ continuous on $\mathcal {N_{\lambda }}$ and condition (C3) in lemma 2.21 holds.

Proof. Suppose that $u_{n}\rightharpoonup u_{0}$ in $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$. Set $u_{n}=m_{\lambda }(v^+_{n})=v^++\widetilde {w}_{\Omega }(v^+_{n})$. Since $\mathcal {V}_{\Omega }^+$ and $\widetilde {\mathcal {W}}_{\Omega }$ are complementary subspaces, $v^+_{n}$ is bounded in $\mathcal {V}_{\Omega }^+$. Then passing to a subsequence, we have $v^+_{n}\rightharpoonup v^+_{0}$ in $\mathcal {V}_{\Omega }^+$ , $v^+_{n}\longrightarrow v^+_{0}$ in $L^{2}(\Omega,\mathbb {R}^{3})$ and a.e. in $\Omega$. Therefore, by the concentration–compactness lemma 2.18, we have $\widetilde {w}_{\Omega }(v^+_{n})\longrightarrow \widetilde {w}_{\Omega }(v^+_{0})$ in $L^{2}(\Omega,\mathbb {R}^{3})$ and also a.e. in $\Omega$. Hence, we also have $u_{n}\longrightarrow u_{0}$ a.e. in $\Omega$. Then by the Viltali convergence principle, $J'_{\lambda }$ is weak-to-weak$^{\ast }$ continuous. Moreover, by the lower semi-continuity of $I_{\lambda }$, (C3) holds.

Now, we set a compactly perturbed problem. Take $X^{0}:=\widetilde {V}_{\Omega }$, $X^{1}:=\mathcal {W}_{\Omega }$ and let us consider the functional $J_{cp}: X =\mathcal {V}_{\Omega }\oplus \mathcal {W}_{\Omega }\longrightarrow \mathbb {R}$ given by

\[ J_{cp}(u)=J_{0}(u)=\frac{1}{2}\int_{\Omega}|\nabla\times u|^{2}\,{\rm d}x+\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x. \]

Moreover we define the corresponding Nehari–Pankov manifold

(4.4)\begin{equation} \mathcal{N}_{cp} = \{E\in (\mathcal{V}_{\Omega}\oplus\mathcal{W}_{\Omega})\setminus\mathcal{W}_{\Omega}: J'_{cp}(u)|_{\mathbb{R}u\oplus\mathcal{W}_{\Omega}}=0\}. \end{equation}

Observe that as in lemma 4.1 we show that $J_{cp}$ satisfies the corresponding condition (A1)–(A4) and (B1)–(B3).

Lemma 4.3 Condition (C1) in lemma 2.21 holds.

Proof. For any bounded sequence $u_{n}\subset \mathcal {N}_{cp}$, we have $u_{n}\rightharpoonup u$ in $\mathcal {N}_{cp}$. By the concentrated– compactness lemma 2.18, we have $u_{n}\longrightarrow u$ in $L^{2}(\Omega,\mathbb {R}^{3})$. Since $J_{\lambda }(u)-J_{cp}(u)=\frac {\lambda }{2}\int _{\Omega }|u|^{2}dx$, we have condition (C1) holds.

Lemma 4.4 $J_{\lambda }$ is coercive on $\mathcal {N}_{\lambda }$ and $J_{cp}$ is coercive on $\mathcal {N}_{cp}$.

Proof. The proof is similar to lemma 4.6 in [30]. Let $u_{n}=v_{n}+w_{n}\in \mathcal {N}_{\lambda }$ and suppose that $||u_{n}||\longrightarrow \infty$. Observe that

\[ J_{\lambda}(u_{n})=J_{\lambda}(u_{n})-\frac{1}{2}J'_{\lambda}(u_{n})(u_{n})=\left(\frac{1}{2}-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\right)|u_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2\cdot2^{{\ast}}_{\alpha}}\geq C_{1}|w_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2\cdot2^{{\ast}}_{\alpha}} \]

for some constant $C_{1}>0$, since $\mathcal {W}_{\Omega }$ is closed, $\mathrm {cl}\mathcal {V}_{\Omega }\cap \mathcal {W}_{\Omega }=\{0\}$ in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$ and the projection $\mathrm {cl}\mathcal {V}_{\Omega }\oplus \mathcal {W}_{\Omega }=\{0\}$ onto $\mathcal {W}_{\Omega }$ is continuous. Hence, if $|u_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}\longrightarrow \infty$, then $J_{\lambda }(u_{n})\longrightarrow \infty$ as $n\longrightarrow \infty$. Suppose that $|u_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}$ is bounded, Then $||v_{n}||\longrightarrow \infty$ and

\begin{align*} J_{\lambda}(u_{n})& =J_{\lambda}(u_{n})-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}J'_{\lambda}(u_{n})(u_{n})\\ & =\left(\frac{1}{2}-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\right)\left(\int_{\Omega}|\nabla\times v_{n}|^{2}\,{\rm d}x+\lambda\int_{\Omega}|v_{n}+w_{n}|^{2}\,{\rm d}x\right)\\ & \geq\left(\frac{1}{2}-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\right)\left(\int_{\Omega}|\nabla\times v_{n}|^{2}\,{\rm d}x+\lambda C_{2}|u_{n}|_{Q^{\alpha,2^{{\ast}}_{\alpha}}}^{2}\right), \end{align*}

for some constant $C_{2}>0$. Thus $J_{\lambda }(u_{n})\longrightarrow \infty$. Similarly, we show that $J_{cp}$ is coercive on $\mathcal {N}_{cp}$.

Lemma 4.5 Let $c_{\lambda }< c_{0}$ and $(u_{n})_{n\in \mathbb {N}}\subset \mathcal {N}_{\lambda }$ be a Palais-Smale sequence at $c_{\lambda }$, i.e. $J_{\lambda }(u_{n})\longrightarrow c_{\lambda }$ and $J'_{\lambda }(u_{n})\longrightarrow 0$ as $n\longrightarrow \infty$. Then $u_{n}\rightharpoonup u_{0}\neq 0$ for some $u_{0}$ in $W^{\alpha,2^{\ast }_{\alpha }}(\mathrm {curl};\Omega )$. Moreover, $c_{\lambda }$ is achieved by a critical point of $J_{\lambda }$.

Proof. The conclusion follows from lemmas 4.1, 4.3, 4.2, 4.4 and 2.21(a)(b).

As we introduced before, we shall verify the $(PS)_{c}$ condition. Similar to [33, lemma 6.4] we need the following version of the Brezis-Lieb lemma. Setting

\[ N(u)=\left(I_{\alpha}\ast|u|^{2^{{\ast}}_{\alpha}}\right)|u(x)|^{2^{{\ast}}_{\alpha}-2}u(x), \]

then we have the following lemma.

Lemma 4.6 Suppose $(u_{n})$ is bounded in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$ and $u_{n}\longrightarrow u$ a.e. in $\Omega$. Then

\[ N(u_{n})-N(u_{n}-u)\longrightarrow N(u)\ in\ (Q^{\alpha,2^{{\ast}}_{\alpha}}(\Omega,\mathbb{R}^{3}))'\ as\ n\longrightarrow\infty. \]

Proof. By the proof of lemma 2.7, we have $N(u):Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})\longrightarrow (Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3}))'$. Therefore, it turns to prove that $G(u_{n})-G(u_{n}-u)\longrightarrow G(u)$ in $L^{\frac {2\cdot 2^{\ast }_{\alpha }}{2\cdot 2^{\ast }_{\alpha }-1}}(\Omega,L^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}} (\Omega ))$. Since $u_{n}\longrightarrow u$ a.e. in $\Omega$, we have $G(u_{n})-G(u_{n}-u)\longrightarrow G(u)$ a.e. in $\Omega$. Since $u_{n}$ is bounded in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$, we have $G(u_{n})$ is bounded in $L^{\frac {2\cdot 2^{\ast }_{\alpha }}{2\cdot 2^{\ast }_{\alpha }-1}}\left (\Omega,L^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}(\Omega )\right )$ , so is $G(u_{n})-G(u_{n}-u)$. Then we have $G(u_{n})-G(u_{n}-u)\rightharpoonup G(u)$. Therefore, we only need to prove that

\[ |G(u_{n})-G(u_{n}-u)|_{L^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}(\Omega,L^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}(\Omega))}\rightarrow |G(u)|_{L^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}\left(\Omega,L^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}(\Omega)\right)}. \]

Indeed, let $A=\left (\frac {1}{|x-y|^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}N-\frac {2\cdot 2^{\ast }_{\alpha }}{2\cdot 2^{\ast }_{\alpha }}\alpha }}\right )^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}$. Then by using Vitali's convergence theorem we obtain

\begin{align*} & \int_{\Omega}\left(\int_{\Omega}|G(u_{n})-G(u_{n}-u)|^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}\,{\rm d}y\right)^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}\,{\rm d}x\\ & \quad=\int_{\Omega}\left(\int_{\Omega}(A)\left(|u_{n}(y)|^{2^{{\ast}}_{\alpha}}|u_{n}(x)|^{2^{{\ast}}_{\alpha}-1}\right.\right.\\ & \quad\left.\left. -\,|u_{n}(y)-u(y)|^{2^{{\ast}}_{\alpha}}|u_{n}(x)-u (x)|^{2^{{\ast}}_{\alpha}-1}\right)^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}\,{\rm d}y\right)^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}\,{\rm d}x\\ & \quad=\int_{\Omega}\left(\int_{\Omega}(A)\left(\int_{0}^{1}\frac{\mathrm{d}}{\mathrm{d}t}\left(|u_{n}(y)+(t-1)u(y)|^{2^{{\ast}}_{\alpha}}|u_{n}(x)\right.\right.\right.\\ & \quad\left.\left.\left. -\,(t-1)u(x)|^{2^{{\ast}}_{\alpha}-1}\right)^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}\,{\rm d}t\right){\rm d}y\right)^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}\,{\rm d}x\\ & \quad=\int_{0}^{1}\left[\int_{\Omega}\left(\int_{\Omega}(A)\left(\frac{(2^{{\ast}}_{\alpha})^{2}}{2^{{\ast}}_{\alpha}-1}\left\langle|u_{n}(y)+(t-1)u(y)|^{\frac{(2^{{\ast}}_{\alpha})^{2}}{2^{{\ast}}_{\alpha}-1}-2}|u_{n}(x)\right.\right.\right.\right.\\ & \quad\left.\left.\left.\left. +\,(t-1)u(x)|^{2^{{\ast}}_{\alpha}}(u_{n}(y)+(t-1)u(y)),u(y)\vphantom{\left[\int_{\Omega}\left(\int_{\Omega}(A)\left(\frac{(2^{{\ast}}_{\alpha})^{2}}{2^{{\ast}}_{\alpha}-1}\left\langle|u_{n}(y)+(t-1)u(y)|^{\frac{(2^{{\ast}}_{\alpha})^{2}}{2^{{\ast}}_{\alpha}-1}-2}|u_{n}(x)\right.\right.\right.\right.}\right\rangle\right.\right.\right.\\ & \quad\left.\left.\left.+2^{{\ast}}_{\alpha}\left\langle|u_{n}(y)+(t-1)u(y)|^{\frac{(2^{{\ast}}_{\alpha})^{2}}{2^{{\ast}}_{\alpha}-1}}|u_{n}(x)\right.\right.\right.\right.\\ & \quad\left.\left.\left.\left. +(t-1)u(x)|^{2^{{\ast}}_{\alpha}-2}(u_{n}(x)+(t-1)u(x)),u(x)\right\rangle\right){\rm d}y\right)^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}\,{\rm d}x\right]{\rm d}t\\ & \longrightarrow\int_{0}^{1}\left[\int_{\Omega}\left(\int_{\Omega}(A) \left(\frac{(2^{{\ast}}_{\alpha})^{2}}{2^{{\ast}}_{\alpha}-1}\left\langle|tu(y)|^{\frac{(2^{{\ast}}_{\alpha})^{2}}{2^{{\ast}}_{\alpha}-1}-2}|tu(x)|^{2^{{\ast}}_{\alpha}}(tu(y)),u(y)\right\rangle\right.\right.\right.\\ & \quad\left.\left.\left.+2^{{\ast}}_{\alpha}\left\langle|tu(y)|^{\frac{(2^{{\ast}}_{\alpha})^{2}}{2^{{\ast}}_{\alpha}-1}}|tu(x)|^{2^{{\ast}}_{\alpha}-2}(tu(x)),u(x)\right\rangle\right){\rm d}y\right)^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}\,{\rm d}x\right]{\rm d}t\\ & \quad=\int_{\Omega}\left(\int_{\Omega}\left(A\right)\left(|u(y)|^{2^{{\ast}}_{\alpha}}|u(x)|^{2^{{\ast}}_{\alpha}-1}\right)^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}\,{\rm d}y\right)^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}\,{\rm d}x\\ & \quad =\int_{\Omega}\left(\int_{\Omega}|G(u)|^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}\,{\rm d}y\right)^{\frac{2\cdot2^{{\ast}}_{\alpha}}{2\cdot2^{{\ast}}_{\alpha}-1}}\,{\rm d}x. \end{align*}

□

Lemma 4.7 Let $c_{\lambda }< c_{0}$ and $(u_{n})_{n\in \mathbb {N}}\subset \mathcal {N}_{\lambda }$ be the Palais-Smale sequence at $c_{\lambda }$. Then $u_{n}\longrightarrow u_{0}\neq 0$ in $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\rm {curl};\Omega )$ along a subsequence, where $u_{0}$ is the nontrivial weak limit in lemma 4.5.

Proof. Let $(u_{n})$ be a $(PS)_{c_{\lambda }}-$sequence such that $(u_{n})\subset \mathcal {N}_{\lambda }$. By lemma 4.4, $(u_{n})$ is bounded and we can assume that $u_{n}\rightharpoonup u_{0}$ in $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$. Then as in the proof of lemma 4.2, we have $J^{'}_{\lambda }(u_{0})=0$, this implies that $u_{0}$ is a solution for (1.10). Moreover, by the concentration–compactness lemma, we have $u_{n}\longrightarrow u_{0}$ in $L^{2^{\ast }_{\alpha }}_{loc}(\Omega )$, see the same analysis in lemma 4.2. On the other hand, By the compactly perturbed analysis in lemma 4.5, the weak limits $u_{0}\neq 0$. Then by the general principle for the refined nonlocal Brezis-Lieb identity in [34, proposition 4.3 (ii) (iii)], we have

\begin{align*} & \mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}\left(\int_{\Omega}(I_{\alpha}\ast|u_{n}|^{2^{{\ast}}_{\alpha}})|u_{n}|^{2^{{\ast}}_{\alpha}}\,{\rm d}x-\int_{\Omega}(I_{\alpha}\ast|u_{n}-u_{0}|^{2^{{\ast}}_{\alpha}})|u_{n}-u_{0}|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)\\ & \quad \longrightarrow \int_{\Omega}(I_{\alpha}\ast|u_{0}|^{2^{{\ast}}_{\alpha}})|u_{0}|^{2^{{\ast}}_{\alpha}}\,{\rm d}x, \end{align*}

we hence have

\[ \mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}\left(J_{\lambda}(u_{n})-J_{\lambda}(u_{n}-u_{0})\right)=J_{\lambda}(u_{0})\geq0, \]

and by lemma 4.6

\[ \mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}\left(J'_{\lambda}(u_{n})-J'_{\lambda}(u_{n}-u_{0})\right)=J'_{\lambda}(u_{0})=0. \]

Since $J'(u_{n})\longrightarrow 0$ and $u_{n}\longrightarrow u_{0}$ in $L^{2}(\Omega,\mathbb {R}^{3})$, we have

(4.5)\begin{equation} \mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}J'_{0}(u_{n}-u_{0})=0. \end{equation}

Suppose $\mathop {\mathrm {lim}~\mathrm {inf}}\limits _{n\longrightarrow \infty }||u_{n}-u_{0}||>0$. Since $\mathop {\mathrm {lim}}\limits _{n\longrightarrow \infty }J'_{0}(u_{n}-u_{0})(u_{n}-u_{0})=0$, we infer that

\[ \mathop{\mathrm{lim}~\mathrm{inf}}\limits_{n\longrightarrow\infty}|\nabla\times(u_{n}-u_{0})|_{2}>0. \]

Let $u_{n}-u_{0}=v_{n}+\widetilde {w}_{\Omega }(v_{n})\in \mathcal {V}_{\Omega }\oplus \mathcal {W}_{\Omega }$ according to the Helmholtz decomposition in $W^{\alpha,2^{\ast }_{\alpha }}_{0}(\mathrm {curl};\Omega )$. If $v_{n}\longrightarrow 0$ in $Q^{\alpha,2^{\ast }_{\alpha }}(\Omega,\mathbb {R}^{3})$, then by (4.5) we have $J'_{0}(u_{n}-u_{0})v_{n}\longrightarrow 0$, thus

\begin{align*} |\nabla\times(u_{n}-u_{0})|^{2}_{2}& =|\nabla\times v_{n}|^{2}_{2}=J'_{0}(u_{n}-u_{0})v_{n}\\ & \quad +\int_{\Omega}\left\langle\left(I_{\alpha}\ast|u_{n}-u_{0}|^{2^{{\ast}}_{\alpha}}\right)|u_{n}-u_{0}|^{2^{{\ast}}_{\alpha}-2}(u_{n}-u_{0}),v_{n}\right\rangle\,{\rm d}x\longrightarrow0 \end{align*}

as $n\longrightarrow \infty$, which is a contradiction. Therefore $|v_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}$ is bounded away from 0. If $w_{n}:=\widetilde {w}_{\Omega }(u_{n}-u_{0})\in \mathcal {W}_{\Omega }$, then $(w_{n})$ is bounded and since $u_{n}-u_{0}+w_{n}=v_{n}+\widetilde {w}_{\Omega }(v_{n})\in \mathcal {V}_{\Omega }\oplus \mathcal {W}_{\Omega }$, $|u_{n}-u_{0}+w_{n}|_{Q^{\alpha,2^{\ast }_{\alpha }}}$ is bounded away from 0. Choose $t_{n}$ so that $t_{n}(u_{n}-u_{0}+w_{n})\in \mathcal {N}_{\Omega }$, see (3.10). As in (3.12) we have

\[ t_{n}^{2}=\frac{\left(\int_{\Omega}|\nabla\times (u_{n}-u_{0})|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}-1}}} {\left(\int_{\Omega}|I_{\alpha/2}\ast|u_{n}-u_{0}+w_{n}|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}-1}}}, \]

and so $(t_{n})$ is bounded. Then using lemma 3.4, we have

\[ J_{0}(u_{n}-u_{0})\geq J_{0}(t_{n}(u_{n}-u_{0}+w_{n}))-J'_{0}(u_{n}-u_{0})\left[\frac{t^{2}_{n}-1}{2}(u_{n}-u_{0}+t^{2}_{n}w_{n})\right], \]

so by (4.5) and since $u_{n}\longrightarrow u_{0}$ in $L^{2}(\Omega,\mathbb {R}^{3})$,

\[ c_{\lambda}=\mathop{\mathrm{lin}}\limits_{n\longrightarrow\infty} J_{\lambda}(u_{n}-u_{0})=\mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}J_{0}(u_{n}-u_{0})\geq\mathop{\mathrm{lim}}\limits_{n\longrightarrow\infty}J_{0}(t_{n}(u_{n}-u_{0}+w_{n}))\geq c_{0}, \]

which is a contradiction. Therefore, passing to a subsequence, $u_{n}\longrightarrow u_{0}$, hence also in the $\mathcal {T}-$topology.

Finally, we shall compare $c_{\lambda }$ and $c_{0}$ in some ranges of $\lambda$. Recall from the third identity in (3.11), we note that $c_{0}=\frac {2^{\ast }_{\alpha }-1}{2\cdot 2^{\ast }_{\alpha }} \bar {S}^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}_{\mathrm {curl},HL}(\Omega )\geq \frac {2^{\ast }_{\alpha }-1}{2\cdot 2^{\ast }_{\alpha }}S_{HL}^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}$.

Lemma 4.8 Let $\lambda \in (-\lambda _{\nu },-\lambda _{\nu -1}]$ for some $\nu \geq 1$. There holds

\begin{align*} & c_{\lambda}=\mathop{\mathrm{inf}}\limits_{\mathcal{N}_{\lambda}}J_{\lambda}\leq\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}(\lambda+\lambda_{\nu})^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}|\mathrm{diam}\Omega|^{\frac{3\cdot2^{{\ast}}_{\alpha}-\alpha-3}{2^{{\ast}}_{\alpha}-1}},\\ & c_{\lambda}< c_{0}\ if\ \lambda<{-}\lambda_{\nu}+\bar{S}_{\mathrm{curl},HL}(\Omega)|\mathrm{diam}\Omega|^{-\frac{3\cdot2^{{\ast}}_{\alpha}-\alpha-3}{2^{{\ast}}_{\alpha}}}. \end{align*}

Proof. Let $e_{\nu }$ be an eigenvector corresponding to $\lambda _{\nu }$. Then $e_{\nu }\in \mathcal {V}_{\Omega }^+$. Choose $t>0$, $\widetilde {v}\in \widetilde {\mathcal {V}}_{\Omega }$ and $w\in \mathcal {W}_{\Omega }$ so that $u=v+w=t e_{\nu }+\widetilde {\nu }+w\in \mathcal {N}_{\lambda }$. Since $\lambda _{k}\leq \lambda _{\nu }$ for $k<\nu$,

\begin{align*} c_{\lambda}& \leq J_{\lambda}(te_{\nu}+\widetilde{v}+w)\\ & =\frac{\lambda_{\nu}}{2}\int_{\Omega}|te_{v}|^{2}\,{\rm d}x+ \frac{1}{2}\int_{\Omega}|\nabla\times \widetilde{v}|^{2}\,{\rm d}x+\frac{\lambda}{2}\int_{\Omega}|u|^{2}\,{\rm d}x-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\\ & \leq\frac{\lambda_{\nu}}{2}\int_{\Omega}|v|^{2}\,{\rm d}x+\frac{\lambda}{2}\int_{\Omega}|u|^{2}\,{\rm d}x-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x\\ & \leq\frac{\lambda+\lambda_{\nu}}{2}\int_{\Omega}|u|^{2}\,{\rm d}x-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\int_{\Omega}|I_{\alpha/2}\ast|u|^{2^{{\ast}}_{\alpha}}|^{2}\,{\rm d}x,\\ & \leq\frac{\lambda+\lambda_{\nu}}{2}\int_{\Omega}|u|^{2}\,{\rm d}x-\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\frac{1}{|\mathrm{diam}\Omega|^{3-\alpha}}\left(\int_{\Omega}|u|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{2}, \end{align*}

where $|\mathrm {diam}\Omega |=\mathop {max}\limits _{x,y\in \Omega }|x-y|$. Then using the Hölder inequality, we get

\begin{align*} c_{\lambda}& \leq\frac{\lambda+\lambda_{\nu}}{2}\left[\left(\int_{\Omega}|u|^{2^{{\ast}}_{\alpha}}dx\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\right]^{2}|\Omega|^{\frac{2^{{\ast}}_{\alpha}-2}{2^{{\ast}}_{\alpha}}} -\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\frac{1}{|\mathrm{diam}\Omega|^{3-\alpha}}\left(\int_{\Omega}|u|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{2}\\ & \leq\frac{\lambda\!+\!\lambda_{\nu}}{2}\left[\left(\int_{\Omega}|u|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{\frac{1}{2^{{\ast}}_{\alpha}}}\right]^{2}|\mathrm{diam}\Omega|^{3\cdot\frac{2^{{\ast}}_{\alpha}-2}{2^{{\ast}}_{\alpha}}} \!-\!\frac{1}{2\cdot2^{{\ast}}_{\alpha}}\frac{1}{|\mathrm{diam}\Omega|^{3-\alpha}}\left(\int_{\Omega}|u|^{2^{{\ast}}_{\alpha}}\,{\rm d}x\right)^{2}\\ & \leq\frac{2^{{\ast}}_{\alpha}-1}{2\cdot2^{{\ast}}_{\alpha}}(\lambda+\lambda_{\nu})^{\frac{2^{{\ast}}_{\alpha}}{2^{{\ast}}_{\alpha}-1}}|\mathrm{diam}\Omega|^{\frac{3\cdot2^{{\ast}}_{\alpha}-\alpha-3}{2^{{\ast}}_{\alpha}-1}}, \end{align*}

where the last inequality follows from the inequality $\frac {A}{2}t^{2}-\frac {1}{p}t^{p}\leq (\frac {1}{2}-\frac {1}{p})A^{\frac {p}{p-2}} (A>0)$.

Since $c_{0}=\frac {2^{\ast }_{\alpha }-1}{2\cdot 2^{\ast }_{\alpha }} \bar {S}^{\frac {2^{\ast }_{\alpha }}{2^{\ast }_{\alpha }-1}}_{\mathrm {curl},HL}(\Omega )$, the second inequality follows immediately.

Proof. Complete of the Proof of theorem 1.4

Note that if $\lambda <-\lambda _{\nu }+\bar {S}_{\mathrm {curl},HL}(\Omega )|\mathrm {diam}\Omega |^{-\frac {3\cdot 2^{\ast }_{\alpha }-\alpha -3}{2^{\ast }_{\alpha }}}$, then $c_{\lambda }< c_{0}$, and by lemma 4.7, $J_{\lambda }$ satisfies the $(PS)_{c_{\lambda }}$ condition, hence satisfies the $(PS)_{c_{\lambda }}^{\mathcal {T}}$ condition. Then statement (a) follows from lemma 4.5, and the remaining statements (b)–(d) are similar to [33, theorem 1.4] and can be proved by the same strategy.