A Berestycki–Lions type result for a class of degenerate elliptic problems involving the Grushin operator

Abstract

In this work we study the existence of nontrivial solution for the following class of semilinear degenerate elliptic equations

\[ -\Delta_{\gamma} u + a(z)u = f(u) \quad \mbox{in}\ \mathbb{R}^{N}, \]

where $\Delta _{\gamma }$ is known as the Grushin operator, $z:=(x,y)\in \mathbb {R}^{m}\times \mathbb {R}^{k}$ and $m+k=N\geqslant 3$, $f$ and $a$ are continuous function satisfying some technical conditions. In order to overcome some difficulties involving this type of operator, we have proved some compactness results that are crucial in the proof of our main results. For the case $a=1$, we have showed a Berestycki–Lions type result.

1. Introduction

In this paper, we study the existence of nontrivial solutions for the following class of semilinear degenerate elliptic equations

\[{(P)_{a}} \quad \quad \quad -\Delta_{\gamma} u + a(z)u = f(u) \quad \mbox{in} \ \mathbb{R}^{N}, \]

where $\Delta _{\gamma }$ given by

\[ \Delta_{\gamma}u(z)=\Delta_{x}u(z)+|x|^{2\gamma}\Delta_{y}u(z) \]

is known as the Grushin operator, $\Delta _x$ and $\Delta _y$ are the Laplace operators in the variables $x$ and $y$, respectively, with $z:=(x,y)\in \mathbb {R}^{m}\times \mathbb {R}^{k}$ and $m+k=N\geqslant 3$. Here, $\gamma \geqslant 0$ is a real number and $N_{\gamma }=m+(1+\gamma )k$ is the appropriate homogeneous dimension. The nonlinearity $f:\mathbb {R} \longrightarrow \mathbb {R}$ and the weight $a:\mathbb {R}^{N}\longrightarrow \mathbb {R}$ are continuous functions that satisfy some technical conditions.

To begin with, we would like to observe that $\Delta _{\gamma }$ is elliptic for $x\in \mathbb {R}^{m}$, $|x|\neq 0$ and degenerates on the manifold $\{0\}\times \mathbb {R}^{k}$. This operator was studied by Grushin in [20, 21], and it appears in the study of P.D.E. on manifolds. For example, in [24, p. 4641] the authors showed that problem $(P)_a$ appears in the study of a P.D.E. in the Heisenberg group, while in [9] it was showed a strong relation with a P.D.E. on the Hyperbolic space. Furthermore, if $\gamma =0$, the operator $\Delta _{0}$ is the well-known classical Laplace operator. On the other hand, $\Delta _{\gamma }$ is contained in a family of operators of the kind

(1.1)\begin{equation} \Delta_\lambda:=\sum_{i=1}^{N}\partial_{x_i}(\lambda_i^{2}\partial_{x_i}), \quad \lambda=(\lambda_1,\ldots,\lambda_N), \end{equation}

which was first introduced and studied by Franchi and Lanconelli [18, 19]. For the interested read we cite for instance [5, 24–26, 32, 37] and references therein.

In the last few years some authors have dedicated a special attention for problems involving the Grushin operator, see for example [14, 15, 33, 34] and more recently [16, 17, 30, 31].

The main motivation to study the above problem $(P)_a$ comes from of the papers due to [6, 7], that have considered the case $\gamma =0$ and $a(z)=1$ for all $z\in \mathbb {R}^{N}$. More precisely, in [6], Berestycki and Lions have considered the existence of solution for

(1.2)\begin{equation} - \Delta u = g(u), \quad \mbox{in} \ \mathbb{R}^{N}, \end{equation}

by assuming that $N \geqslant 3$ and the following conditions on $g$:

\[{(g_1)} \quad\quad\quad - \infty < \liminf_{s \to 0^{+}}\frac{g(s)}{s} \leqslant \limsup_{s \to 0^{+}}\frac{g(s)}{s}\leqslant -m<0, \]

\[{(g_2)} \quad\quad\quad \limsup_{s \to +\infty}\frac{g(s)}{s^{2^{*}-1}}\leqslant 0, \]

\[{(g_3)}\quad\quad\quad \mbox{there is} \quad \xi>0 \ \mbox{such that} \ G(\xi)>0,\]

where $G(s)=\int _{0}^{s}g(t)\,{\rm d}t$.

In [7], Berestycki et al. have studied the case where $N=2$ and the nonlinearity $g$ possesses an exponential growth of the type

\[ \limsup_{s \to +\infty}\frac{g(s)}{{\rm e}^{\beta s^{2}}}=0, \quad \forall \beta >0. \]

In the two abovementioned papers, the authors obtained a solution for (1.2) by solving in the first moment a minimization problem, where the properties of the radially symmetric functions play an essential rule in their approach. After that, by using Lagrange multipliers, they were able to get a nontrivial solution for (1.2).

A version of the problem (1.2) for the critical case have been made in [2] for $N \geqslant 3$ and $N=2$, see also [41] for $N=3$. The reader can found in [3, 8, 42] the same type of results involving the fractional Laplacian operator, more precisely, for a problem like

\[ (- \Delta)^{\alpha}u = g(u), \quad \mbox{in} \ \mathbb{R}^{N}, \]

with $\alpha \in (0,1)$ and $N \geqslant 1$. For more details about this subject, we would like to cite the references found in the abovementioned papers.

Recently, Alves et al. [1] have proved an abstract theorem that was used to solve a large class of Berestycki–Lions type problems, which includes anisotropic operator, discontinuous nonlinearity, etc. In that paper, the authors have used the deformation lemma together with a notation of Pohozaev set to prove the abstract theorem.

Initially, in the present paper, we will deal with the existence of nontrivial solution for the problem $(P)_1$, that is,

\[{(P)_{1}}\quad\quad\quad -\Delta_{\gamma} u + u = f(u) \quad \mbox{in} \ \mathbb{R}^{N}, \]

for the positive mass case. In this case, the nonlinearity $f$ satisfies the following conditions:

1. (f1) $f(s)/s = o(1)$ as $s \to 0$ ;

2. (f2) There exists $q \in (2,2^{*}_{\gamma })$ such that

   \[ \limsup_{|s| \to +\infty}\frac{|f(s)|}{|s|^{q-1}} <{+}\infty, \quad \forall s \in \mathbb{R}, \]
   where $2^{*}_{\gamma }={2N_{\gamma }}/({N_{\gamma }-2})$;

3. (f3) There exists $s_0>0$ such that $F(s_0)>\frac {1}{2}|s_0|^{2}$, where $F(s) = \int _0^{s} f(t)\,{\rm d}t$.

Using the above assumptions we are ready to state our first result.

Theorem 1.1 (The positive mass case: $a(z)=1$)

Assume $0 \leqslant \gamma < 1$, $(f_1),(f_2)$ and $(f_3)$. Then problem $(P)_1$ has at least one nonnegative nontrivial weak solution.

Our next result is associated with the zero mass case due to [6], more precisely, we intend to find a solution for the following class of problems

\[{(P)_0}\quad\quad\quad -\Delta_{\gamma} u = f(u) \quad \mbox{in} \ \mathbb{R}^{N}, \]

where $f$ satisfies some conditions that will be fixed later on.

Problems of this type have attracted the attention of many mathematicians with regards to the non-existence of a solution. For instance, if $\Delta _{H}$ is the Heisenberg Laplacian on $H^{N}=\mathbb {C}^{N}\times \mathbb {R}$ and $u$ is radial in the variable $x$, $u=u(|x|,y)$, then $-\Delta _{H}u =\Delta _{x}u+4|x|^ 2\partial _{yy}u$ is a Grushin operator with $\gamma =1$, and so, $(P)_0$ is strongly related to equation

\[ -\Delta_{H} u = f(u) \quad \mbox{in} \ H^{N}. \]

In [10], Birindelli and Prajapat studied the nonexistence of cylindrical solution for above problem with $f(u)= u^{p}$, $0< p<(Q+2)/(Q-2)$, where $Q$ is the homogeneous dimension of $H^{N}$. Recently, Yu [39] studied the Liouville type theorem in the Heisenberg group for general nonlinearity. In [40], Yu studied the nonexistence of positive solutions for a general Grushin operator, and the main tool in this paper is the moving plane method. Monti and Morbidelli in [34] investigated the uniqueness and symmetry of solutions for the problem $(P)_0$ with critical growth and introduced a suitable Kelvin-type transform in this context. Finally, related to the bounded domains, Kogoj and Lanconelli [24] showed regularity, existence and Pohozaev-type nonexistence results for the Dirichlet problem

\[ \left \{ \begin{array}{ll} -\Delta_{\lambda} u = f(u) & \mbox{in} \ \Omega, \\ u = 0 & \mbox{on} \ \partial\Omega, \end{array} \right. \]

where $\Omega$ is a bounded open subset of $\mathbb {R}^{N}$, for general nonlinearities $f$.

In order to solve $(P)_0$, we are assuming the following conditions on function $f$:

1. (f4) There are $q >2^{*}_{\gamma }$ and $c,\delta >0$ such that

   \[ |f(s)| \leqslant c|s|^{q-1}, \quad \mbox{for} \ |s| \leqslant \delta. \]

2. (f5) $\displaystyle \lim _{|s| \to +\infty }\frac {f(s)}{|s|^{2^{*}_{\gamma }-1}}=0.$

3. (f6) There is $s_0>0$ such that $F(s_0)>0$.

Theorem 1.2 (The zero mass case: $a(z)=0$)

Assume $0 \leqslant \gamma < 1$, $(f_4),(f_5)$ and $(f_6)$. Then problem $(P)_0$ has at least one nonnegative nontrivial weak solution.

Related to a more general class of degenerate operators, in $2017$, Chen et al. studied in [13] (see also [5, 24, 32, 37]) the following problem

\[ \left \{ \begin{array}{@{}ll} -\Delta_{\lambda} u + a(z)u = f(x,u) & \mbox{in} \ \Omega, \\ u = 0 & \mbox{on}\ \partial\Omega, \end{array} \right. \]

where $\Omega$ is a bounded domain in $\mathbb {R}^{N}$, $\Delta _{\lambda }$-Laplace is given by (1.1), $a(z)$ is allowing to be sign-changing and $f$ is a function with a more general super-quadratic growth.

If we suppose that $f$ satisfies the well-known Ambrosetti–Rabinowitz superlinear condition, that is, there exists $\theta > 2$ such that

\[{(f_7)}\quad\quad\quad 0<\theta F(t)\leqslant f(t)t, \quad t\in\mathbb{R}\backslash\{0\},\]

we are able to establish the existence of nontrivial solution for the problem $(P)_a$ for all $\gamma \geqslant 0$ by supposing that

(A1)\begin{equation} a(z)\geqslant a_0>0, \quad \hbox{for all}\ z\in\mathbb{R}^{N} \end{equation}

and

(A2)\begin{equation} a(x,y)=a(x',y) \quad \hbox{for all}\ x, x'\in\mathbb{R}^{m}\ \hbox{with} |x|=|x'|\ \hbox{and all}\ y\in\mathbb{R}^{k}. \end{equation}

In addition to the above conditions, we will consider one of the following conditions:

The function $a$ is periodic on variable $y$, that is,

(A3)\begin{equation} a(x,y)=a(x,y+y_0),\quad \hbox{for all}\ x\in\mathbb{R}^{m}, y\in\mathbb{R}^{k} \hbox{and} \ y_0\in\mathbb{Z}^{k}, \end{equation}

or the function $a$ is coercive on variable $y$, that is,

(A4)\begin{equation} a(x,y)\longrightarrow +\infty\quad \hbox{when}\ |y|\longrightarrow\infty, \hbox{uniformly for}\ x\in\mathbb{R}^{m}. \end{equation}

Our main result involving the above conditions is the following:

Theorem 1.3 (Nonconstant case)

Assume $\gamma \geqslant 0$, $(f_1),(f_2)$ and $(f_7)$ and let $a:\mathbb {R}^{N}\longrightarrow \mathbb {R}$ be a continuous function satisfying (A1) and (A2). If (A3) or (A4) holds, then problem $(P)_a$ has at least one nonnegative nontrivial weak solution.

Theorem 1.3 complements the study made in [12, 27, 36], because in those papers it was considered the case $\gamma =0$ and $a$ being a periodic function.

Our last result is associated with the behaviour of the solutions at infinite.

Theorem 1.4 Let $u$ be a solution of problem $(P)_a$ with $a \not =0$. Then $u\in L^{\infty }(\mathbb {R}^{N})$ and $\lim _{|z|\to \infty }u(z)=0.$

In what follows, we say that a function $u\in \mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ is a weak solution to problem $(P)_a$, with $a\neq 0$, if $u\geqslant 0$ a.e. in $\mathbb {R}^{N}$, $u \not \equiv 0$ and

(1.3)\begin{equation} \int_{\mathbb{R}^{N}}\big(\nabla_{\gamma} u\nabla_{\gamma}\varphi +a(z)u\varphi \big)\,{\rm d}z = \int_{\mathbb{R}^{N}} f(u)\varphi \,{\rm d}z, \quad \forall \varphi\in\mathcal{H}_{\gamma}^{a}(\mathbb{R}^{N}). \end{equation}

Similarly, $u\in \mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$ is a weak solution to problem $(P)_0$ if (1.3) holds for all $\varphi \in \mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$. The spaces $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$, $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$ and the notation $\nabla _{\gamma }$ are defined in § 2.

Before concluding this section, we would like to point out that in the proof of theorems 1.1, 1.2 and 1.3 we have found some difficulties to apply variational methods when $\gamma >0$, because different from the case $\gamma =0$, the energy functional associated with problem $(P)_a$ is not invariant on the group $O(N)$ and it is also not invariant by for some types of translations. These facts bring to the problem some technical difficulties that must be analysed in a very cautious way. Having this in mind, in § 2, we made a careful study about the space functions that will be used in this paper, which includes compactness results and a version of a result due to [29, lemma I.1] for the space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

Throughout the paper, unless explicitly stated, the symbol ${C}$ will always denote a generic positive constant, which may vary from line to line. The usual norm in $L^{p}(\mathbb {R}^{N})$ will be denoted by $|\cdot |_{p}$ and $H^{1}(\Omega )$ will be used to denote standard Sobolev spaces in $\Omega \subset \mathbb {R}^{N}$.

2. Function spaces

In this section, we will introduce some notations and preliminary results that will be used later on. First of all, let $\gamma$ be a nonnegative real number and $z=(x,y)=(x_1,\ldots,x_m,y_1,\ldots,y_k)\in \mathbb {R}^{m}\times \mathbb {R}^{k}$ with $m, k\geqslant 1$ and $m+k=N\geqslant 3$. In what follows, we designate by $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ the weighted Sobolev space

\[ \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big)=\left\{u\in L^{2}\big(\mathbb{R}^{N}\big):\frac{\partial u}{\partial x_i}, ~|x|^{\gamma}\frac{\partial u}{\partial y_j}\in L^{2}\big(\mathbb{R}^{N}\big), i=1,\ldots,m, j=1,\ldots,k\right\}. \]

For simplicity, if $u\in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, we denote by $\nabla _{\gamma }$ the gradient operator defined by

\[ \nabla_{\gamma}u=\left(\nabla_{x}u, |x|^{\gamma}\nabla_{y} u \right)= \left(u_{x_1},\ldots,u_{x_m}, |x|^{\gamma}u_{y_1},\ldots,|x|^{\gamma}u_{y_k} \right), \]

and so,

\[ |\nabla_{\gamma}u|^{2}=|\nabla_{x}u|^{2}+|x|^{2\gamma}|\nabla_{y}u|^{2}. \]

We would like to mention that $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ is a Hilbert space, when endowed with the scalar product given by

\[ \langle u,v \rangle_{\mathcal{H}_{\gamma}^{1,2}(\mathbb{R}^{N})} = \int_{\mathbb{R}^{N}}\big(\nabla_{\gamma} u\nabla_{\gamma} v + uv \big) \,{\rm d}z, \]

whose the corresponding norm is

\[ ||u||_{\gamma}:=\sqrt{\langle u,u \rangle_{\mathcal{H}_{\gamma}^{1,2}(\mathbb{R}^{N})}}=\left(\int_{\mathbb{R}^{N}}\big(|\nabla_{\gamma}u|^{2}+|u|^{2}\big) \, {\rm d}z\right)^{{1}/{2}}. \]

By Sobolev inequality found in [33], we have

(2.1)\begin{equation} \left(\int_{\mathbb{R}^{N}}|u|^{2_{\gamma}^{*}} \, {\rm d}z\right)^{2/2_{\gamma}^{*}}\leqslant C(m,k,\gamma)\int_{\mathbb{R}^{N}}|\nabla_{\gamma}u|^{2} \, {\rm d}z, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big), \end{equation}

from where it follows that the embedding $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{2_{\gamma }^{*}}(\mathbb {R}^{N})$ is continuous, then there is $\mathcal {C}>0$ such that

\[ |u|_{2_{\gamma}^{*}}\leqslant \mathcal{C}||u||_{\gamma}, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2} \big(\mathbb{R}^{N}\big). \]

Since

\[ |u|_{2}\leqslant ||u||_{\gamma}, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big), \]

the interpolation inequality on the Lebesgue space ensures that the embedding $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{p}(\mathbb {R}^{N})$ is continuous for every $p\in [2, 2^{*}_{\gamma }]$. Thus, for each $p \in [2, 2^{*}_{\gamma }]$ there is $\mathcal {C}_p>0$ such that

(2.2)\begin{equation} |u|_{p}\leqslant \mathcal{C}_p||u||_{\gamma}, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big). \end{equation}

Here, we would like point out that if $\Omega _1 \subset \mathbb {R}^{m}$ and $\Omega _2 \subset \mathbb {R}^{k}$ are smooth bounded domains, the embeddings (2.2) still hold by replacing $\mathbb {R}^{N}$ for any set of the form $\Omega =\mathbb {R}^{m} \times \Omega _2$, $\Omega =\Omega _1 \times \mathbb {R}^{k}$ or $\Omega =\Omega _1 \times \Omega _2$, that is,

(2.3)\begin{equation} |u|_{L^{p}(\Omega)}\leqslant \mathcal{C}_p||u||_{\mathcal{H}_{\gamma}^{1,2}(\Omega)}, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2}(\Omega), \end{equation}

where

\[ \mathcal{H}_{\gamma}^{1,2}\big(\Omega\big)=\left\{u\in L^{2}\big(\Omega\big):\frac{\partial u}{\partial x_i}, \ |x|^{\gamma}\frac{\partial u}{\partial y_j}\in L^{2}\big(\Omega\big), i=1,\ldots,m, j=1,\ldots,k\right\} \]

and

\[ ||u||_{\mathcal{H}_{\gamma}^{1,2}(\Omega)}:=\left(\int_{\Omega}\big(|\nabla_{\gamma}u|^{2}+|u|^{2}\big) \, {\rm d}z\right)^{{1}/{2}}. \]

The proof of (2.3) follows by showing the existence of an extension operator that can be done as in [11, § 9.2]. Moreover, when $\Omega =\Omega _1 \times \Omega _2 \subset \mathbb {R}^{N}=\mathbb {R}^{m} \times \mathbb {R}^{k}$ the embedding $\mathcal {H}_{\gamma }^{1,2}(\Omega ) \hookrightarrow L^{p}(\Omega )$ is compact for every $p\in [1, 2^{*}_{\gamma })$, which is a consequence of the Strauss's compactness lemma [6, theorem A.I].

Setting the space

\[ \mathcal{H}_{\gamma}^{a}(\mathbb{R}^{N})=\left\{u\in\mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big):\int_{\mathbb{R}^{N}}a(z)|u|^{2} \, {\rm d}z <\infty\right\}, \]

it is easy to see that condition (A1) implies that $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ is Hilbert space with scalar product and norm given by

\[ \langle u,v \rangle_{a} = \int_{\mathbb{R}^{N}}\big(\nabla_{\gamma} u\nabla_{\gamma} v + a(z)uv \big) \,{\rm d}z \quad \hbox{and} \quad ||u||^{2}:=\sqrt{\langle u,u \rangle_{a}}. \]

Moreover, (A1) also yields $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ is embedded continuously into $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$. Hence, by (2.2), $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ is embedded continuously in $L^{p}(\mathbb {R}^{N})$ for every $p\in [2, 2^{*}_{\gamma }]$.

Another important space that will consider in the present paper is $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$, which is the closure of $C_{0}^{\infty }(\mathbb {R}^{N})$ with relation to the norm

\[ \|\varphi \|_{\mathcal{D}_{\gamma}^{1,2}(\mathbb{R}^{N})}=\left(\int_{\mathbb{R}^{N}}|\nabla_{\gamma}u|^{2} \, {\rm d}z\right)^{{1}/{2}}, \quad \varphi \in C_{0}^{\infty}(\mathbb{R}^{N}). \]

From (2.1) we derive that the embedding $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{2^{*}_{\gamma }}(\mathbb {R}^{N})$ is continuous, that is, there is $\mathcal {S}>0$ such that

\[ |u|_{2^{*}_{\gamma}}\leqslant \mathcal{S}\|u\|_{\mathcal{D}_{\gamma}^{1,2}(\mathbb{R}^{N})}, \quad \forall u \in \mathcal{D}_{\gamma}^{1,2}(\mathbb{R}^{N}). \]

2.1 Compactness results

In this section, we will prove some important compactness results that will be used in the proof of theorems 1.1, 1.2 and 1.3. Our first result is a version of a famous result due to [29, lemma I.1] for the space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

Lemma 2.1 Let $r>0, 2\leqslant q < 2^{*}_{\gamma }$ and $(u_n) \subset \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ be a bounded sequence with

\[ \sup_{y\in\mathbb{R}^{k}}\int_{\mathbb{R}^{m}\times B^{k}_r(y)}|u_n|^{q}\,{\rm d}z \longrightarrow 0 \quad \mbox{as} n\longrightarrow\infty, \]

where $B_r^{k}(y)=\{\xi \in \mathbb {R}^{k}\,:\,|\xi - y|< r\}.$ Then, $u_n\longrightarrow 0$ in $L^{p}(\mathbb {R}^{N})$ for $2< p<2^{*}_{\gamma }$.

Proof. As a first step we consider

\[ \mathbb{R}^{N}=\mathbb{R}^{m}\times\mathbb{R}^{k}\subset\bigcup_{y\in\mathbb{R}^{k}}\left(\mathbb{R}^{m}\times B_r(y)\right), \]

in such way that each point of $\mathbb {R}^{k}$ is contained in a finite quantity $l$ of balls of radius $r$. Now, let $q< s<2^{*}_{\gamma }$ and $\alpha :={(s-q)2^{*}_{\gamma }}/{(2^{*}_{\gamma }-q)s}$. Then, if $u\in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, the embedding (2.3) together with the interpolation inequality on the Lebesgue space says that

\begin{align*} |u|_{L^{s}(\mathbb{R}^{m}\times B_r(y))} & \leqslant |u|^{1-\alpha}_{L^{q}(\mathbb{R}^{m}\times B_r(y))}|u|^{\alpha}_{L^{2^{*}_{\gamma}}(\mathbb{R}^{m}\times B_r(y))} \\ & \leqslant \mathcal{C}|u|^{1-\alpha}_{L^{q}(\mathbb{R}^{m}\times B_r(y))}\left(\int_{\mathbb{R}^{m}\times B_r(y)}\big(|\nabla_{\gamma}u|^{2}+|u|^{2}\big) \, {\rm d}z\right)^{{\alpha}/{2}}. \end{align*}

Choosing $\alpha =2/s$, we obtain the inequality

\[ \int_{\mathbb{R}^{m}\times B_r(y)}|u|^{s} \, {\rm d}z\leqslant \mathcal{C}^{s}|u|^{(1-\alpha)s}_{L^{q}(\mathbb{R}^{m}\times B_r(y))}\int_{\mathbb{R}^{m}\times B_r(y)}\big(|\nabla_{\gamma}u|^{2}+|u|^{2}\big) \, {\rm d}z, \]

which leads to

\[ \int_{\mathbb{R}^{N}}|u|^{s} \, {\rm d}z \leqslant \mathcal{C}^{s}l\sup_{y\in\mathbb{R}^{k}}\left(\int_{\mathbb{R}^{m}\times B_r(y)}|u|^{q} \,{\rm d}z\right)^{(1-\alpha)s/q}\int_{\mathbb{R}^{N}}\big(| \nabla_{\gamma}u|^{2}+|u|^{2}\big)\, {\rm d}z. \]

As $(u_n)$ is bounded sequence in $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ the last inequality allows us to conclude that $u_n\to 0$ in $L^{s}(\mathbb {R}^{N})$. Now, applying again the interpolation inequality on the Lebesgue spaces we conclude that $u_n \to 0$ in $L^{p}(\mathbb {R}^{N})$ for $2< p<2^{*}_{\gamma }$.

Next, we will show some compactness results that are crucial in our approach. Have this in mind, we will fix the following spaces:

\[ \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big)=\left\{\mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big):u(x,y)=u(|x|,|y|) \ \hbox{for all}\ (x,y) \in \mathbb{R}^{m} \times \mathbb{R}^{k}\right\}, \]

\[ \mathcal{H}_{\gamma,rad_x}^{1,2}\big(\mathbb{R}^{N}\big)=\left\{\mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big):u(x,y)=u(|x|,y) \ \hbox{for all} \ (x,y) \in \mathbb{R}^{m} \times \mathbb{R}^{k}\right\}, \]

\begin{align*} \mathcal{H}_{\gamma,rad_x}^{1,2}(\mathbb{R}^{m} \times B_r(0))& =\left\{\mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{m} \times B_r(0)):u(x,y)\right.\nonumber\\ & \left.\quad =u(|x|,y)\ \hbox{for all}\ (x,y) \in \mathbb{R}^{m} \times B_r(0)\right\}, \end{align*}

\[ \mathcal{H}_{\gamma, rad_x}^{a}\big(\mathbb{R}^{N}\big)=\left\{\mathcal{H}_{\gamma}^{a}\big(\mathbb{R}^{N}\big):u(x,y)=u(|x|,y) \ \hbox{for all}\ (x,y) \in \mathbb{R}^{m} \times \mathbb{R}^{k}\right\} \]

and

\[ \mathcal{D}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big)=\left\{\mathcal{D}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big):u(x,y)=u(|x|,|y|) \ \hbox{for all}\ (x,y) \in \mathbb{R}^{m} \times \mathbb{R}^{k}\right\}. \]

Lemma 2.2 Let $p\in (2,2^{*}_{\gamma })$ and $r>0$. Then, the embedding of $\mathcal {H}_{\gamma,rad_x}^{1,2}(\mathbb {R}^{m}\times B_r(0))$ into $L^{p}(\mathbb {R}^{m}\times B_r(0))$ is compact.

Proof. It is enough to show that if $(u_n)$ is a sequence in $\mathcal {H}_{\gamma,rad_x}^{1,2}(\mathbb {R}^{m}\times B_r(0))$ such that $u_n \rightharpoonup 0$ in $\mathcal {H}_{\gamma,rad_x}^{1,2}(\mathbb {R}^{m}\times B_r(0))$, then $u_n\to 0$ in $L^{p}(\mathbb {R}^{m}\times B_r(0))$. Let $\delta >0$ and $h \in C^{\infty }(\mathbb {R},[0,1])$ defined by

\[ h(t)=\left \{ \begin{array}{@{}ll} 0 & \mbox{if} \ |t|\leqslant \delta/2, \\ 1 & \mbox{if} \ |t|\geqslant \delta. \end{array} \right. \]

Notice that if $u\in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{m}\times B_r(0))$ and

\[ v(z):=h(|x|^{2})u(z), \quad z=(x,y)\in\mathbb{R}^{m}\times B_r(0), \]

then $|v|^{2}\leqslant |u|^{2}$,

\[ \left|\frac{\partial v}{\partial x_i}\right|^{2}\leqslant {C}\bigg(|u|^{2}+\left|\frac{\partial u}{\partial x_i}\right|^{2} \bigg), \quad i=1,\ldots,m \]

and

\[ \left|\frac{\partial v}{\partial y_j}\right|^{2} = h^{2}(|x|^{2})\left|\frac{\partial u}{\partial y_j}\right|^{2}, \quad j=1,\ldots,k. \]

Hence,

\begin{align*} \int_{\mathbb{R}^{m}\times B_r(0)}|\nabla_{y}v|^{2}\,{\rm d}z & = \int_{[|x|\geqslant\delta/2]\times B_r(0)}h^{2}(|x|^{2})|\nabla_{y}u|^{2}\,{\rm d}z \\ & \leqslant {C}\int_{[|x|\geqslant\delta/2]\times B_r(0)}|x|^{2\gamma}|\nabla_{y}u|^{2}\,{\rm d}z \\ & \leqslant {C}\int_{\mathbb{R}^{m}\times B_r(0)}|x|^{2\gamma}|\nabla_{y}u|^{2}\,{\rm d}z. \end{align*}

Therefore, $v\in H^{1}(\mathbb {R}^{m}\times B_r(0))$ and

\[ ||v||_{H^{1}(\mathbb{R}^{m}\times B_r(0))}\leqslant {C}||u||_{\mathcal{H}_{\gamma}^{1,2}(\mathbb{R}^{m}\times B_r(0))}. \]

Thereby, setting $v_n(x,y)=h(|x|^{2})u_n(x,y)$ we have $(v_n)$ is a bounded sequence in $H_{rad,x}^{1}(\mathbb {R}^{m}\times B_r(0))$ and $v_n \rightharpoonup 0$ in $H_{rad,x}^{1}(\mathbb {R}^{m}\times B_r(0))$. By [28, lemme III.2],

\[ v_n\to 0 \quad \hbox{in} \ L^{p}(\mathbb{R}^{m}\times B_r(0)),\quad 2< p<2^{*}. \]

Since $v_n(x,y)=u_n(x,y)$ for $(x,y) \in [|x| \geqslant \delta ] \times B_r(0)$, we derive that

(2.4)\begin{equation} u_n\to 0 \quad \hbox{in}\ L^{p}([|x| \geqslant \delta] \times B_r(0)),\quad 2< p<2^{*}. \end{equation}

Furthermore, recalling that if $\Omega =B_\delta (0) \times B_r(0) \subset \mathbb {R}^{N}$ the embedding $\mathcal {H}_{\gamma }^{1,2}(\Omega )$ into $L^{p}(\Omega )$ is compact for $1\leqslant p<2_{\gamma }^{*}$, then

(2.5)\begin{equation} u_n\to 0 \quad \hbox{in} L^{p}([|x|\leqslant\delta]\times B_r(0)). \end{equation}

Accordingly to (2.4) and (2.5), $u_n\to 0$ in $L^{p}(\mathbb {R}^{m}\times B_r(0))$ with $p\in (2,2^{*}_{\gamma })$.

Lemma 2.3 Let $p\in (2,2^{*}_{\gamma })$. Then the embedding of $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{p}(\mathbb {R}^{N})$ is compact.

Proof. In what follows, we set $q \in (2,2^{*}_{\gamma })$ and

\[ k(y,r,O(k))=\{n \in \mathbb{N}\,:\, \exists g_1,g_2,\ldots,g_n \in O(k)\,:\,B^{k}_{r}(g_i(y))\cap B^{k}_r(g_j(y))=\emptyset\}. \]

From definition of $k(y,r,O(k))$,

\[ \lim_{|y| \to +\infty}k(y,r,O(k))={+}\infty. \]

A simple computation gives that for each $u \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$,

\[ \int_{\mathbb{R}^{m} \times B^{k}_{r}(y)}|u|^{q}\,{\rm d}z \leqslant \frac{|u|_q^{q}}{k(y,r,O(k))}. \]

Arguing as in the last lemma, let $(u_n)$ be a sequence in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that $u_n \rightharpoonup 0$ in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. As $(u_n)$ is a bounded sequence in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, given $\epsilon >0$, the definition of $k(y,r,O(k))$ ensures that there is $R>0$ such that

\[ \sup_{|y| \geqslant R}\int_{\mathbb{R}^{m} \times B^{k}_{r}(y)}|u_n|^{q}\,{\rm d}z< \frac{\epsilon}{2}, \quad \forall n \in\mathbb{N}. \]

On the other hand, lemma 2.2 yields

\[ \int_{\mathbb{R}^{m} \times B_{R+r}(0)}|u_n|^{q}\,{\rm d}z \to 0, \]

and so,

\[ \sup_{y \in \mathbb{R}^{k}}\int_{\mathbb{R}^{m} \times B^{k}_{r}(y)}|u_n|^{q}\,{\rm d}z \to 0. \]

Now the lemma follows by employing lemma 2.1.

Corollary 2.4 If (A1) holds, then the embedding $\mathcal {H}^{a}_{\gamma,rad_{x,y}}(\mathbb {R}^{N}) \hookrightarrow L^{p}(\mathbb {R}^{N})$ is compact.

Proof. The corollary is an immediate consequence of the last lemma, because we have the continuous embedding of $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ into $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

Lemma 2.5 Let $(u_n) \subset \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ be a bounded sequence with $u_n \rightharpoonup 0$ in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. Then, for each $0< a< b<+\infty$ and $p \in (2,2^{*}_{\gamma })$,

\[ \int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z \to 0. \]

Proof. In the sequel, let us consider $g \in C^{\infty }(\mathbb {R},\mathbb {R})$ satisfying

\[ g(t)=0 \quad \mbox{for} \ |t| \leqslant \frac{a}{2} \quad \mbox{or} \ |t| \geqslant b+1 \]

and

\[ g(t)=t \quad \mbox{for} \ a\leqslant |t| \leqslant b. \]

Using the function $g$, we set the sequence $v_n(x,y)=g(u_n(x,y))$. A direct computation shows that $v_n \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ and that it is a bounded sequence in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. Thereby, by lemma 2.3, $v_n \to 0$ in $L^{p}(\mathbb {R}^{N})$, that is,

\[ \int_{\mathbb{R}^{N}}|v_n|^{p}\,{\rm d}z \to 0, \]

and so,

\[ \int_{\{a<|u_n|< b\}}|v_n|^{p}\,{\rm d}z \to 0. \]

Using the fact that $g(t)=t$ for $a \leqslant |t| \leqslant b$, it follows that

\[ \int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z \to 0, \]

showing the lemma.

Lemma 2.6 Assume that (A1) and (A4) hold. Then the embedding of $\mathcal {H}_{\gamma,rad_x}^{a}(\mathbb {R}^{N}) \hookrightarrow L^{p}(\mathbb {R}^{N})$ is compact for all $p\in [2,2^{*}_{\gamma })$.

Proof. It suffices to show that for any sequence $(u_n) \subset \mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$ with $u_n \rightharpoonup 0$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$, we must have $u_n\to 0$ in $L^{p}(\mathbb {R}^{N})$. By (A4), given $\varepsilon >0$, we can take $R_{\varepsilon }>0$ such that

\[ \frac{1}{a(x,y)}\leqslant\frac{\varepsilon}{2}, \quad \hbox{for all} (x,y)\in\mathbb{R}^{m}\times [|y|\geqslant R_{\varepsilon}]. \]

From this,

\[ \int_{\mathbb{R}^{m}\times [|y|\geqslant R_{\varepsilon}]} |u_n|^{2} \, {\rm d}z \leqslant\frac{\varepsilon}{2}\int_{\mathbb{R}^{m}\times [|y|\geqslant R_{\varepsilon}]} a(x,y)|u_n|^{2} \, {\rm d}z < \varepsilon M, \]

where $M=\sup _{n}||u_n||^{2}_a$.

Notice that (A1) and (2.3) imply that the space $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{m}\times [|y|\geqslant R_{\varepsilon }])$ is continuously immersed in $L^{2^{*}_{\gamma }}(\mathbb {R}^{m}\times [|y|\geqslant R_{\varepsilon }])$. Then the last inequality together with the interpolation inequality ensures that for each $p\in (2, 2^{*}_{\gamma })$,

\[ \int_{\mathbb{R}^{m}\times [|y|\geqslant R_{\varepsilon}]} |u_n|^{p} \, {\rm d}z < \varepsilon^{{(1-\alpha)p}/{2}} \mathcal{C}_p, \]

where $\alpha ={(p-2)2^{*}_{\gamma }}/{(2^{*}_{\gamma }-2)p}$. Once again by (A1) and lemma 2.2, it is easy to conclude that the embedding of $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{m}\times B_{R_{\varepsilon }}(0))$ into $L^{p}(\mathbb {R}^{m}\times B_{R_{\varepsilon }}(0))$ is compact for all $p\in [2,2^{*}_{\gamma })$, and so, $u_n\to 0$ in $L^{p}(\mathbb {R}^{m}\times B_{R_{\varepsilon }}(0))$. This proves the desired result.

3. Proof of theorem 1.1

From now on, since we intend to find nonnegative solutions, without loss of generality we assume that

\[ f(s)=0, \quad \forall s \leqslant 0. \]

The energy functional associated with problem $(P)_1$ is the functional $I:\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})\longrightarrow \mathbb {R}$ given by

\[ I(u)=\frac{1}{2}\int_{\mathbb{R}^{N}}\big(|\nabla_{\gamma}u|^{2} +|u|^{2}\big) \, {\rm d}z - \int_{\mathbb{R}^{N}} F(u)\, {\rm d}z. \]

The reader is invited to see that weak solutions of problem $(P)_1$ are critical points of $I$.

In order to overcome the loss of compactness involving the space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, we will restrict to functional $I$ to the space $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, and in this sense, an important tool in our approach is the Principle of Symmetric Criticality due to Palais (see [38, theorem 1.28]) that will be used to obtain solutions on the whole space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

Our first lemma establishes that $I$ satisfies the mountain pass geometry.

Lemma 3.1 There exist positive numbers $\rho$ and $\alpha$ such that:

1. (i) $I(u)\geqslant \alpha$ if $||u||_{\gamma }=\rho$.

2. (ii) There exists $v\in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that $||v||_{\gamma }>\rho$ and $I(v)< 0$.

Proof. (i) From $(f_1)$ and $(f_2)$, there is a constant $C>0$ such that

\[ |F(t)|\leqslant \frac{1}{4}|t|^{2}+C|t|^{q}, \quad \hbox{for all} t\in\mathbb{R}. \]

Hence,

\[ I(u) = \frac{1}{2}||u||_{\gamma}^{2} -\int_{\mathbb{R}^{N}}F(u)\,{\rm d}z \geqslant \frac{1}{2}||u||_{\gamma}^{2} - \frac{1}{4}\int_{\mathbb{R}^{N}}|u|^{2}\,{\rm d}z-C\int_{\mathbb{R}^{N}}|u|^{q}\,{\rm d}z. \]

Since the embedding $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{q}(\mathbb {R}^{N})$ is continuous,

\[ I(u)\geqslant \frac{1}{2}||u||_{\gamma}^{2}-\frac{1}{4}||u||_{\gamma}^{2}-C||u||_{\gamma}^{q}, \]

and so,

\[ I(u)\geqslant \frac{1}{4}||u||_{\gamma}^{2}-C||u||_{\gamma}^{q}. \]

Now, if $||u||_{\gamma }=\rho$ is small enough, it follows that

\[ I(u)\geqslant \frac{1}{4}\rho^{2}-C\rho^{q}:=\alpha>0. \]

(ii) By $(f_3)$ there is a nonnegative function $\varphi \in C_0^{\infty }(\mathbb {R}^{N}) \cap \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \setminus \{0\}$ satisfying

(3.1)\begin{equation} \displaystyle \int_{\mathbb{R}^{N}}(F(\varphi)-\frac{1}{2}|\varphi|^{2})\,{\rm d}z>0. \end{equation}

By using the function $\varphi$ above, for each $t>0$ we fix the function

\[ \varphi_t(z)=\varphi(z/t), \quad \forall z \in \mathbb{R}^{N}. \]

A direct computation leads to

\begin{align*} I(\varphi_t)& =\frac{t^{N-2}}{2}\int_{\mathbb{R}^{N}}|\nabla_{x}\varphi|^{2}\,{\rm d}z +\frac{t^{N+2\gamma-2}}{2}\int_{\mathbb{R}^{N}}|x|^{2\gamma}|\nabla_{y}\varphi|^{2}\,{\rm d}z\\ & \quad + t^{N}\left(\frac{1}{2}\int_{\mathbb{R}^{N}}|\varphi|^{2} \, {\rm d}z - \int_{\mathbb{R}^{N}} F(\varphi)\, {\rm d}z\right). \end{align*}

Now, the fact that $0\leqslant \gamma <1$ combines with (3.1) to give

\[ I(\varphi_t) \to -\infty \quad \mbox{as} \ t \to +\infty. \]

Therefore, the result follows by fixing $v=\varphi _t$ for $t$ large enough.

The last lemma permits to conclude that the mountain pass level given by

\[ d:=\inf_{g\in\Gamma}\max_{t\in [0,1]}I(g(t))\geqslant\alpha>0, \]

where

\[ \Gamma=\{\xi \in C([0,1],\mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N})):\xi(0)=0 \hbox{and}\ \xi(1)=v \}, \]

is well defined.

Since we are not assuming in this section the Ambrosetti–Rabinowitz condition, we will adapt for our problem the same approach explored in [23], which was also used in [22].

Hereafter, let us consider an auxiliary functional $\tilde {I}:\mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \to \mathbb {R}$ defined by

\[ \tilde{I}(s,u(z))=I(u({\rm e}^{{-}s}x,{\rm e}^{-(\gamma+1)s}y)), \quad z=(x,y) \in \mathbb{R}^{N}=\mathbb{R}^{m} \times \mathbb{R}^{k}, \]

that is,

\[ \tilde{I}(s,u)= \frac{{\rm e}^{(m+(\gamma+1)-2)s}}{2}\|u\|_{\mathcal{D}_{\gamma}^{1,2} \big(\mathbb{R}^{N}\big)}^{2}+\frac{{\rm e}^{(m+(\gamma+1)k)s}}{2}|u|_{2}^{2}- {\rm e}^{(m+(\gamma+1)k)s}\int_{\mathbb{R}^{N}}F(u)\,{\rm d}z. \]

It is easy to check that

\[ \tilde{I}(0,u)=I(u), \quad \forall u \in \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N}). \]

In what follows, we equip the space $\mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ with the standard product norm $\|(s,u)\|_{\mathbb {R} \times \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})}=(|s|^{2}+\|u\|_{\gamma }^{2})^{{1}/{2}}$.

Arguing of the same way as in lemma 3.1, it is easy to see that $\tilde {I}$ also satisfies the mountain pass geometry, more precisely, we have the result below:

Lemma 3.2 There exist positive numbers $\rho$ and $\alpha$ such that:

1. (i) $\tilde {I}(s,u)\geqslant \alpha$ if $\|(s,u)\|_{\mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})}=\rho$.

2. (ii) There exists $v\in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ such that $\|(0,v)\|_{\mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})}=\|v\|>\rho$ and $\tilde {I}(0,v)=I(v)< 0$.

From now on, we denote by $\tilde {d}$, the mountain pass level associated with $\tilde {I}$ given by

\[ \tilde{d}:=\inf_{\tilde{g}\in\Gamma}\max_{t\in [0,1]}\tilde{I}(\tilde{g}(t))\geqslant\alpha>0, \]

where

\[ \Gamma=\{\tilde{g} \in C([0,1],\mathbb{R} \times \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N})\,:\,\tilde{g}(0)=0 \hbox{and} \ \tilde{g}(1)=(0,v) \}. \]

From definition of $I$ and $\tilde {I}$, we derive the equality $d=\tilde {d}$, which permits to repeat the same arguments explored in the proof of [22, propostion 4.2] (see also [23, propostion 2.2]) to show the following proposition:

Proposition 3.3 There is a sequence $\{(s_n,u_n)\} \subset \mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that

1. (a) $s_n \to 0$;

2. (b) $\tilde {I}(s_n,u_n) \to \tilde {d}$;

3. (c) $({\partial }/{\partial u})\tilde {I}(s_n,u_n)\to 0$ strongly in $(\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}))^{*}$;

4. (d) ${\partial }/{\partial s}\tilde {I}(s_n,u_n)\to 0$ in $\mathbb {R}$.

The next lemma establishes the boundedness of the sequence $(u_n) \subset \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ that was obtained in the last proposition.

Lemma 3.4 The sequence $(u_n) \subset \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ is bounded.

Proof. From proposition 3.3,

\[ \left\{ \begin{array}{l} \dfrac{{\rm e}^{(m+(\gamma+1)-2)s_n}}{2}\|u_n\|_{\mathcal{D}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big)}^{2}+ \dfrac{{\rm e}^{(m+(\gamma+1)k)s_n}}{2}|u_n|_{2}^{2}-{\rm e}^{(m+(\gamma+1)k)s_n}\\ \quad\displaystyle \int_{\mathbb{R}^{N}}F(u_n)\,{\rm d}z=\tilde{d}+o_n(1)\\ \mbox{} \\ \dfrac{(m+(\gamma+1)-2)\,{\rm e}^{(m+(\gamma+1)-2)s_n}}{2}\|u_n\|_{\mathcal{D}_{\gamma}^{1,2} \big(\mathbb{R}^{N}\big)}^{2}\\\quad +\dfrac{(m+(\gamma+1)k)\,{\rm e}^{(m+(\gamma+1)k)s_n}}{2} |u_n|_{2}^{2}\\ \mbox{}\\ \quad -(m+(\gamma+1)k)\,{\rm e}^{(m+(\gamma+1)k)s_n}\displaystyle\int_{\mathbb{R}^{N}}F(u_n)\,{\rm d}z=o_n(1) \\ \mbox{}\\ {{\rm e}^{(m+(\gamma+1)-2)s_n}}\|u_n\|_{\mathcal{D}_{\gamma}^{1,2} \big(\mathbb{R}^{N}\big)}^{2}+{{\rm e}^{(m+(\gamma+1)k)s_n}}|u_n|_{2}^{2}- {\rm e}^{(m+(\gamma+1)k)s_n}\\ \quad\displaystyle \int_{\mathbb{R}^{N}}f(u_n)u_n\,{\rm d}z=o_n(1). \end{array} \right. \]

The first and second equation of the previous system ensure that $(\|u_n\|_{\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})})$ is bounded. From $(f_1)-(f_2)$, there is $C>0$ such that

\[ |f(t)t| \leqslant \frac{1}{4}|t|^{2}+C|t|^{2^{*}}, \quad \forall t \in \mathbb{R}. \]

Using this information in the last equation of the above system, we get

\[ |u_n|_{2}^{2} \leqslant C\int_{\mathbb{R}^{N}}|u_n|^{2^{*}}\,{\rm d}z+o_n(1), \quad \forall n \in \mathbb{N}. \]

Now, employing (2.1) we arrive at

\[ |u_n|_{2}^{2} \leqslant C\|u_n\|_{\mathcal{D}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big)}^{2}+o_n(1), \quad \forall n \in \mathbb{N}, \]

from where it follows that $(u_n)$ is a bounded sequence in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$.

3.1 Proof of theorem 1.1

By proposition 3.3 there is $\{(s_n,u_n)\} \subset \mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ satisfying

1. (a) $s_n \to 0;$

2. (b) $\tilde {I}(s_n,u_n) \to \tilde {d};$

3. (c) $({\partial }/{\partial u})\tilde {I}(s_n,u_n)\to 0$ strongly in $(\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}))^{*}$;

4. (d) $({\partial }/{\partial s})\tilde {I}(s_n,u_n)\to 0$ in $\mathbb {R}$.

By lemma 3.4 the sequence $(u_n)$ is bounded in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, then for some subsequence, still denoted by itself, there is $u_0 \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that

\[ u_n \rightharpoonup u_0 \quad \mbox{in} \ \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N}). \]

The item $(d)$ implies that $({\partial }/{\partial u})\tilde {I}(s_n,u_n)w\to 0$ for all $w \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, that is,

\begin{align*} & {\rm e}^{(m+(\gamma+1)-2)s_n}\int_{\mathbb{R}^{N}}\nabla_{\gamma} u_n\nabla_{\gamma}w\,{\rm d}z\\ & \quad+{{\rm e}^{(m+(\gamma+1)k)s_n}}\int_{\mathbb{R}^{N}}u_nw\,{\rm d}z- {\rm e}^{(m+(\gamma+1)k)s_n}\displaystyle \int_{\mathbb{R}^{N}}f(u_n)w\,{\rm d}z=o_n(1). \end{align*}

Since $s_n \to 0$, it is easy to check that the above equality leads to

\[ \int_{\mathbb{R}^{N}}\nabla_{\gamma} u_0\nabla_{\gamma} w\,{\rm d}z+\int_{\mathbb{R}^{N}}u_0w\,{\rm d}z-\displaystyle \int_{\mathbb{R}^{N}}f(u_0)w\,{\rm d}z=0, \quad \forall w \in \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N}), \]

showing that $I'(u_0)w=0$ for all $w \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, that is, $u_0$ is a critical point of $I$. We claim that $u_0 \not = 0$, because if $u_0=0$ the lemma 2.3 gives that

(3.2)\begin{equation} \int_{\mathbb{R}^{N}}|u_n|^{p}\,{\rm d}z \to 0, \quad \forall p \in (2,2^{*}_{\gamma}). \end{equation}

From $(f_1)-(f_2)$, given $\epsilon >0$, there is $C>0$ such that

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n| \,{\rm d}z\leqslant \epsilon \int_{\mathbb{R}^{N}}|u_n|^{2}\,{\rm d}z+C\int_{\mathbb{R}^{N}}|u_n|^{p}\,{\rm d}z. \]

As $(u_n)$ is bounded in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, it follows that

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n| \,{\rm d}z\leqslant \epsilon M+C\int_{\mathbb{R}^{N}}|u_n|^{p}\,{\rm d}z, \]

where $M=\sup _{n \in \mathbb {N}} \int _{\mathbb {R}^{N}}|u_n|^{2}\,{\rm d}z$. Now, using the limit (3.2) we derive that

\[ \limsup_{n \to +\infty} \int_{\mathbb{R}^{N}}|f(u_n)u_n|\,{\rm d}z \leqslant \epsilon M. \]

As $\epsilon$ is arbitrary, the last limit yields

\[ \lim_{n \to +\infty} \int_{\mathbb{R}^{N}}f(u_n)u_n\,{\rm d}z=0. \]

This limit combined together with the limit $({\partial }/{\partial u})\tilde {I}(s_n,u_n)u_n=o_n(1)$ allows to deduce that $u_n \to 0$ in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. Hence,

\[ \tilde{I}(s_n,u_n) \to 0, \]

which is absurd, because $\tilde {I}(s_n,u_n) \to \tilde {d}>0$. Thus $u_0$ is a nontrivial critical point of $I$ in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$.

Next, we will show that $u_0$ is in fact a critical point of $I$ in the space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$. To see why, we will apply the Principle of Symmetric Criticality [35] (see also [38, theorem 1.28]). In what follows, let us denote by $G$ the following subgroup of $O(N)$:

\[ G=\left\{g\in O(N)\,:\,g(x,y)=(h(x),l(y))\ \hbox{with}\ h\in {O}(m) \ \mbox{and} \ l \in {O}(k) \right\} \]

and the action $G\times \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N}) \longrightarrow \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ is given by

\[ (gu)(x,y)=u(h(x),l(y))\quad \hbox{for all} (x,y)\in\mathbb{R}^{m}\times\mathbb{R}^{k}, \]

which is isometric, that is, $\|gu\|_{\gamma }=\|u\|_{\gamma }$. Furthermore, the functional $I$ is invariant under the action of $G$, since for all $g\in G$ and $u \in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$,

\[ I(gu) = \frac{1}{2}||gu||^{2}_{\gamma} -\int_{\mathbb{R}^{N}}F(gu) \,{\rm d}z=\frac{1}{2}||u||^{2}_{\gamma} - \int_{\mathbb{R}^{N}}F(u) \,{\rm d}z = I(u). \]

From the above commentaries, it is easy to check that

\[ Fix(G) = \{ u\in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big)\,:\, gu=u, \forall g\in G\}=\mathcal{H}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big). \]

Hence, by the Principle of Symmetric Criticality of Palais, $u_0$ is a nontrivial critical point of $I$ in $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, and consequently $u_0$ is the required nontrivial solution of $(P)_1$. Finally, we can see that $u_0$ is nonnegative, because it satisfies

\[ \int_{\mathbb{R}^{N}}\big(\nabla_{\gamma} u_0\nabla_{\gamma}\varphi +u_0\varphi \big)\,{\rm d}z =\int_{\mathbb{R}^{N}} f(u_0)\varphi \,{\rm d}z, \quad \forall \varphi \in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big). \]

Then, choosing the test function $\varphi =u_0^{-}:=\min \{u,0\}$, we find that $||u_0^{-}||_{\gamma }=0$. This proves that $u_0$ is nonnegative.

4. Proof of theorem 1.2

In what follows, as in the previous section, without loss of generality we assume that

\[ f(s)=0, \quad \forall s \leqslant 0. \]

The energy functional associated with problem $(P)_0$ is the functional $J:\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})\longrightarrow \mathbb {R}$ given by

\[ J(u)=\frac{1}{2}\int_{\mathbb{R}^{N}}|\nabla_{\gamma}u|^{2} \, {\rm d}z - \int_{\mathbb{R}^{N}} F(u)\, {\rm d}z. \]

The reader is invited to see that weak solutions of problem $(P)_0$ are critical points of $I$.

In order to overcome the loss of compactness involving the space $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$, we will restrict to functional $J$ to the space $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. In this section, $\|\,\,\|$ denotes the norm $\|\,\,\|_{\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})}$.

Our first lemma shows that $J$ verifies the mountain pass geometry. However, we will omit its proof because the arguments are similar those used in the proof of lemma 3.1.

Lemma 4.1 There exist positive numbers $\rho$ and $\alpha$ such that:

1. (i) $J(u)\geqslant \alpha$ if $||u||=\rho$.

2. (ii) There exists $v\in \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that $\|v\|>\rho$ and $J(v)< 0$.

The last lemma permits to conclude that the mountain pass level given by

\[ d_0:=\inf_{g\in\Gamma}\max_{t\in [0,1]}J(g(t))\geqslant\alpha>0, \]

where

\[ \Gamma=\{g\in C([0,1],\mathcal{D}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big):g(0)=0 \ \hbox{and}\ g(1)=v \} \]

is well defined.

Arguing as in § 3, we will consider an auxiliary functional $\tilde {J}:\mathbb {R} \times \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \to \mathbb {R}$ given by

\[ \tilde{J}(s,u(z))=J(u({\rm e}^{{-}s}x,{\rm e}^{-(\gamma+1)s}y)), \quad z=(x,y) \in \mathbb{R}^{N}=\mathbb{R}^{m} \times \mathbb{R}^{k}, \]

that is,

\[ \tilde{J}(s,u)= \frac{{\rm e}^{(m+(\gamma+1)-2)s}}{2}\|u\|^{2}-{\rm e}^{(m+(\gamma+1)k)s}\int_{\mathbb{R}^{N}}F(u)\,{\rm d}z. \]

It is easy to prove that $\tilde {J}$ also satisfies the mountain pass geometry, more precisely we have the result below

Lemma 4.2 There exist positive numbers $\rho$ and $\alpha$ such that:

1. (i) $\tilde {J}(s,u)\geqslant \alpha$ if $\|(s,u)\|_{\mathbb {R} \times \mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})}=\rho$.

2. (ii) There exists $v\in \mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$ such that $\|(0,v)\|_{\mathbb {R} \times \mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})}=\|v\|>\rho$ and $\tilde {J}(0,v)=J(v)< 0$.

From now on, we denote by $\tilde {d}_0$, the mountain pass level associated with $\tilde {J}$ given by

\[ \tilde{d}_0:=\inf_{\tilde{g}\in\Gamma}\max_{t\in [0,1]}\tilde{J}(\tilde{g}(t))\geqslant\alpha>0, \]

where

\[ \Gamma=\{\tilde{g} \in C([0,1],\mathbb{R} \times \mathcal{D}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big):\tilde{g}(0)=0 \ \hbox{and}\ \tilde{g}(1)=(0,v) \}. \]

From definition of $J$ and $\tilde {J}$, we also derive that equality $d_0=\tilde {d}_0$, which permits to repeat the same arguments explored in the proof of [22, propostion 4.2] (see [23, propostion 2.2]) to get the following proposition

Proposition 4.3 There is a sequence $\{(s_n,u_n)\} \subset \mathbb {R} \times \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that

1. (a) $s_n \to 0;$

2. (b) $\tilde {J}(s_n,u_n) \to \tilde {d};$

3. (c) $({\partial }/{\partial u})\tilde {J}(s_n,u_n)\to 0$ strongly in $(\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}))^{*}$;

4. (d) $({\partial }/{\partial s})\tilde {J}(s_n,u_n)\to 0$ in $\mathbb {R}$.

The next lemma establishes the boundedness of the sequence $(u_n) \subset \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ that was obtained in the last proposition. Since the proof follows the same spirit of lemma 3.4 we also omit its proof.

Lemma 4.4 The sequence $(u_n) \subset \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ is bounded.

4.1 Proof of theorem 1.2

By proposition 4.3, there is $\{(s_n,u_n)\} \subset \mathbb {R} \times \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ satisfying

1. (a) $s_n \to 0;$

2. (b) $\tilde {J}(s_n,u_n) \to \tilde {d};$

3. (c) $({\partial }/{\partial u})\tilde {J}(s_n,u_n)\to 0$ strongly in $(\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}))^{*}$;

4. (d) $({\partial }/{\partial s})\tilde {J}(s_n,u_n)\to 0$ in $\mathbb {R}$.

By lemma 3.4 the sequence $(u_n)$ is bounded in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, then for some subsequence, still denoted by itself, there is $u_0 \in \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that

\[ u_n \rightharpoonup u_0 \quad \mbox{in} \ \mathcal{D}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big). \]

The item (d) implies that $({\partial }/{\partial u})\tilde {J}(s_n,u_n)w\to 0$ for all $w \in \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, that is,

\[ {\rm e}^{(m+(\gamma+1)-2)s_n}\int_{\mathbb{R}^{N}}\nabla_{\gamma} u_n\nabla_{\gamma} w\,{\rm d}z-{\rm e}^{(m+(\gamma+1)k)s_n}\displaystyle \int_{\mathbb{R}^{N}}f(u_n)w\,{\rm d}z=o_n(1). \]

Since $s_n \to 0$, a simple computation guarantees that $u_0$ satisfies the equality below

\[ \int_{\mathbb{R}^{N}}\nabla_{\gamma} u_0\nabla_{\gamma} w\,{\rm d}z-\displaystyle \int_{\mathbb{R}^{N}}f(u_0)w\,{\rm d}z=0, \]

showing that $J'(u_0)w=0$ for all $w \in \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, that is, $u_0$ is a critical point of $J$ in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. We claim that $u_0 \not = 0$, because if $u_0=0$, lemma 2.5 gives that

(4.1)\begin{equation} \int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z \to 0, \end{equation}

for all $0< a< b<+\infty$ and $p \in (2,2^{*}_{\gamma })$.

From $(f_4)-(f_5)$, given $\epsilon >0$ and $p \in (2,2^{*}_{\gamma })$, there are $C>0$ and $a< b<+\infty$ such that

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n| \,{\rm d}z\leqslant \epsilon \int_{\mathbb{R}^{N}}|u_n|^{2^{*}_{\gamma}}\,{\rm d}z+C\int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z. \]

As $(u_n)$ is bounded in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, it follows that

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n| \,{\rm d}z\leqslant \epsilon M+C\int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z, \]

where $M=\sup _{n \in \mathbb {N}}\int _{\mathbb {R}^{N}}|u_n|^{2^{*}_{\gamma }}\,{\rm d}z$. Now, using the limit (4.1) we see that

\[ \limsup_{n \to +\infty} \int_{\mathbb{R}^{N}}|f(u_n)u_n|\,{\rm d}z \leqslant \epsilon M. \]

Thereby, as $\epsilon$ is arbitrary, we can infer that

\[ \lim_{n \to +\infty} \int_{\mathbb{R}^{N}}f(u_n)u_n\,{\rm d}z=0. \]

This limit combined with the limit $({\partial }/{\partial u})\tilde {J}(s_n,u_n)u_n=o_n(1)$ ensures that $u_n \to 0$ in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. Hence,

\[ \tilde{J}(s_n,u_n) \to 0, \]

which is absurd, because $\tilde {J}(s_n,u_n) \to \tilde {d}_0>0$. Therefore, $u_0$ is a nontrivial critical point of $J$ in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. As in § 3, we can again use the Principle of Symmetric Criticality to prove that $u_0$ is a critical point of $J$ in $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

5. Proof of theorem 1.3

In this section, the energy functional associated with $(P)_a$ is given by

\[ I(u)=\frac{1}{2}\int_{\mathbb{R}^{N}}\big(|\nabla_{\gamma}u|^{2} +a(z)|u|^{2}\big) \, {\rm d}z - \int_{\mathbb{R}^{N}} F(u)\, {\rm d}z, \quad \forall u \in \mathcal{H}_{\gamma}^{a}(\mathbb{R}^{N}). \]

Unfortunately the space $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ does not have good compactness embedding to help us to find a nontrivial critical point $u$ for $I$. Following the same approach explored in the previous sections, we will restrict $I$ to the space $\mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$. Since $I$ satisfies the mountain pass geometry in $\mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$, there is $(u_n) \subset \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ such that

(5.1)\begin{equation} I(u_n)\longrightarrow d_a \quad \hbox{and}\quad I'(u_n)\longrightarrow 0, \end{equation}

where

\[ d_a:=\inf_{g\in\Gamma}\max_{t\in [0,1]}I(g(t))\geqslant\alpha>0, \]

where

\[ \Gamma_a=\{g\in C([0,1], \mathcal{H}^{a}_{\gamma,rad_{x}}(\mathbb{R}^{N})\,:\,g(0)=0 \ \hbox{and} g(1)=v \}. \]

Using well-known arguments, the condition $(f_7)$ helps us to prove that $(u_n)$ is bounded in $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$. Since $\mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ is reflexive, we can assume that there exists $u_0\in \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ such that, up to subsequence,

1. • $u_n \rightharpoonup u_0$ in $\mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N}),$

2. • $u_n \rightarrow u_0$ in $L_{loc}^{p}(\mathbb {R}^{N})\ \text {for all}\ p\in (2,2^{*}_{\gamma })$,

3. • $u_n \rightarrow u_0\ a. e.\ \textit {in}\ \mathbb {R}^{N}$.

As $I'(u_n)v = o_n(1)$ for all $v \in \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$, the above convergences yield

\[ I'(u_0)v = 0\quad \hbox{for all}\ v \in\mathcal{H}^{a}_{\gamma,rad_{x}}(\mathbb{R}^{N}), \]

from where it follows that $u_0$ is a critical point of $I$.

Lemma 5.1 Let $(u_n)\subset \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ be a sequence such that

\[ I(u_n)\longrightarrow d_a>0\quad \hbox{and}\quad I'(u_n)\longrightarrow 0. \]

Then, there exist $(y_n)\subset \mathbb {R}^{k}$ and $r, \beta >0$ such that

\[ \int_{\mathbb{R}^{m}\times B_r(y_n)}|u_n|^{q}\,{\rm d}z\geqslant\beta, \quad \forall n\in\mathbb{N} . \]

Proof. In what follows, we will argue by contradiction. If the lemma is not true, we can employ lemma 2.1 to get

\[ u_n\to 0\ \hbox{in}\ L^{q}(\mathbb{R}^{N})\quad \hbox{for}\ 2< q<2^{*}_{\gamma}. \]

From $(f_1)$ and $(f_2)$, for each $\varepsilon >0$, there is a constant $C_{\varepsilon }>0$ such that

\[ |f(u_n)u_n|\leqslant \varepsilon |u_n|^{2}+C_{\varepsilon}|u_n|^{q},\quad \hbox{for all}\ n\in\mathbb{N}. \]

Hence,

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n|\, {\rm d}z\leqslant\varepsilon C+C_{\varepsilon}\int_{\mathbb{R}^{N}}|u_n|^{q}\, {\rm d}z, \]

and so,

\[ \limsup_{n}\int_{\mathbb{R}^{N}}|f(u_n)u_n|\, {\rm d}z\leqslant\varepsilon C, \]

leading to

\[ \lim_{n}\int_{\mathbb{R}^{N}}|f(u_n)u_n|\, {\rm d}z=0. \]

As $I'(u_n)u_n = o_n(1)$, we have

\[ ||u_n||_{\gamma}^{2}=I'(u_n)u_n+\int_{\mathbb{R}^{N}}|f(u_n)u_n|\, {\rm d}z=o_n(1), \]

that is,

\[ u_n\to 0\quad \hbox{in}\ \mathcal{H}^{a}_{\gamma,rad_{x}}(\mathbb{R}^{N}). \]

Therefore

\[ I(u_n)\to I(0)=0, \]

which is a contradiction, because $I(u_n)\longrightarrow d_a>0$.

5.1 Proof of theorem 1.3

In what follows, we will divide the proof into two cases:

Case 1: (A3) holds.

In this case, we can assume that the weak limit $u_0 \in \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ of the sequence given in (5.1) is nontrivial. Otherwise, by lemma 5.1, there exist $(y_n)\subset \mathbb {R}^{k}$ and $r, \beta >0$ such that

\[ \int_{\mathbb{R}^{m}\times B_r(y_n)}|u_n|^{2} \, {\rm d}z \geqslant\beta, \quad \forall n\in\mathbb{N}. \]

Increasing $r$ if necessary, we can suppose that $y_n \in \mathbb {Z}^{k}$. Now, setting the sequence

\[ v_n(z)=v_n(x,y)=u_n(x,y+y_n), \quad \forall z=(x,y)\in\mathbb{R}^{N}, \]

it follows that $(v_n)\subset \mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$. As we are supposing that (A3) occurs, a simple computation shows that

\[ ||v_n||_{a}=||u_n||_{a}. \]

Hence $(v_n)$ is bounded, and for a subsequence of $(v_n)$, still denoted by itself, there is $v\in \mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$ such that $v_n \rightharpoonup v$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$. By lemma 2.2 $v_n \rightarrow v$ in $L^{p}(\mathbb {R}^{m}\times B_r(0))$ for all $p\in (2,2^{*}_{\gamma })$ and $v_n(z) \rightarrow v(z)$ almost everywhere in $\mathbb {R}^{N}$. Furthermore,

\[ I(v_n)=I(u_n)\quad \hbox{and}\quad I'(v_n)\longrightarrow 0, \]

from where it follows that $I'(v)w=0$ for all $w\in \mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$, showing that $v$ is a critical point for $I$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$. On the other hand, the equality

\[ \int_{\mathbb{R}^{m}\times B_r(0)}|v_n|^{q} \, {\rm d}z=\int_{\mathbb{R}^{m}\times B_r(y_n)}|u_n|^{q} \, {\rm d}z\geqslant\beta, \quad \forall n\in\mathbb{N}, \]

together with the limit $v_n \rightarrow v$ in $L^{p}(\mathbb {R}^{m}\times B_r(0))$ (see lemma 2.2) gives

\[ \int_{\mathbb{R}^{m}\times B_r(0)}|v|^{q} \, {\rm d}z\geqslant\beta, \quad \forall n\in\mathbb{N}, \]

showing that $v\neq 0$.

Case 1: (A4) holds.

If (A4) holds, lemma 2.6 implies that

\[ u_n \rightarrow u_0\quad in\ L^{p}(\mathbb{R}^{N})\quad \hbox{for all}\ p\in[2,2^{*}_{\gamma}). \]

This limit combined with lemma 2.6 ensures that

\[ \int_{\mathbb{R}^{N}}f(u_n)(u_n-u_0) \,{\rm d}z\longrightarrow 0. \]

Now, since $I'(u_n)\to 0$ and $u_n \rightharpoonup u_0$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$, we find that

\begin{align*} ||u_n-u_0||_{a}^{2} & = I'(u_n)u_n + \int_{\mathbb{R}^{N}}f(u_n)u_n \,{\rm d}z - I'(u_n)u_n \\ & \quad- \int_{\mathbb{R}^{N}}f(u_n)u_0 \,{\rm d}z + o_n(1)=o_n(1), \end{align*}

which shows that $u_n \to u_0$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$. From this,

\[ I(u_0)=d_a>0, \]

and so, $u_0$ is a nontrivial critical point for $I$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$.

Finally, in order to conclude the proof, we must show that $u$ is a critical point for $I$ in $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$. As in§ 3 and 4, it is enough to apply the Principle of Symmetric Criticality. However, since the argument are the same we omit its proof.

6. Proof of theorem 1.4

Suppose that $u\in \mathcal {H}^{a}(\mathbb {R}^{N})$ is a solution of $(P)_a$. For any $R>0$, $0< r\leqslant R/2$, let $\eta \in C^{\infty }(\mathbb {R}^{N})$, $0\leqslant \eta \leqslant 1$ with

\[ \eta(z)=\left \{\begin{array}{@{}ll} 1 & \mbox{if} \ |z|\geqslant R, \\ 0 & \mbox{if} \ |z|\leqslant R-r, \end{array}\quad \hbox{and}\quad |\nabla\eta|\leqslant\frac{2}{r\sqrt{R_{\gamma}}} \right. \]

where $R_{\gamma }=\max \{1, R^{2\gamma }\}$. For each $L>0$, let us define

\[ u_{L}=\left \{\begin{array}{@{}ll} u & \mbox{if}\ u\leqslant L, \\ L & \mbox{if}\ u\geqslant L, \end{array} \right. \]

$v_L=\eta ^{2}uu_{L}^{2(\beta -1)}$ and $w_L=\eta uu_{L}^{\beta -1}$ with $\beta >1$ to be determined later on. Then

\[ \nabla_{\gamma}u\nabla_{\gamma}v_{L}=\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}+2(\beta-1)\eta^{2}uu_{L}^{2\beta-3}\nabla_{\gamma}u\nabla_{\gamma}u_{L}+2\eta uu_{L}^{2(\beta-1)}\nabla_{\gamma} u\nabla_{\gamma}\eta \]

and choosing $v_L$ as test function, we obtain

\begin{align*} & \int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z+2(\beta-1)\int_{\mathbb{R}^{N}}\eta^{2}uu_{L}^{2\beta-3}\nabla_{\gamma}u\nabla_{\gamma}u_{L}\, {\rm d}z \\ & \quad +\int_{\mathbb{R}^{N}}a(z)\eta^{2}u^{2}u_{L}^{2(\beta-1)} \, {\rm d}z\!=\! \int_{\mathbb{R}^{N}} f(u)\eta^{2}uu_{L}^{2(\beta\!-\!1)} \, {\rm d}z\!-\!2\int_{\mathbb{R}^{N}}\eta uu_{L}^{2(\beta-1)}\nabla_{\gamma} u\nabla_{\gamma}\eta \, {\rm d}z. \end{align*}

As

\[ \int_{\mathbb{R}^{N}}\eta^{2}uu_{L}^{2\beta-3}\nabla_{\gamma}u\nabla_{\gamma}u_{L}\,{\rm d}z=\int_{[u\leqslant L]}\eta^{2}u^{2(\beta-1)}|\nabla_{\gamma}u|^{2} \,{\rm d}z\geqslant 0 \]

and $a(z)\geqslant a_0>0$ in $\mathbb {R}^{N}$ we derive that

\begin{align*} & \int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z+a_0\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)} \, {\rm d}z\\ & \quad \leqslant \int_{\mathbb{R}^{N}} f(u)\eta^{2} uu_{L}^{2(\beta-1)} \,{\rm d}z-2\int_{\mathbb{R}^{N}}\eta uu_{L}^{2(\beta-1)}\nabla_{\gamma} u\nabla_{\gamma}\eta \, {\rm d}z. \end{align*}

By conditions $(f_1)$ and $(f_2)$, $f(u)\leqslant ({a_0}/{2})u+Cu^{q-1}$, and so,

\begin{align*} & \int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z +\frac{a_0}{2}\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)} \, {\rm d}z\\ & \quad \leqslant C\int_{\mathbb{R}^{N}} \eta^{2}u^{q}u_{L}^{2(\beta-1)} \,{\rm d}z +2\int_{\mathbb{R}^{N}}\eta uu_{L}^{2(\beta-1)}|\nabla_{\gamma} u||\nabla_{\gamma}\eta| \, {\rm d}z. \end{align*}

Using Young's inequality with $\varepsilon >0$ sufficiently small, we get

(6.1)\begin{align} \int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z +\frac{a_0}{2}\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)} \, {\rm d}z & \leqslant C\int_{\mathbb{R}^{N}} \eta^{2}u^{q}u_{L}^{2(\beta-1)} \,{\rm d}z \nonumber\\ & \quad +C_{\varepsilon}\int_{\mathbb{R}^{N}}u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2} \, {\rm d}z. \end{align}

From (2.2),

\[ |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\left(\int_{\mathbb{R}^{N}}|\nabla_{\gamma}w_L|^{2}\,{\rm d}z+\int_{\mathbb{R}^{N}}|w_L|^{2}\,{\rm d}z\right), \]

which leads to

\begin{align*} |w_L|_{2^{*}_{\gamma}}^{2}& \leqslant\mathcal{C}\int_{\mathbb{R}^{N}}\left(\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}+ \eta^{2}(\beta-1)^{2}u^{2}u_{L}^{2(\beta-2)}|\nabla_{\gamma}u_{L}|^{2}\right.\\ & \left.\quad +u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2}\right) {\rm d}z \\ & \quad +\,\mathcal{C}\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)}\, {\rm d}z, \end{align*}

and thus

\begin{align*} |w_L|_{2^{*}_{\gamma}}^{2}& \leqslant\mathcal{C}\beta^{2}\left(\int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z +\frac{a_0}{2}\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)}\,{\rm d}z\right.\nonumber\\ & \left.\quad +\, \int_{\mathbb{R}^{N}}u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2} \, {\rm d}z\right). \end{align*}

Then from (6.1)

(6.2)\begin{equation} |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\beta^{2}\left(\int_{\mathbb{R}^{N}} (\eta uu_{L}^{\beta-1})^{2}u^{q-2} \,{\rm d}z +\int_{\mathbb{R}^{N}}u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2} \,{\rm d}z\right). \end{equation}

Using Hölder inequality with exponents ${{2^{*}_{\gamma }}}/({q-2})$ and ${\alpha ^{*}}/{2}:={{2^{*}_{\gamma }}}/{2^{*}_{\gamma }-(q-2)}$,

\begin{align*} |w_L|_{2^{*}_{\gamma}}^{2}& \leqslant\mathcal{C}\beta^{2}\left(\int_{\mathbb{R}^{N}} (\eta uu_{L}^{\beta-1})^{\alpha^{*}} \,{\rm d}z\right)^{2/\alpha^{*}}\left(\int_{\mathbb{R}^{N}} u^{2^{*}_{\gamma}} \, {\rm d}z \right)^{q-2/2^{*}_{\gamma}}\\ & \quad +\mathcal{C}\beta^{2}\int_{\mathbb{R}^{N}}u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2} \,{\rm d}z. \end{align*}

Since $u\in L^{2^{*}_{\gamma }}(\mathbb {R}^{N})$, we choose $\beta ={{2^{*}_{\gamma }}}/{\alpha ^{*}}>1$ so that $u\in L^{\alpha ^{*}\beta }(\mathbb {R}^{N})\cap L^{2\beta }(\mathbb {R}^{N})$, and so,

\[ |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\beta^{2}\left(\int_{\mathbb{R}^{N}} u^{\alpha^{*}\beta} \,{\rm d}z\right)^{2/\alpha^{*}} +\frac{\mathcal{C}\beta^{2}}{r^{2}}\int_{\mathbb{R}^{N}}u^{2\beta}\,{\rm d}z < \infty. \]

Hence

\[ \left(\int_{[|z|\geqslant R]} (uu_{L}^{\beta-1})^{2^{*}_{\gamma}} \,{\rm d}z\right)^{2/2_{\gamma}^{*}}\leqslant K <\infty. \]

By employing the Fatou's lemma in the variable $L$ we conclude that

(6.3)\begin{equation} \int_{[|z|\geqslant R]} u^{{(2^{*}_{\gamma})^{2}}/{\alpha^{*}}} \,{\rm d}z <\infty. \end{equation}

Once again from inequality (6.2),

\[ |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\beta^{2}\left(\int_{[|z|\geqslant R-r]} u^{q-2}u^{2\beta} \, {\rm d}z + \int_{[R\geqslant |z|\geqslant R-r]}u^{2\beta} \, {\rm d}z\right). \]

We now consider $t={(2^{*}_{\gamma })^{2}}/{\alpha ^{*}(q-2)}>1$ and $\beta ={2^{*}_{\gamma }(t-1)}/{2t}>1$. Then by Hölder inequality with exponents $t/(t-1)$ and $t$,

\begin{align*} |w_L|_{2^{*}_{\gamma}}^{2}& \leqslant \mathcal{C}\beta^{2}\left[\left(\int_{[|z|\geqslant R-r]} u^{(q-2)t} \, {\rm d}z\right)^{1/t}\left(\int_{[|z|\geqslant R-r]} u^{2\beta t/(t-1)} \, {\rm d}z\right)^{(t-1)/t} \right.\\ & \quad \left.+\left(\int_{[R\geqslant |z|\geqslant R-r]}1 \, {\rm d}z\right)^{1/t} \left(\int_{[R\geqslant |z|\geqslant R-r]}u^{2\beta t/(t-1)} \, {\rm d}z\right)^{(t-1)/t}\right]. \end{align*}

It follows from (6.3) that

(6.4)\begin{equation} |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\beta^{2}\left(\int_{[|z|\geqslant R-r]} u^{2\beta t/(t-1)} \, {\rm d}z\right)^{(t-1)/t}. \end{equation}

On the other hand,

\[ \left(\int_{[|z|\geqslant R]} u_{L}^{2^{*}_{\gamma}\beta} \, {\rm d}z\right)^{2/2^{*}_{\gamma}}\leqslant\left(\int_{\mathbb{R}^{N}} (\eta uu_{L}^{\beta-1})^{2_{\gamma}^{*}} \,{\rm d}z\right)^{2/2_{\gamma}^{*}}=|w_{L}|_{2^{*}_{\gamma}}^{2}. \]

This combined with (6.4) leads to

\[ |u_{L}|^{2\beta}_{L^{2^{*}_{\gamma}\beta}([|z|\geqslant R])}\leqslant\mathcal{C}\beta^{2}\left(\int_{[|z|\geqslant R-r]} u^{2\beta t/(t-1)} \, {\rm d}z\right)^{(t-1)/t}=\mathcal{C}\beta^{2}|u|^{2\beta}_{L^{2^{*}_{\gamma}}([|z|\geqslant R-r])}. \]

Hence, by Fatou's lemma

\[ |u|^{2\beta}_{L^{2^{*}_{\gamma}\beta}([|z|\geqslant R])}\leqslant\mathcal{C}\beta^{2}|u|^{2\beta}_{L^{2^{*}_{\gamma}}([|z|\geqslant R-r])}. \]

Setting $\chi :={2^{*}_{\gamma }(t-1)}/{2t}$ and $s:={2t}/({t-1})$, the last inequality allows to conclude that

\[ |u|_{L^{\chi^{m+1}s}([|z|\geqslant R])}\leqslant\mathcal{C}^{\sum_{i=1}^{m}\chi^{{-}i}}\chi^{\sum_{i=1}^{m}i\chi^{{-}i}}\leqslant|u|_{L^{2^{*}_{\gamma}}([|z|\geqslant R-r])}. \]

Letting $m\to \infty$ we obtain

\[ |u|_{L^{\infty}([|z|\geqslant R])}\leqslant|u|_{L^{2^{*}_{\gamma}}([|z|\geqslant R-r])}, \]

showing that $\lim _{|z|\to \infty }u(z)=0$.

Finally, in order to show that $u\in L^{\infty }(\mathbb {R}^{N})$, we need only show that the above inequality is true on balls of $\mathbb {R}^{N}$, that is, there exist $R>0$ and $\mathcal {C}>0$ such that

\[ |u|_{L^{\infty}(B_{R}(z))}\leqslant|u|_{L^{2^{*}_{\gamma}}(B_{2R}(z))}, \quad \forall \ z\in\mathbb{R}^{N}. \]

The argument follows with some adjusts from the ideas found in [4, lemma 2.13].