Asymptotic distribution for pairs of linear and quadratic forms at integral vectors

Abstract

We study the joint distribution of values of a pair consisting of a quadratic form ${\mathbf q}$ and a linear form ${\mathbf l}$ over the set of integral vectors, a problem initiated by Dani and Margulis [Orbit closures of generic unipotent flows on homogeneous spaces of $\mathrm{SL}_3(\mathbb{R})$. Math. Ann. 286 (1990), 101–128]. In the spirit of the celebrated theorem of Eskin, Margulis and Mozes on the quantitative version of the Oppenheim conjecture, we show that if $n \ge 5$, then under the assumptions that for every $(\alpha , \beta ) \in {\mathbb {R}}^2 \setminus \{ (0,0) \}$, the form $\alpha {\mathbf q} + \beta {\mathbf l}^2$ is irrational and that the signature of the restriction of ${\mathbf q}$ to the kernel of ${\mathbf l}$ is $(p, n-1-p)$, where ${3\le p\le n-2}$, the number of vectors $v \in {\mathbb {Z}}^n$ for which $\|v\| < T$, $a < {\mathbf q}(v) < b$ and $c< {\mathbf l}(v) < d$ is asymptotically $ C({\mathbf q}, {\mathbf l})(d-c)(b-a)T^{n-3}$ as $T \to \infty $, where $C({\mathbf q}, {\mathbf l})$ only depends on ${\mathbf q}$ and ${\mathbf l}$. The density of the set of joint values of $({\mathbf q}, {\mathbf l})$ under the same assumptions is shown by Gorodnik [Oppenheim conjecture for pairs consisting of a linear form and a quadratic form. Trans. Amer. Math. Soc. 356(11) (2004), 4447–4463].

1 Introduction

The Oppenheim conjecture [16], settled by Gregory Margulis in 1986 [15], states that for any non-degenerate irrational indefinite quadratic form ${\mathbf q}$ over ${\mathbb {R}}^n$ , $n \ge 3$ , the set ${\mathbf q}({\mathbb {Z}}^n)$ of values of ${\mathbf q}$ over integral vectors is a dense subset of ${\mathbb {R}}$ .

Margulis’ proof uses the dynamics of Lie group actions on homogeneous spaces. More precisely, he shows that every pre-compact orbit of the orthogonal group ${\mathrm {SO}}(2,1)$ on the homogeneous space $\mathrm {SL}_3({\mathbb {R}})/\mathrm {SL}_3({\mathbb {Z}})$ is compact. This proof also settled a special case of Raghunathan’s conjecture on the action of unipotent groups on homogenous spaces. Raghunathan’s conjecture was posed in the late seventies (appearing in print in [8]) suggesting a different route towards resolving the Oppenheim conjecture. This conjecture was later settled in its full generality by Marina Ratner [18].

Ever since Margulis’ proof, homogenous dynamics has turned into a powerful machinery for studying similar questions of number theoretic nature. In particular, various extensions and refinements of the Oppenheim conjectures have been studied. In the quantitative direction, one can inquire about the distribution of values of ${\mathbf q}({\mathbb {Z}}^n \cap B(T))$ , where $B(T)$ denotes the ball of radius T centred at zero. It was shown in a groundbreaking work by Eskin, Margulis and Mozes [9] that the number $\mathcal {N}_{T, I}({\mathbf q})$ of vectors $v \in B(T)$ with ${\mathbf q}(v) \in I:= (a,b)$ satisfies the asymptotic formula

(1.1) $$ \begin{align} \mathcal{N}_{T, I}({\mathbf q}) \sim C({\mathbf q}) (b-a) T^{n-2}\quad \text{as}\;T\rightarrow \infty, \end{align} $$

assuming that ${\mathbf q}$ is non-degenerate, indefinite and irrational, and has signature different from $(2,1)$ and $(2,2)$ . Prior to [9], an asymptotically exact lower bound was established by Dani and Margulis [7] under the condition $n \ge 3$ .

It is noteworthy that equation (1.1) does not hold for all irrational quadratic forms of signatures $(2,1)$ and $(2,2)$ . However, for quadratic forms of signature $(2,2)$ that are not well approximable by rational forms, an analogous quantitative result for a modified counting function has been established in [10]. The question for forms of signature $(2, 1)$ remains open.

Let ${\mathbf q}$ be an indefinite quadratic form of signature $(p,q)$ . The approach taken up in [9] translates the problem of determining the asymptotic distribution of ${\mathbf q}({\mathbb {Z}}^n)$ to the question of studying the distribution of translated orbits $a_tKx_0$ in the space $\mathrm {SL}_n({\mathbb {R}})/\mathrm {SL}_n({\mathbb {Z}})$ of unimodular lattices in ${\mathbb {R}}^n$ . Here, $a_t$ is a one-parameter diagonal subgroup of the orthogonal group ${\mathrm {SO}}(p, q)$ defined in equation (2.5), K is isomorphic to the maximal compact subgroup of the connected component of identity in ${\mathrm {SO}}(p, q)$ and $x_0 \in \mathrm {SL}_n({\mathbb {R}})/\mathrm {SL}_n({\mathbb {Z}})$ is determined by the quadratic form ${\mathbf q}$ . One of the major challenges of the proof is that the required equidistribution result involves integrals of unbounded observables (or test functions). This difficulty is overcome by introducing a set of height functions, which can be used to track the elements $k \in K$ for which the lattice $a_tkx_0$ has a large height, and thereby reducing the problem to bounded observables.

1.1 Pairs of quadratic and linear forms

In this paper, we study the joint distribution of the values of pairs $({\mathbf q}, {\mathbf l})$ consisting of a quadratic and a linear form. This problem was first studied by Dani and Margulis [6] who proved a result for the density of the joint values of pairs of a quadratic form and a linear form in three variables. This result was extended by Gorodnik [11] to forms with $n \ge 4$ variables. Our goal in this paper is to prove a quantitative version of these qualitative results.

Fix $n \ge 4$ , and write ${\mathbf q}$ for a non-degenerate indefinite quadratic form on ${\mathbb {R}}^n$ and ${\mathbf l} $ for a non-zero linear form on ${\mathbb {R}}^n$ . Denote by $\mathscr {S}^0_n$ the set of all such pairs $({\mathbf q}, {\mathbf l})$ satisfying the following two conditions.

1. (A) The restriction of ${\mathbf q}$ to the subspace defined by ${{\mathbf l}=0}$ is indefinite.

2. (B) For every $(\alpha , \beta ) \in {\mathbb {R}}^2 \setminus \{ (0,0) \}$ , the form $\alpha {\mathbf q} + \beta {\mathbf l}^2$ is irrational.

The main result of [11, Theorem 1] shows that under these assumptions, the set of joint values

$$ \begin{align*}\{ ({\mathbf q}(v), {\mathbf l}(v)): v \in {\mathbb{Z}}^n \} \subseteq {\mathbb{R}}^2\end{align*} $$

is dense. Note that condition (A) is necessary for the set of values to be dense in ${\mathbb {R}}^2$ . Condition (B), however, can conceivably be weakened, see a remark in [11, §6].

Our goal in this work is to study a quantitative refinement of this problem. More precisely, we will ask the following question.

Question 1.1. For $({\mathbf q}, {\mathbf l}) \in \mathscr {S}^0_n$ and intervals $I= (a, b), J=(c,d)$ , denote by $\mathcal {N}_{T,I, J}({\mathbf q}, {\mathbf l})$ the number of vectors $v \in {\mathbb {Z}}^n$ for which $\|v\| < T$ , $ {\mathbf q}(v) \in I $ and $ {\mathbf l}(v) \in J$ . Find conditions under which the following asymptotic behaviour holds:

$$ \begin{align*} \mathcal{N}_{T,I, J}({\mathbf q}, {\mathbf l}) \sim C({\mathbf q}, {\mathbf l}) \ (b-a)(d-c) T^{n-3} \end{align*} $$

as $T \to \infty $ . Here, $C({\mathbf q}, {\mathbf l})$ is a positive constant that depends only on ${\mathbf q}$ and ${\mathbf l}$ .

Note that the above asymptotic behaviour is consistent with the general philosophy in [9]. The ball $B(T)$ of radius T centred at zero contains about $T^n$ integral vectors. As v ranges in $B(T)$ , ${\mathbf q}(v)$ takes values in an interval of length approximately $T^2$ , while the values of $ {\mathbf l}(v)$ range in an interval of length comparable to T. Packing the $T^n$ points $({\mathbf q}(v), {\mathbf l}(v))$ in a box of volume comparable to $T^3$ , one might expect that a rectangle of fixed size is hit approximately $T^{n-3}$ times.

1.2 Statement of results

Let $| I |$ denote the length of the interval $I \subseteq {\mathbb {R}}$ . Our main result is the following.

Theorem 1.2. Let ${\mathbf q}$ be a non-degenerate indefinite quadratic form on ${\mathbb {R}}^n$ for $n \ge 5$ and let ${\mathbf l}$ be a non-zero linear form on ${\mathbb {R}}^n$ . For $T>0$ , open bounded intervals $I, J \subseteq {\mathbb {R}}$ , let $\mathcal {N}_{T, I, J}({\mathbf q}, {\mathbf l})$ denote the number of vectors v for which

$$ \begin{align*}\|v\| < T, \quad {\mathbf q}(v) \in I , \quad {\mathbf l}(v) \in J.\end{align*} $$

Let $\mathscr {S}_n$ be the set of $({\mathbf q}, {\mathbf l})\in \mathscr {S}^0_n$ for which the restriction of ${\mathbf q}$ to $\ker {\mathbf l}$ is non-degenerate in $\ker {\mathbf l}$ and not of signature $(2,2)$ . Then for any $({\mathbf q}, {\mathbf l})\in \mathscr {S}_n$ , we have

$$ \begin{align*} \lim_{T \to \infty} \frac{\mathcal{N}_{T, I, J}({\mathbf q}, {\mathbf l})}{T^{n-3}} = C({\mathbf q}, {\mathbf l}) |I| \ |J|,\end{align*} $$

where $C({\mathbf q}, {\mathbf l})$ is a positive constant depending only on ${\mathbf q}$ and ${\mathbf l}$ .

Remark 1.3. Theorem 1.2 does not generally hold if the restriction of ${\mathbf q}$ to $\ker {\mathbf l}$ is of signature $(2,2)$ , see §6 for a counterexample. Based on the main result of [10], it seems reasonable that a modified result, under certain diophantine condition, might still hold.

1.3 Strategy of proof

The proof follows the same roadmap as in [9]. We will start by translating the question into one about the distribution of translated orbits on homogenous spaces.

It is well known that the space $\mathscr {X}_n$ of unimodular lattices in ${\mathbb {R}}^n$ can be identified with the homogenous space $\mathrm {SL}_n({\mathbb {R}})/\mathrm {SL}_n({\mathbb {Z}})$ . This space is non-compact and carries an $\mathrm {SL}_n({\mathbb {R}})$ -invariant probability measure. In many problems in homogenous dynamics, it is useful to quantify the extent to which a lattice lies in the cusp of $\mathscr {X}_n$ .

We will translate Question 1.1 to the problem of showing that certain translated orbits of the form $a_tK \Lambda $ become asymptotically equidistributed in $\mathscr {X}_n$ as t goes to $\infty $ . Here, K is the maximal compact subgroup of the connected component of identity in ${\mathrm {SO}}(p,q-1)$ , where $(p,q-1)$ denotes the signature of the restriction of ${\mathbf q}$ to $\ker {\mathbf l}$ . At this point, several problems will arise. However, the existence of various intermediate subgroups make the application of the Dani–Margulis theorem more difficult. Dealing with this problem requires us to classify all intermediate subgroups that can arise. The second problem, similar to that in [9], involves the unboundedness of test functions to which the equidistribution result must be applied. We will adapt the technique used in [9] with one twist. Namely, we will prove a boundedness theorem for the integrals of $ \alpha ( a_tk \Lambda )^s$ for some $s>1$ , where $ \alpha $ is the Margulis height function defined as follows: for a lattice $\Lambda $ ,

$$ \begin{align*} \alpha ( \Lambda)= \max \{ { \| v \|}^{-1}: v \in \Omega( \Lambda) \},\end{align*} $$

where

$$ \begin{align*}\Omega(\Lambda)=\{v=v_1\wedge \cdots \wedge v_i : v_1, \ldots, v_i \in \Lambda, \quad 1 \le i \le n \} \setminus \{ 0 \}.\end{align*} $$

More precisely, we will show that for $p\ge 3$ , $q\ge 2$ and $0<s<2$ , for every $g \in \mathrm {SL}_n({\mathbb {R}})$ , we have

$$ \begin{align*} \sup_{t>0} \int_K \alpha(a_tk.g{\mathbb{Z}}^n)^s \,dm(k) < \infty. \end{align*} $$

The strategy in [9] requires K not to have non-trivial fixed vectors in certain representation spaces. Since this is no longer the case here, we need to use a refined version of the $\alpha $ function developed by Benoist and Quint [4, 19] which we recall now.

Let H be a connected semisimple Lie subgroup of $\mathrm {SL}_n({\mathbb {R}})$ . Denote by $\bigwedge ({\mathbb {R}}^n)$ the exterior power of ${\mathbb {R}}^n$ , that is, the direct sum of all $ \bigwedge ^i({\mathbb {R}}^n)$ for $ 0 \le i \le n$ . Let $\rho : H \rightarrow \mathrm {GL}(\bigwedge {\mathbb {R}}^n)$ be the representation of H induced by the linear representation of H on ${\mathbb {R}}^n$ .

Since H is semisimple, $\rho $ decomposes into a direct sum of irreducible representations of H parametrized by their highest weights $\unicode{x3bb} $ . For each $\unicode{x3bb} $ , denote by $V^{\unicode{x3bb} }$ the direct sum of all irreducible subrepresentations of $\rho $ with highest weight $\unicode{x3bb} $ . Denote by $\tau _{\unicode{x3bb} }$ the canonical orthogonal projection of $\bigwedge ({\mathbb {R}}^n)$ onto $V^{\unicode{x3bb} }$ . Fix $\varepsilon>0$ . Following [4, 19], define the Benoist–Quint $\varphi $ -function

$$ \begin{align*}\varphi_{\varepsilon} : \bigwedge({\mathbb{R}}^n)\mapsto [0, \infty]\end{align*} $$

for $v \in \bigwedge ^i({\mathbb {R}}^n), 0<i <n$ , by

$$ \begin{align*}\varphi_{\varepsilon} (v) = \begin{cases} \min_{\unicode{x3bb}\neq 0} \varepsilon^{(n-i)i}\|\tau_{\unicode{x3bb}}(v)\|^{-1} & \text{if}\ \|\tau_0(v)\|\leq \varepsilon^{(n-i)i}, \\ 0 & \text{otherwise.} \end{cases} \end{align*} $$

Let us define $f_{\varepsilon }: \mathrm {SL}_n({\mathbb {R}})/\mathrm {SL}_n({\mathbb {Z}}) \to [0, \infty ]$ by

$$ \begin{align*}f_\varepsilon(\Lambda)= \max \{ \varphi_{\varepsilon} (v): v \in \Omega(\Lambda) \}. \end{align*} $$

1.4 Outline of the paper

This paper is organized as follows. In §2, after recalling some preliminaries, we state and prove results about the equidistribution of translated orbits of the form $a_tKg{\mathbb {Z}}^n$ in the orbit closure. This requires us to classify all the intermediate subgroups that can potentially appear in the conclusion of Ratner’s theorem. In §3, we recall Siegel’s integral formula and prove Theorem 3.5, which is an analogue for a subset of lattices that all share a rational vector. This proof relies on the boundedness of some integrals (see Theorem 3.3) involving $\alpha $ -function, which is proven in the beginning of this section. In §4, we will show that the integral of the $\alpha $ -function along certain orbit translates is uniformly bounded. This is one of the major ingredients of the proof. In §5, we will use results of the previous sections to establish Theorem 1.2. Finally, §6 is devoted to presenting counterexamples illustrating that the analogue of Theorem 1.2 does not hold for certain forms of signatures $(2, 2)$ and $(2,3).$

2 Equidistribution results

In this section, we will relate Question 1.1 to the question of equidistribution of certain orbit translates in homogeneous spaces. In §2.1, we recall some preliminaries and in §2.2, we establish a connection to the homogeneous dynamics.

2.1 Preliminaries: canonical forms for pairs $({\mathbf q}, {\mathbf l})$ and their stabilizers

In this subsection, we will first introduce some notation and recall a number of basic facts about the space of unimodular lattices in ${\mathbb {R}}^n$ . Then we will recall the classification in [11] of pairs consisting of a quadratic form and a linear form under the action of $\mathrm {SL}_n({\mathbb {R}})$ .

Let ${\mathbf q}$ be a non-degenerate isotropic quadratic form on ${\mathbb {R}}^n$ . There exists $ 1 \le p \le n-1$ , $ \unicode{x3bb} \in {\mathbb {R}} \setminus \{ 0 \}$ and $g \in \mathrm {SL}_n({\mathbb {R}})$ such that

$$ \begin{align*} \unicode{x3bb} \cdot {\mathbf q}(gx)= 2x_1x_2+ x_3^2+ \cdots + x_{p+1}^2 -(x_{p+2}^2 + \cdots + x_n^2).\end{align*} $$

We say that ${\mathbf q}$ has signature $(p, n-p)$ . We need a similar classification for pairs of quadratic and linear forms. Let ${\mathbf q}$ be as above and let ${\mathbf l}$ be a non-zero linear form on ${\mathbb {R}}^n$ . For $g \in \mathrm {SL}_n({\mathbb {R}})$ , define

$$ \begin{align*}{\mathbf q}^g(x)= {\mathbf q}(gx), \quad {\mathbf l}^g(x)= {\mathbf l}(gx).\end{align*} $$

For $i=1,2$ , let ${\mathbf q}_i$ and ${\mathbf l}_i$ be as above. We say that $({\mathbf q}_1, {\mathbf l}_1)$ is equivalent to $({\mathbf q}_2, {\mathbf l}_2)$ if ${\mathbf q}_1= \unicode{x3bb} \cdot {\mathbf q}_2^g$ and ${\mathbf l}_1 = \mu \cdot {\mathbf l}_2^g$ for some $g \in \mathrm {SL}_n({\mathbb {R}})$ and non-zero scalars $\unicode{x3bb} $ and $\mu $ . Using the action of $\mathrm {SL}_n({\mathbb {R}})$ , we can transform any pair $({\mathbf q}, {\mathbf l})$ into a standard pair.

Proposition 2.1. [11, Proposition 2]

Every pair $({\mathbf q}, {\mathbf l})$ as above is equivalent to one and only one of the following:

(2.1) $$ \begin{align} \ \kern1pt\mathrm{(I)} \;\; & (2 x_1 x_2 + x_3^2+ \cdots + x_{p+1}^2 - x^2_{p+2}+ \cdots - x_{n}^2, x_n) \quad p=1, \ldots, n-1, \end{align} $$

(2.2) $$ \begin{align} \mathrm{(II)} \;\; & (2x_1 x_2 \kern1.3pt{+}\kern1.3pt x_3^2\kern1.3pt{+} \cdots {+}\kern1.3pt x_{p}^2 \kern1.3pt{-}\kern1.3pt x_{p+1}^2\kern1.3pt{-}\cdots {-}\kern1.3ptx_{n-2}^2\kern1.3pt{+}\kern1.3pt2x_{n-1}x_n, x_n) \kern-1pt\quad p\kern1.3pt{=}\kern1.3pt1, \ldots, \kern-1pt[n/2\kern1.3pt{-}\kern1.3pt1]. \end{align} $$

Pairs in equations (2.1) and (2.2) are referred to as type I and II, respectively. It can be seen that the pair $({\mathbf q}, {\mathbf l})$ is of type I if and only if the restriction of ${\mathbf q}$ to $\ker {\mathbf l}$ is non-degenerate. In this paper, we deal only with pairs $({\mathbf q}, {\mathbf l})$ of type I satisfying conditions (A) and (B). We denote this set by $\mathscr {S}_n$ .

Remark 2.2. For pairs of type II satisfying conditions (A) and (B), it appears that the maximal compact subgroup K preserving both ${\mathbf q}$ and ${\mathbf l}$ is not sufficiently large for our methods to apply.

2.2 Connection to the homogenous dynamics and equidistribution results

Let ${G=\mathrm {SL}_n({\mathbb {R}})}$ and $\Gamma = \mathrm {SL}_n({\mathbb {Z}})$ . Denote the Lie algebra of G by $\mathfrak {sl}_n({\mathbb {R}})$ . Suppose that $({\mathbf q}, {\mathbf l})$ is equivalent to

(2.3) $$ \begin{align} ({\mathbf q}_0, {\mathbf l}_0) = (2 x_1 x_2 + x_3^2+ \cdots + x_{p+1}^2 - x_{p+2}^2 - \cdots - x_{n}^2, x_n). \end{align} $$

Let H be the subgroup of $\mathrm {SL}_n({\mathbb {R}})$ defined by

$$ \begin{align*} H= \left(\begin{array}{c|c} & 0 \\ {\mathrm{SO}}(p,q-1)^\circ & \vdots \\ & 0 \\ \hline 0 \quad \cdots \quad 0 & 1 \end{array}\right)\!. \end{align*} $$

Denote by ${\mathrm {SO}}({\mathbf q}_0,{\mathbf l}_0)$ the subgroup of ${\mathrm {SO}}({\mathbf q}_0)$ that stabilizes ${\mathbf l}_0$ , where $({\mathbf q}_0, {\mathbf l}_0)$ is defined as in equation (2.5) so that ${\mathrm {SO}}({\mathbf q}_0,{\mathbf l}_0)^\circ $ is isomorphic to H. The Lie algebra of H, denoted by , consists of the subalgebra consisting of matrices of the form

(2.4)

It is not difficult to see that $K := H \cap {\mathrm {SO}}(n)$ is a maximal compact subgroup of H and is isomorphic to ${\mathrm {SO}}(p)\times {\mathrm {SO}}(q-1)$ . Denote the canonical basis of $\mathbb {R}^n$ by $\{e_1, \ldots , e_n \}$ . Let $a_t$ denote the one-parameter subgroup defined by

(2.5) $$ \begin{align} a_t e_1= e^{-t} e_1, \quad a_t e_2= e^t e_2, \quad a_t e_j = e_j, \quad 3 \le j \le n. \end{align} $$

Using this notation, we can state one of the main results of this paper.

Theorem 2.3. For $p\ge 3$ , $q\ge 2$ and $0<s<2$ . Then for every $ \Lambda \in \mathscr {X}_n$ , we have

$$ \begin{align*} \sup_{t>0} \int_K \alpha(a_tk \Lambda)^s \,dm(k) < \infty. \end{align*} $$

This theorem is analogous to [9, Theorem 3.2]. What makes the proof of Theorem 2.3 more difficult is that the integration is over a proper subgroup of ${\mathrm {SO}}(p) \times {\mathrm {SO}}(q)$ . In general, one can see that if K is replaced by an arbitrary subgroup of ${\mathrm {SO}}(p) \times {\mathrm {SO}}(q)$ with large co-dimension, then the analogue of Theorem 2.3 may not hold. As a result, establishing the boundedness of the integral requires a more delicate analysis of the excursion to the cusp of the translated orbit $a_tK \Lambda $ . Using Theorem 2.3, we will prove the theorem below from which Theorem 1.2 will be deduced.

Theorem 2.4. Suppose $p\ge 3$ , $q\ge 2$ and $s>1$ . Let $\phi : \mathscr {X}_n \to {\mathbb {R}}$ be a continuous function such that

$$ \begin{align*} | \phi( \Lambda) | \le C \alpha( \Lambda)^s \end{align*} $$

for all $ \Lambda \in \mathscr {X}_n$ and some constant $C>0$ . Let $ \Lambda \in \mathscr {X}_n$ be such that $\overline {H \Lambda }$ is either $\mathscr {X}_n$ or is of the form $ (\mathrm {SL}_{n-1}({\mathbb {R}}) \ltimes _l {\mathbb {R}}^{n-1}) \Lambda $ , where $\mathrm {SL}_{n-1}({\mathbb {R}}) \ltimes _l {\mathbb {R}}^{n-1}$ is defined by equation (2.6). Then,

$$ \begin{align*} \lim_{ t \to \infty} \int_K \phi(a_tk \Lambda) \, dm(k) = \int_{_{\overline{H \Lambda }} } \phi \, d\mu_{ _{\overline{H \Lambda }}}, \end{align*} $$

where $\mu _{ _{\overline {H \Lambda }}}$ is the H-invariant probability measure on $\overline {H \Lambda }$ .

We shall see that Theorem 2.4 will apply to $ \Lambda =g_0 {\mathbb {Z}}^n$ , when $({\mathbf q}_0^{g_0}, {\mathbf l}_0^{g_0}) \in \mathscr {S}_n$ , see Theorem 2.8.

The methods used are inspired by those employed in [9]. We will recall a theorem of Dani and Margulis after introducing some terminology and set some notation. Let G be a real Lie group with the Lie algebra . Let denote the adjoint representation of G. An element $g \in G$ is called Ad-unipotent if ${\mathrm {Ad}}(g)$ is a unipotent linear transformation. A one-parameter group $\{ u_t \}$ is called Ad-unipotent if every $u_t$ is an Ad-unipotent element of G. In this section, we will recall some results from [7, 9] that will be needed in the following.

As in the proof of the quantitative Oppenheim conjecture [9], a key role is played by Ratner’s equidistribution theorem. Suppose G is a connected Lie group, $\Gamma <G$ a lattice and H is a connected subgroup of G generated by unipotent elements in H. Ratner’s orbit closure theorem asserts that for every point $x \in G/\Gamma $ , there exists a connected closed subgroup L containing H such that $\overline {Hx}= Lx$ . Moreover, $Lx$ carries an L-invariant probability measure $\mu _{L}$ . To apply Ratner’s theorem in concrete situations, one needs to be able to classify all subgroups L that can arise. In the next subsection, we will classify all connected subgroups of $\mathrm {SL}_n({\mathbb {R}})$ containing H. Using well-known results in Lie theory, this classification problem is equivalent to the problem of classifying all Lie subalgebras of containing .

2.3 Intermediate subgroups

We will maintain the notation as in §2.2. Since is semisimple, , regarded as an -module, can be decomposed as the direct sum of irreducible -invariant subspaces. For $ 1 \le i, j \le n$ , let $E_{ij}$ be the $n \times n$ matrix whose only non-zero entry is $1$ and is located on the ith row and jth column. We will refer to $\{ E_{ij}: 1 \le i, j \le n \}$ as the canonical basis of the Lie algebra $\mathfrak {gl}_n({\mathbb {R}})$ .

Proposition 2.5. The Lie algebra splits as the direct sum of irreducible -invariant subspaces

where:

• • consists of all matrices of the form

  $$ \begin{align*} \left(\begin{array}{cc|c} A & B & 0 \\ -B^{t} & D & 0 \\ \hline 0 & 0 & 0 \end{array}\right)\!, \end{align*} $$
  and A and D are symmetric matrices of size p and $(q-1)$ , respectively, such that $\mathrm {tr}(A)+\mathrm {tr}(D)=0$ , and B is an arbitrary p by $q-1$ matrix;

• • is the $(n-1)$ -dimensional subspace spanned by $ E_{in}, 1 \le i \le n-1$ ;

• • is the $(n-1)$ -dimensional subspace spanned by $ E_{ni}, 1 \le i \le n-1$ ;

• • is the one-dimensional subspace spanned by $E_{11}+ \cdots + E_{n-1,n-1}-(n-1)E_{nn}$ .

Proof. The only challenging assertion lies in demonstrating that an -invariant subspace is -irreducible. Using the weight decomposition of for the restricted root system of , one can establish this assertion by showing that any weight vector of can be transformed into another weight vector via the adjoint action of restricted roots (for further elaboration, refer to [13]).

Let map $E_{in}$ to $E_{ni}$ for $ 1 \le i \le p$ and $E_{in}$ to $- E_{ni}$ for ${ p\kern1.5pt{+}\kern1.5pt1 \kern1.5pt{\le}\kern1.5pt i \kern1.5pt{\le}\kern1.5pt n\kern1.5pt{-}\kern1.5pt1}$ . In other words,

$$ \begin{align*} \Phi \bigg( \sum_{ i =1}^{p} v_i E_{in} + \sum_{ i =p+1}^{n-1} v_i E_{in} \bigg) := \sum_{ i =1}^{p} v_i E_{ni} - \sum_{ i =p+1}^{n-1} v_i E_{ni}. \end{align*} $$

One can verify that $\Phi $ is an -module isomorphism. For any non-zero $\xi \in {\mathbb {R}}$ , consider the subspace

It is clear that . Set also . Note that for $ \xi \neq 0, \infty $ , the subspace is not a subalgebra of .

Remark 2.6. Define the quadratic form ${\mathbf q}_{\xi }$ by

$$ \begin{align*} {\mathbf q}_{\xi}(v)=(x_1^2+\cdots+x_p^2-x_{p+1}^2-\cdots-x_{n-1}^2)+\xi x_n^2. \end{align*} $$

The Lie algebra for $ \xi \in {\mathbb {R}} \setminus \{ 0 \}$ decomposes as . Moreover, any quadratic form ${\mathbf q}'$ for which ${\mathrm {SO}}({\mathbf q}')$ contains H is of the form ${\mathbf q}_\xi $ up to scalar multiplication.

Proposition 2.7. Let be a subalgebra of containing . Then is one of the Lie algebras in Table 1.

Table 1 List of intermediate subalgebras.

Proof. Before we start the proof, let us recall that

Let be as in the statement of Proposition 2.7. Since is semisimple and is an -submodule of , decomposes into a direct sum of and irreducible -invariant subspaces, each isomorphic to one of and . Note that aside from and , which are isomorphic -modules, no other two of these -modules are isomorphic. One can thus write , where is a direct sum of with a subset of , and is an -submodule of . We will consider several cases. First assume that . All -submodules of are of the form for $ \xi \in {\mathbb {R}} \cup \{ \infty \}$ . This leads to the submodules , and , all of which are subalgebras of . Consider the case . One can easily see that , are both subalgebras of . However, the inclusion

rules out the potential candidate . The case can be dealt with similarly. In view of the inclusion

the potential candidates for $ \xi \neq 0, \infty $ are ruled out, while and are all possible. The last case can be studied similarly.

For a subgroup F of $\mathrm {SL}_{n-1}({\mathbb {R}})$ , denote

(2.6) $$ \begin{align} F\ltimes_u {\mathbb{R}}^{n-1}=\left( \begin{array}{c|c} & \\ F & {\mathbb{R}}^{n-1} \\ & \\ \hline 0 \:\cdots\: 0 & 1 \end{array} \right)\quad\text{and}\quad F\ltimes_l {\mathbb{R}}^{n-1}=\left( \begin{array}{c|c} & 0 \\ F & \vdots \\ & 0 \\ \hline \:{\mathbb{R}}^{n-1}\: & 1 \end{array} \right)\!. \end{align} $$

Theorem 2.8. (Classification of possible orbit closures)

Assume that $({\mathbf q}, {\mathbf l})\in \mathscr {S}_n$ . Let $g_0\in \mathrm {SL}_n({\mathbb {R}})$ be such that ${\mathrm {SO}}({\mathbf q}, {\mathbf l})^\circ =g_0^{-1}Hg_0$ . Let $F\le G$ denote the closed Lie subgroup containing H with the property that $\overline {H g_0\Gamma }=g_0F\Gamma \subseteq G/\Gamma $ . Then either $F=G$ or $F =g_0^{-1} (\mathrm {SL}_{n-1}({\mathbb {R}}) \ltimes _l {\mathbb {R}}^{n-1}) g_0$ .

One ingredient of the proof is the following theorem of Shah.

Theorem 2.9. [20, Proposition 3.2]

Let ${\mathbf {G}} \le \mathrm {SL}_n$ be a ${\mathbb {Q}}$ -algebraic group and ${G= \mathbf {G}({\mathbb {R}})^{\circ }}$ . Set $\Gamma = \mathbf {G}({\mathbb {Z}})$ and let L be a subgroup which is generated by algebraic unipotent one-parameter subgroups of G contained in L. Let $ \overline {L\Gamma } = F \Gamma $ for a connected Lie subgroup F of G. Let $ \mathbf {F}$ be the smallest algebraic ${\mathbb {Q}}$ -group containing L. Then the radical of $ \mathbf {F}$ is a unipotent ${\mathbb {Q}}$ -group and $ F = \mathbf {F}({\mathbb {R}})^{\circ }$ .

Proof of Theorem 2.8

The proof relies on Proposition 2.7. Recall that ${\mathrm {SO}}({\mathbf q},{\mathbf l})\simeq {\mathrm {SO}}(p,q-1)$ is semisimple and there are two proper ${\mathrm {SO}}({\mathbf q},{\mathbf l})$ -invariant subspaces $\mathcal L_1$ and $\mathcal L_2$ in the dual space $({\mathbb {R}}^n)^*$ of ${\mathbb {R}}^n$ with $\dim \mathcal L_1=n-1$ and $\dim \mathcal L_2=1$ . Notice that since $({\mathbf q}, {\mathbf l})\in \mathscr {S}_n$ , $\mathcal L_2$ is an irrational subspace.

Let . After conjugation by $g_0$ , since , the Lie algebra is a subalgebra of appearing in Table 1 of Proposition 2.7. We will show that if , then only possible is .

Claim 1. does not contain .

By Theorem 2.9, F is (the connected component of) the smallest algebraic ${\mathbb {Q}}$ -group and the radical of F is a unipotent algebraic ${\mathbb {Q}}$ -group. According to Table 1, if , the radical of is one of , or , which is not possible since is not unipotent.

Claim 2. is not contained in the radical of .

If is in the radical of , then is either or . In both cases, F has invariant subspaces $\mathcal L_1$ and $\mathcal L_2$ in $({\mathbb {R}}^n)^*$ . Since F is a ${\mathbb {Q}}$ -group, any F-invariant subspace in $({\mathbb {R}}^n)^*$ is defined over ${\mathbb {Q}}$ . In particular, $\mathcal L_2$ must be a rational subspace, which is a contradiction.

Claim 3. F is not semisimple.

If $F\lneq G$ is semisimple, then F is either ${\mathrm {SO}}({\mathbf q},{\mathbf l})^\circ $ or ${\mathrm {SO}}({\mathbf q}+\xi {\mathbf l}^2)^\circ $ for some ${\xi \in {\mathbb {R}}-\{0\}}$ . If F is ${\mathrm {SO}}({\mathbf q},{\mathbf l})^\circ $ , F has an invariant subspace $\mathcal L_2$ in $({\mathbb {R}}^n)^*$ , which leads to a contradiction as in Claim 2. If $F\simeq {\mathrm {SO}}({\mathbf q}+\xi {\mathbf l}^2)^\circ $ , since F is defined over ${\mathbb {Q}}$ , ${\mathbf q}+\xi {\mathbf l}^2$ is a scalar multiple of a rational form. This contradicts our assumption that $\alpha {\mathbf q}+\beta {\mathbf l}^2$ is not rational for all non-zero $(\alpha ,\beta )\in {\mathbb {R}}^2$ .

Thus, aside from , the only possible option for is .

Claim 4. Levi subgroup of F is not isomorphic to ${\mathrm {SO}}({\mathbf q},{\mathbf l})^\circ $ .

Suppose not. Let L be a unipotent radical of F. By the Levi–Malcev theorem ([14], see also [1, Corollary 3.5.2]), there is $\ell \in L$ such that $\ell ^{-1}{\mathrm {SO}}({\mathbf q},{\mathbf l})\ell ={\mathrm {SO}}({\mathbf q}^{\ell }, {\mathbf l}^{\ell })$ is a Levi subgroup of F, which is defined over ${\mathbb {Q}}$ .

Choose a basis ${\mathbf l}_1, \ldots , {\mathbf l}_{n-1}$ of $\mathcal L_1$ and ${\mathbf l}_n$ of $\mathcal L_2$ such that ${\mathbf q}={\mathbf l}_1^2+\cdots +{\mathbf l}_p^2-{\mathbf l}_{p+1}^2-\cdots -{\mathbf l}_{n}^2$ . Since the action of L fixes elements of $\mathcal L_1$ , the space $\mathcal L_1$ is an ${\mathrm {SO}}({\mathbf q}^{\ell }, {\mathbf l}^{\ell })$ -invariant subspace which is defined over ${\mathbb {Q}}$ by the assumption of ${\mathrm {SO}}({\mathbf q}^{\ell }, {\mathbf l}^{\ell })$ . Choose a rational linear form ${\mathbf l}_0\in ({\mathbb {R}}^n)^*$ such that $\langle {\mathbf l}_0\rangle $ is ${\mathrm {SO}}({\mathbf q}^{\ell }, {\mathbf l}^{\ell })$ -invariant and $({\mathbb {R}}^n)^*=\mathcal L_1 \oplus \langle {\mathbf l}_0\rangle $ . Clearly, ${{\mathbf l}_0=c{\mathbf l}^{\ell }}$ for some $c\in {\mathbb {R}}-\{0\}$ . Moreover, by Remark 2.6, since any quadratic forms fixed by ${\mathrm {SO}}({\mathbf q}^{\ell }, {\mathbf l}^{\ell })$ are of the form

$$ \begin{align*} {\mathbf q}'&=\alpha' ({\mathbf l}_1^2+\cdots+{\mathbf l}_p^2-{\mathbf l}_{p+1}^2-\cdots-{\mathbf l}_{n-1}^2)^{\ell} + \beta' {\mathbf l}_0^2\\ &=\alpha' ({\mathbf l}_1^2+\cdots+{\mathbf l}_p^2-{\mathbf l}_{p+1}^2-\cdots-{\mathbf l}_{n-1}^2)+\beta'{\mathbf l}_0^2, \end{align*} $$

there is a non-trivial $(\alpha ',\beta ')\in {\mathbb {R}}^2$ such that ${\mathbf q}'$ is rational. Since $\mathcal L_1$ is an $(n-1)$ - dimensional rational subspace of $({\mathbb {R}}^n)^*$ , there is a rational vector $v\in {\mathbb {R}}^n$ such that ${\mathbf l}_j(v)=0$ for all $1\le j\le n-1$ and ${\mathbf l}_0(v)\neq 0$ . Evaluating ${\mathbf q}'$ on v, we have $\beta '{\mathbf l}_0(v)\in {\mathbb {Q}}$ so that $\beta '$ is a rational number. It follows that ${\mathbf q}+{\mathbf l}^2=(1/\alpha ')({\mathbf q}'-{\beta '} {\mathbf l}_0^2)$ is a rational quadratic form, which is a contradiction.

Proposition 2.10. Let $G=\mathrm {SL}_n({\mathbb {R}})$ and $\Gamma =\mathrm {SL}_n({\mathbb {Z}})$ . Let $({\mathbf q}, {\mathbf l})\in \mathscr {S}_n$ and F be a closed subgroup of G for which $\overline {{\mathrm {SO}}({\mathbf q},{\mathbf l})^\circ \Gamma }=F\Gamma $ . Then $F\simeq \mathrm {SL}_{n-1}({\mathbb {R}})\ltimes _l {\mathbb {R}}^{n-1}$ if and only if there exists a non-zero $v\in {\mathbb {Q}}^{n}$ that is ${\mathrm {SO}}({\mathbf q},{\mathbf l})$ -invariant.

Proof. Suppose that $F\simeq \mathrm {SL}_{n-1}({\mathbb {R}})\ltimes _l {\mathbb {R}}^{n-1}$ . Since F is a ${\mathbb {Q}}$ -group by Theorem 2.9, there is $g_1\in \mathrm {SL}_n({\mathbb {Q}})$ for which ${\mathrm {SO}}({\mathbf q},{\mathbf l})\subseteq g_1^{-1} (\mathrm {SL}_{n-1}({\mathbb {R}})\ltimes _l {\mathbb {R}}^{n-1})g_1$ . Since $\mathrm {SL}_{n-1}({\mathbb {R}})\ltimes _l {\mathbb {R}}^{n-1}$ fixes $e_n$ , ${\mathrm {SO}}({\mathbf q},{\mathbf l})$ fixes $g_1e_n$ which is a non-zero rational vector.

Conversely, suppose that ${\mathrm {SO}}({\mathbf q}, {\mathbf l})$ fixes a non-zero rational vector $v\in {\mathbb {Q}}^n$ . Since

$$ \begin{align*} F:=\{g\in \mathrm{SL}_n({\mathbb{R}}) : gv=v \} \end{align*} $$

is an algebraic group defined over ${\mathbb {Q}}$ , $F\cap \Gamma $ is a lattice subgroup of F. Since F contains ${\mathrm {SO}}({\mathbf q}, {\mathbf l})$ , it follows that $\overline {{\mathrm {SO}}({\mathbf q},{\mathbf l})^\circ \Gamma }\subseteq F\Gamma $ . Then the equality automatically holds by Theorem 2.8.

For closed subgroups U, H of G, define

$$ \begin{align*} X(H,U)=\{g\in G : Ug\subseteq gH \}. \end{align*} $$

Note that if $g \in X(H, U)$ and $ H\Gamma \subseteq G/\Gamma $ is closed, then the orbit $ U g \Gamma $ is included in the closed subset $ gH\Gamma $ and hence cannot be dense. The next theorem asserts that for a fixed $ \varepsilon>0$ and a continuous compactly supported test function $\phi $ by removing finitely many compact subsets $C_i$ of such sets, the time average over $[0, T]$ of $\phi $ remains within $ \varepsilon $ of the space average for sufficiently large values of T.

Theorem 2.11. [7, Theorem 3]

Let G be a connected Lie group and $\Gamma $ be a lattice in G. Denote by $\mu $ the G-invariant probability measure on $G/\Gamma $ . Let $U= \{ u_t \}$ be an Ad-unipotent one-parameter subgroup of G and let $ \phi : G/\Gamma \to {\mathbb {R}}$ be a bounded continuous function. Suppose $\mathcal D$ is a compact subset of $G/\Gamma $ and $ \varepsilon>0$ . Then there exist finitely many proper closed subgroups $H_1, \ldots , H_k$ such that $H_i \cap \Gamma $ is a lattice in $H_i$ for all $ 1\le i \le k$ , and compact subsets $C_i \subseteq X(H_i, U)$ such that the following holds. For every compact subset $\mathcal F \subseteq \mathcal D- \bigcup _{ i=1}^{ k } C_i \Gamma /\Gamma $ , there exists $T_0 \ge 0$ such that for all $x \in {\mathcal {F} }$ and all $T> T_0$ , we have

$$ \begin{align*} \bigg| \frac{1}{T} \int_0^T \phi( u_tx) \, dt - \int_{G/\Gamma} \phi \,d\mu \bigg| < \varepsilon. \end{align*} $$

If H is isomorphic to ${\mathrm {SO}}(p,q-1)^\circ $ , since we have a classification of all intermediate (connected) Lie subgroups between ${\mathrm {SO}}(p,q-1)^\circ $ and $\mathrm {SL}_n({\mathbb {R}})$ , one can obtain concrete statements. Using Theorem 2.8, we will prove Theorem 2.13 below, which is in the spirit of [9, Theorems 4.4 or 4.5]. However, due to the presence of intermediate subgroups, both the statement and the proof are more involved.

Recall that closed subgroups $H_i$ in Theorem 2.11 are those who give the orbit closures of U in $G/\Gamma $ . Notice that in our case, since $G=\mathrm {SL}_d({\mathbb {R}})$ is ${\mathbb {Q}}$ -algebraic and $\Gamma =\mathrm {SL}_d({\mathbb {Z}})$ is an arithmetic lattice subgroup, one can apply Theorem 2.9, that is, $H_i$ terms are ${\mathbb {Q}}$ -algebraic and with unipotent radical.

We say that $X \subseteq {\mathbb {R}}^d$ is a real algebraic set if X is equal to the set of common zeros of a set of polynomials. We need the following lemma.

Lemma 2.12. Let X be an affine algebraic set over the field of real numbers. Suppose that $Y_1, Y_2, \ldots $ are countably many affine algebraic sets such that X is covered by the union of $Y_i$ , $i \ge 1$ . Then X is covered by the union of only finitely many of $Y_i$ .

Proof. Assume, without loss of generality, that X is irreducible. The intersection ${X_i = X \cap Y_i}$ is an affine algebraic set, and hence is either X or a proper algebraic subset of X. Suppose that there is no $Y_i$ for which $X_i= X$ . Since every proper algebraic subset is of lower dimension, and hence of Lebesgue measure zero, we obtain a contradiction to the assumption that X is covered by countably many $Y_i$ terms. Consequently, there exists $i \ge 1$ such that $X\subseteq Y_i$ .

Theorem 2.13. Let $G, \Gamma , H$ and K be as in §2.2. Let $\phi $ , $\mathcal D$ and $\varepsilon>0$ be as in Theorem 2.11. Let $\psi $ be a bounded measurable function on K. Then there exist a finite set $R \subseteq G/\Gamma $ and closed subgroups $L_x \le G$ associated to every $x \in R$ such that we have the following.

1. (1) For $x\in R$ , $L_x$ is one of the following:

   (2.7) $$ \begin{align} \begin{gathered} TH, \;T\mathrm{SL}_{n-1}({\mathbb{R}}), \;{\mathrm{SO}}({\mathbf q}_{\xi})^\circ\;(\xi\in {\mathbb{Q}}-\{0\}), \;T({\mathrm{SO}}(p,q-1)^\circ\ltimes_{u} {\mathbb{R}}^{n-1})\\ T({\mathrm{SO}}(p,q-1)^\circ\ltimes_{l} {\mathbb{R}}^{n-1}),\;T(\mathrm{SL}_{n-1}({\mathbb{R}})\ltimes_{u} {\mathbb{R}}^{n-1}) \text{ and }\;T(\mathrm{SL}_{n-1}({\mathbb{R}})\ltimes_{l} {\mathbb{R}}^{n-1}), \end{gathered} \end{align} $$
   where ${\mathbf q}_{\xi }$ is defined as in Remark 2.6 and ${T=\{\mathrm {diag}(e^{t}, \ldots , e^{t}, e^{-(n-1)t}): t\in {\mathbb {R}}\}}$ . Here, $TL$ is the subgroup generated by T and L.

   Moreover, for each $x\in R$ , $L_x.x\subseteq G/\Gamma $ is a closed submanifold with a positive codimension. In particular, $\mu (L_x.x)=0$ .

2. (2) For every compact set

   $$ \begin{align*}\mathcal F \subseteq \mathcal D \setminus \bigcup_{x \in R} L_x. x, \end{align*} $$
   there exists $t_0>0$ such that for any $x\in \mathcal F$ and every $t>t_0$ , the following holds:
   $$ \begin{align*} \bigg|\! \int_K \phi(a_tkx)\psi(k)\,dm(k)- \int_{G/\Gamma} \phi\,d\mu \int_K \psi\,dm \bigg|\le \varepsilon. \end{align*} $$

Proof. We will follow the strategy of [9, Theorem 4.4 (II)]. Let us first verify the following statement, which is an analogue of [9, Theorem 4.3]: let $U=\{u_t\}$ be a given Ad-unipotent one-parameter subgroup of H. We need to find sets $R_1$ , $R_2$ and closed subgroups $F_x$ terms so that for any compact set $\mathcal F\subseteq \mathcal D\setminus \bigcup _{x\in R_1\cup R_2} F_x.x$ , there is $T_0>0$ such that for any $x\in \mathcal F$ and $T>T_0$ , it holds that

(2.8) $$ \begin{align} m\bigg(\bigg\{ k \in K : \bigg| \frac 1 T \int_0^T \phi (u_tkx)\,dt - \int_{G/\Gamma} \phi \,d\mu\bigg|>\varepsilon \bigg\}\bigg)\le \varepsilon. \end{align} $$

Let $H_i=H_i(\phi , K\mathcal D, \varepsilon )$ and $C_i=C_i(\phi , K\mathcal D, \varepsilon )$ , $1\le i \le k$ , be as in Theorem 2.11 for U. For each i, define

$$ \begin{align*} Y_i=\{y\in G : Ky \subset X(H_i, U) \}. \end{align*} $$

The group generated by $ \bigcup _{k\in K} k^{-1}Uk $ is normalized by $U \cup K$ . Since K is maximal in H, we obtain $\langle \bigcup _{k\in K} k^{-1}Uk \rangle =H$ . Let $y\in Y_i$ . Since $Uky \subseteq ky H_i$ for all $k \in K$ , the previous assertion implies that $H \le yH_i y^{-1}$ .

Note that $H_i$ is a closed subgroup of G defined over ${\mathbb {Q}}$ and $H_i\cap \Gamma $ is a lattice in $H_i$ . Moreover, the radical of $H_i$ is unipotent by Theorem 2.9. It follows from Theorem 2.7 that $F_{i,y}:=yH_iy^{-1}$ belongs to the following list:

(2.9) $$ \begin{align} \begin{gathered} H, \;\mathrm{SL}_{n-1}({\mathbb{R}}), {\mathrm{SO}}({\mathbf q}_\xi)^\circ \;(\xi\in {\mathbb{Q}}-\{0\}), \;{\mathrm{SO}}(p,q-1)^\circ\ltimes_{u} {\mathbb{R}}^{n-1}\\ {\mathrm{SO}}(p,q-1)^\circ\ltimes_{l} {\mathbb{R}}^{n-1},\;\mathrm{SL}_{n-1}({\mathbb{R}})\ltimes_{u} {\mathbb{R}}^{n-1} \text{ and }\;\mathrm{SL}_{n-1}({\mathbb{R}})\ltimes_{l} {\mathbb{R}}^{n-1}. \end{gathered}\end{align} $$

Note that the only groups conjugate to each other in the list in equation (2.9) are those of the form $ {\mathrm {SO}}({\mathbf q}_\xi )$ for $\xi \in {\mathbb {Q}}-\{0\}$ . Based on this fact, we will distinguish two cases.

Case I: $H_i$ is not isomorphic to $ {\mathrm {SO}}({\mathbf q}_\xi )$ for any $\xi \in {\mathbb {Q}}-\{0\}$ . Consider $y_1,\;y_2\in Y_i$ such that $F_{i,y_1}=F_{i,y_2}=:F_i$ , that is, $y_1^{-1}y_2\in \mathrm {N}_G(F_i)$ , where $F_i$ is one of equation (2.9). Thus, $Y_i\Gamma \subseteq \mathrm {N}_G(F_i)y_1\Gamma $ . Since G is semisimple, $\mathrm {N}_G(F_i)$ is a real algebraic group and has finitely many connected components [22, Theorem 3]. Moreover, it is easy to check that $T \subseteq \mathrm {N}_G(F_i) $ and $\mathrm {N}_G(F_i)^\circ =TF_i$ . Hence, all orbits $Y_i\Gamma /\Gamma $ of this form can be covered by finitely many orbits of $TF_i$ .

Case II: $H_i$ is isomorphic to ${\mathrm {SO}}({\mathbf q}_{\xi })^\circ $ for some $\xi \in {\mathbb {Q}}-\{0\}$ . We will partition $Y_i$ as

(2.10) $$ \begin{align} Y_i = \bigsqcup_{_{\xi \in {\mathbb{Q}}-\{0\} }} (Y_i \cap Z_\xi), \end{align} $$

where $Z_\xi = \{ y \in G : y H_i y^{-1}= {\mathrm {SO}}({\mathbf q}_\xi )^\circ \}.$ For each $\xi \in {\mathbb {Q}}-\{0\}$ , if $Z_\xi $ is non-empty, then it is a coset of $\mathrm {N}_G({\mathrm {SO}}({\mathbf q}_\xi )^\circ )$ , and hence is an algebraic set. Note that

$$ \begin{align*} Y_i=\{g\in G : Kg \subset X(H_i, U) \}= \bigcap_{k\in K} \{ g \in G : kg \in X( H_i, U) \} =\bigcap_{k\in K} k^{-1} X(H_i, U). \end{align*} $$

We claim that $X(H_i, U)$ is an algebraic set and $Y_i$ , being an intersection of algebraic sets, is also algebraic. The proof of this claim is essentially included in [7, Proposition 3.2]. Write $h=\dim H_i$ and define

Note that $\rho _{H_i}$ is an algebraic representation of G. We also know (see [7, Proposition 3.2]) that $g \in X(H_i, U)$ if and only if , which, in turn, is equivalent to the condition that $ (\rho _{H_i}( g) p_{H_i}) \wedge w=0$ for all . This is, clearly, an algebraic condition.

It follows from equation (2.10) and Lemma 2.12 that there exists finitely many rational numbers $\xi _1, \ldots , \xi _m$ such that

$$ \begin{align*} Y_i \subseteq \bigcup_{1 \le j \le m} (Y_i \cap Z_{\xi_j}). \end{align*} $$

For each $1\le j\le m$ , suppose that $y_1,\;y_2 \in Y_i$ are such that $y_1 H_i y_1^{-1}=y_2 H_i y_2^{-1}={\mathrm {SO}}({\mathbf q}_{\xi _j})^\circ $ . Then $y_1^{-1}y_2 \in \mathrm {N}_G({\mathrm {SO}}({\mathbf q}_{\xi _j})^\circ )$ and we conclude that

$$ \begin{align*} Y_i\Gamma = \bigcup_{1\le j\le m} \{ y \in Y_i : y H_i y^{-1}= {\mathrm{SO}}({\mathbf q}_{\xi_j})^\circ \} \Gamma \subseteq \bigcup_{1\le j\le m} \mathrm{N}_G({\mathrm{SO}}({\mathbf q}_{\xi_j})^\circ)y_{\xi_j} \Gamma \end{align*} $$

for some $y_{\xi _j} \in Y_i$ . In view of the fact that $\mathrm {N}_G({\mathrm {SO}}({\mathbf q}_\xi )^\circ )$ is a finite union of right cosets of ${\mathrm {SO}}({\mathbf q}_\xi )^\circ $ , there exists a finite set $R \subseteq G/\Gamma $ and a closed subgroup $L_x$ as in equation (2.9) so that

$$ \begin{align*} \bigcup_{i} Y_i \Gamma/\Gamma \subseteq \bigcup_{x\in R} L_x.x. \end{align*} $$

By the definition of $H_i$ , $L_x.x$ for each $x\in R$ is a proper closed submanifold in $G/\Gamma $ .

Since $X(H_i,U)$ is a real analytic submanifold and K is connected, for any $x\in \mathcal F$ ,

$$ \begin{align*} m\bigg(\bigg\{k\in K : kx\in \bigcup_{1\le i\le k} C_i\Gamma/\Gamma \bigg\}\bigg)=0. \end{align*} $$

By [9, Theorem 4.2], there is an open set $W\subset G/\Gamma $ for which $\bigcup _{1\le i\le k} C_i\Gamma /\Gamma \subseteq W$ and $m(\{k\in K: kx\in W\})<\varepsilon $ for any $x\in \mathcal F$ .

Let $T_0$ be as in Theorem 2.11. Then for any $x\in \mathcal F$ and $k\in K$ with $kx\notin W$ , we have

$$ \begin{align*} \bigg|\frac 1 T \int_0^T \phi(u_tkx)\,dt - \int_{G/\Gamma} \phi \,d\mu\bigg| < \varepsilon, \end{align*} $$

which shows equation (2.8). We will skip the rest of the proof since it closely parallels the proof of [9, Theorem 4.4 (II)] once we replace [9, Theorem 4.3] by the inequality in equation (2.8).

Theorem 2.14. Let $g_0\in \mathrm {SL}_n({\mathbb {R}})$ be such that $G_1:=g_0^{-1} ( \mathrm {SL}_{n-1}({\mathbb {R}})\ltimes _l {\mathbb {R}}^{n-1} ) g_0$ is defined over ${\mathbb {Q}}$ , and $\Gamma _1:=G_1\cap \mathrm {SL}_n({\mathbb {Z}})$ is a lattice in $G_1$ . Let $H^{g_0}=g_0^{-1} H g_0 < G_1$ . Let $K^{g_0}$ be a maximal compact subgroup of $H^{g_0}$ and $\{a^{g_0}_t=g_0^{-1} \mathrm {diag}(e^{-t}, e^t, 1, \ldots , 1) g_0: t\in {\mathbb {R}} \}$ be a one-parameter subgroup of $H^{g_0}$ . Let $\phi $ , $\mathcal D$ and $\varepsilon>0$ be as in Theorem 2.11 for $G=G_1$ and $\Gamma =\Gamma _1$ , and let $\psi $ be a bounded measurable function on $K^{g_0}$ . Then there are finitely many points ${x_i}$ and closed subgroups $L_i$ , $1\le i \le \ell $ , so that $(g_0^{-1} L_ig_0).x_i$ is closed for every $1\le i \le \ell $ , and for any compact $\mathcal F \subseteq (\mathcal D - \bigcup _{i=1}^{\ell } (g_0^{-1} L_ig_0).x_i)$ , there is $t_0>0$ such that for any $x\in \mathcal F$ and $t>t_0$ ,

$$ \begin{align*} \bigg|\! \int_{K^{g_0}} \phi(a^{g_0}_tkx)\psi(k)\,dm(k)- \int_{G_1/\Gamma_1} \phi\,d\mu \int_{K^{g_0}} \psi\,dm \bigg|\le \varepsilon. \end{align*} $$

Here, $L_i$ is one of

(2.11) $$ \begin{align} {\mathrm{SO}}(p,q-1)^\circ, \;\mathrm{SL}_{n-1}({\mathbb{R}}) \quad\text{and}\quad{\mathrm{SO}}(p,q-1)^\circ\ltimes_{l} {\mathbb{R}}^{n-1}. \end{align} $$

Proof. The proof is similar to that of Theorem 2.13. In this case, possible proper intermediate subgroups $H_i$ terms are listed in equation (2.11). It is not hard to see that for each $H_i$ in this list, $H_i= \mathrm {N}_G( H_i)^{\circ }$ .

3 Siegel integral formula for an intermediate subgroup

In this section, we will prove a version of Siegel’s integral formula for intermediate subgroups F in Proposition 2.10. For a bounded and compactly supported function $f:{\mathbb {R}}^n\rightarrow {\mathbb {R}}$ , the Siegel transform of f is defined by

$$ \begin{align*} \widetilde{f} (g) := \widetilde{f} (g {\mathbb{Z}}^n) = \sum_{_{v \in {\mathbb{Z}}^n - \{ 0 \} }} f( gv). \end{align*} $$

Lemma 3.1. (Schmidt [9, Lemma 3.1])

Let $f: {\mathbb {R}}^n \to {\mathbb {R}}$ be a bounded function vanishing outside of a bounded set. Then there exists a constant $c=c(f)$ such that

$$ \begin{align*} \widetilde{ f}( \Lambda) < c \alpha( \Lambda) \end{align*} $$

for all unimodular lattices $ \Lambda $ in ${\mathbb {R}}^n$ .

In the rest of this section, we will change the notation slightly and write $ \alpha (g)$ for $ \alpha (g {\mathbb {Z}}^n)$ . One can see that the inequality $\alpha (g_1g_2)\le \alpha (g_1)\alpha (g_2)$ does not always hold. The following lemma singles out special cases in which this inequality holds.

Lemma 3.2. Let $a, g, g_1, g_2\in \mathrm {SL}_n({\mathbb {R}})$ . Assume, further, that a is self-adjoint. Then we have:

1. (1) $\dfrac { \alpha (g_1g_2)}{ \alpha (g_2)}\le \max \limits _{1\le j\le n} \|\wedge ^{j}g_1^{-1}\|_{\mathrm {op}}$ ;

2. (2) $\alpha (ag)\le \alpha (a)\alpha (g)$ .

Proof. By the definition of $ \alpha $ , we have

$$ \begin{align*} \alpha(g_1g_2)=\max_{1\le j \le n} \left\{\frac 1 {\|g_1v_1 \wedge \cdots \wedge g_1v_j\|} : \begin{array}{l@{}} v_1, \ldots, v_j \in g_2{\mathbb{Z}}^n,\\ v_1 \wedge \cdots \wedge v_j \neq 0 \end{array} \right\}\!. \end{align*} $$

It follows from the definition of the operator norm that for any $1\le j\le n$ and any linearly independent vectors $v_1, \ldots , v_j \in g_2{\mathbb {Z}}^n$ , we have

$$ \begin{align*} \|g_1v_1 \wedge \cdots \wedge g_1v_j\| =\|(\wedge^i g_1)(v_1 \wedge \cdots \wedge v_j)\| \ge \|\wedge^i g_1^{-1}\|^{-1}_{\mathrm{op}} \:\|v_1 \wedge \cdots \wedge v_j\|. \end{align*} $$

This proves item (1).

To show item (2), we first assume that $a=\mathrm {diag}(a_1,\ldots , a_n)$ . Recall that for multi-indices $I=\{1\le i_1 < \cdots < i_j\le n\}$ and $L=\{1\le \ell _1 < \cdots < \ell _j \le n\}$ , the $(I,L)$ -component of $\wedge ^i a$ is

$$ \begin{align*}(\wedge^i a)_{IL} = \begin{cases} \prod_{j} a_{i_j} &\text{if } I=L,\\ 0 &\text{otherwise.}\end{cases} \end{align*} $$

Therefore,

(3.1) $$ \begin{align} \sup_{1\le j\le n} \| \wedge^j a^{-1}\|_{\mathrm{op}} =\sup_{1 \le j \le n} \bigg(\!\sup\bigg\{\frac 1 {a_{i_1}\cdots a_{i_j}} : 1\le i_1 < \cdots < i_j \le n \bigg\}\bigg). \end{align} $$

However, since a is a diagonal matrix,

(3.2) $$ \begin{align} \alpha(a)=\sup_j \bigg(\frac 1 {\min\{{a_{i_1}\cdots a_{i_j}} : 1 \le i_1 < \cdots < i_j \le n\}}\bigg). \end{align} $$

Combining equations (3.1) and (3.2) with the first result, we obtain the second property.

For an adjoint matrix $a'\in \mathrm {SL}_n({\mathbb {R}})$ , we can write $a'=kak^{-1}$ , where a is diagonal and $k\in {\mathrm {SO}}(n)$ . Notice that the $\alpha $ function is invariant under left multiplication by ${\mathrm {SO}}(n)$ . Using item (2),

$$ \begin{align*} \alpha(a'g)=\alpha((kak^{-1})g)=\alpha(ak^{-1}g) \le\alpha(a)\alpha(k^{-1}g)=\alpha(a')\alpha(g).\\[-35pt] \end{align*} $$

The following theorem is an analogue of [9, Lemma 3.10], where a similar statement for the integral of $ \alpha ^r$ over $\mathscr {X}_n$ is proven.

Theorem 3.3. Let $g_0 \in \mathrm {SL}_n({\mathbb {R}})$ be such that the algebraic group

$$ \begin{align*}F=g_0^{-1}(\mathrm{SL}_{n-1}({\mathbb{R}})\ltimes^{}_{l}{\mathbb{R}}^{n-1})g_0\end{align*} $$

is defined over ${\mathbb {Q}}$ and that $\Gamma _F:=F \cap \Gamma $ is a lattice in F. Denote by $\mu _F$ the F-invariant probability measure on $F/\Gamma _F$ , and let $\mathcal F_{F} \subseteq F$ be a fundamental domain for the action of $\Gamma _F$ on F. Then for any $1\le r <n-1$ ,

$$ \begin{align*} \int_{\mathcal F_{F}} \alpha^r(g) \,d\mu^{}_F(g) < \infty. \end{align*} $$

Proof. Since F is defined over ${\mathbb {Q}}$ , there exists $g_1\in \mathrm {SL}_n({\mathbb {R}})$ such that $g_0^{-1}(\mathrm {SL}_{n-1}({\mathbb {R}})\ltimes ^{}_{l}{\mathbb {R}}^{n-1})g_0= g_1^{-1}(\mathrm {SL}_{n-1}({\mathbb {R}})\ltimes ^{}_{l}{\mathbb {R}}^{n-1})g_1$ and $F_0=g_1^{-1}\mathrm {SL}_{n-1}({\mathbb {R}})g_1$ is a Levi subgroup for F defined over ${\mathbb {Q}}$ . Note that the unipotent radical of F is given by $R=g_1^{-1}(\{\mathrm {Id}_{n-1}\}\ltimes {\mathbb {R}}^{n-1})g_1$ and is defined over ${\mathbb {Q}}$ (see [5]).

Recall that if H is a connected algebraic group defined over ${\mathbb {Q}}$ , then the discrete subgroup $H({\mathbb {Z}})$ is a lattice in H if and only if H does not admit a non-trivial character defined over ${\mathbb {Q}}$ (see [17, Theorem 4.13]). Since $F_0$ is semisimple and R is polynomially isomorphic to ${\mathbb {R}}^{n-1}$ , they do not have non-trivial polynomial characters, and hence $F_0({\mathbb {Z}})=F_0 \cap \Gamma _F$ and $R({\mathbb {Z}})=R\cap \Gamma _F$ are lattices in $F_0$ and R, respectively. Moreover, since R is abelian, $R({\mathbb {Z}})$ is cocompact.

Let $\mathcal F_{F_0}$ and $\mathcal F_R$ be fundamental domains for $F_0/F_0({\mathbb {Z}})$ and $R/R({\mathbb {Z}})$ , respectively. One can find a fundamental domain $\mathcal F_{F}\subseteq \mathcal F_{F_0} \times \mathcal F_{R}$ .

Now, we want to cover $\mathcal F_{F_0}$ by a finite union of copies of a Siegel set of $\mathrm {SL}_{n-1}({\mathbb {R}})$ . Recall that the standard Siegel set $\Sigma =\Sigma _{\eta ,\xi }$ of $\mathrm {SL}_n({\mathbb {R}})$ is the product ${\mathrm {SO}}(n) A_{\eta } N_{\xi }$ , where

$$ \begin{align*} A_\eta&=\{\mathrm{diag}(a_1, \ldots, a_n)\in \mathrm{SL}_n({\mathbb{R}}) : 0< a_i<\eta a_{i+1} \}\quad\text{and}\\ N_{\xi}&=\{(u_{ij})\text{: upper unipotent}\in \mathrm{SL}_n({\mathbb{R}}) : |u_{ij}| \le \xi\}. \end{align*} $$

It is well known that a fundamental domain of $\mathrm {SL}_n({\mathbb {R}})/\mathrm {SL}_n({\mathbb {Z}})$ is contained in $\Sigma _{\eta ,\xi }$ for some appropriate $\eta ,\; \xi>0$ (see [17, Theorem 4.4] for instance). Moreover, since $F_0$ is a semisimple Lie group defined over ${\mathbb {Q}}$ and $g_1 F_0 g_1^{-1}$ is self-adjoint, by a theorem of Borel and Harish-Chandra ([3], see also [17, Theorems 4.5 and 4.8]), there are $\gamma _1, \ldots , \gamma _k\in \mathrm {SL}_n({\mathbb {Z}})$ such that for $\mathcal {D}=(\bigcup _{i=1}^k g_1^{-1} \Sigma \gamma _i)\cap F_0$ , one has $\mathcal {D} F_0({\mathbb {Z}})=F_0$ .

Note that $g_1^{-1}\Sigma g_1$ is a Siegel set with respect to the Iwasawa decomposition $K^{g_1}=g_1^{-1}K_0g_1$ , $A^{g_1}=g_1^{-1}A_0g_1$ and $N^{g_1}=g_1^{-1}N_0g_1$ . By [3, Lemma 7.5], for each $g_1^{-1}\Sigma \gamma _i=g_1^{-1}\Sigma g_1 (g_1^{-1}\gamma _i)$ , there are finitely many $g^i_j$ terms for which

$$ \begin{align*} g_1^{-1}\Sigma\gamma_i\cap F_0 \subseteq \bigcup_j g_1^{-1}\Sigma_1 g_1 g^i_j, \end{align*} $$

for some $\Sigma _1$ , where $\Sigma _1$ is some standard Siegel set of $\mathrm {SL}_{n-1}({\mathbb {R}})(\subseteq \mathrm {SL}_n({\mathbb {R}}))$ , so that $g_1^{-1}\Sigma _1 g_1$ is a Siegel set with respect to the Iwasawa decomposition $K^{g_1}\cap F_0$ , $A^{g_1}\cap F_0$ and $N^{g_1}\cap F_0$ . Therefore, by change of variables and using the fact that $\mathrm {SL}_n({\mathbb {R}})$ is unimodular,

$$ \begin{align*} \int_{\mathcal F_{F}} \alpha^r(g) \,d\mu^{}_{\mathcal F}(g) \le \sum_{i,j} \int_{\Sigma_1\times \mathcal F_R} \alpha^r(g_1^{-1}g g_1g^{i}_j \:h) \,d\mu^{}_{\mathrm{SL}_{n-1}({\mathbb{R}})}(g) \,d\mu^{}_R(h). \end{align*} $$

Let $\Sigma _1=(\Sigma _1)_{\eta ',\xi '}$ and denote $g=k' a' n'$ , where $k'\in {\mathrm {SO}}(n-1)$ , $a'=\mathrm {diag}(a^{\prime }_1, \ldots , a^{\prime }_{n-1}, 1)$ for which $a^{\prime }_i \le \eta ' a^{\prime }_{i+1}$ and $n'=(u^{\prime }_{ij})$ is the upper unipotent element in $\mathrm {SL}_{n-1}({\mathbb {R}})\ltimes \{0\}$ such that $|u^{\prime }_{ij}|\le \xi '$ for any $(i,j)$ with $i<j$ . Since $d\mu ^{}_{\mathrm {SL}_{n-1}({\mathbb {R}})\ltimes \{0\}}$ is locally $\Delta (a') dk'da'dn'$ , where $\Delta (a')$ is the product of positive roots, using Lemma 3.2 and [9, Lemma 3.10], it follows that for $1\le r < n-1$ ,

$$ \begin{align*} \int_{\mathcal F_{F}} \alpha^r(g) \,d\mu^{}_{\mathcal F}(g) & \ll_{\,g_1} \sum_{i,j} \int_{A^{\prime}_{\eta'}}\int_{N^{\prime}_{\xi'}\times\mathcal F_R} \alpha^r(a')\alpha^r(n' g_1g^i_j h) \Delta(a') \,da'\,dn' \,d\mu^{}_R(h)\\ &\le C \sum_{i,j} \int_{A^{\prime}_{\eta'}} \alpha^r(a') \Delta(a')\,da' < \infty \end{align*} $$

for some $C>0$ since $N_{\xi '}\times \mathcal F_{R}$ is compact. Here,

$$ \begin{align*} A^{\prime}_{\eta'}&=\{\mathrm{diag}(a_1, \ldots, a_{n-1}, 1) \in \mathrm{SL}_{n-1}({\mathbb{R}}) : 0<a_i\le \eta' a_{i+1} \}\quad\text{and}\\ N^{\prime}_{\xi'}&=\{(u^{\prime}_{ij})\text{: upper unipotent}\in \mathrm{SL}_{n-1}({\mathbb{R}}) : |u^{\prime}_{ij}| \le \xi'\}.\\[-3.1pc] \end{align*} $$

Recall the well-known Siegel integral formula.

Theorem 3.4. (Siegel [21])

For a bounded and compactly supported function ${f:{\mathbb {R}}^n\rightarrow {\mathbb {R}}}$ , we have

$$ \begin{align*} \int_{G/\Gamma} \widetilde{f} (g) \,d\mu(g)= \int_{{\mathbb{R}}^n} f(v)\,dv. \end{align*} $$

We also need the analogue of Siegel’s integral formula for the following specific intermediate subgroup.

Theorem 3.5. Assume that $g_0\in \mathrm {SL}_n({\mathbb {R}})$ is such that $F=g_0^{-1}(\mathrm {SL}_{n-1}({\mathbb {R}})\ltimes ^{}_{l}{\mathbb {R}}^{n-1})g_0$ is an algebraic group defined over ${\mathbb {Q}}$ and that $\Gamma _F:=F \cap \Gamma $ is a lattice. Denote by $\mu _F$ the probability F-invariant measure on $F/\Gamma _F$ and by $\mathcal F_F$ a fundamental domain for $\Gamma _F$ in F. Then for any bounded compactly supported measurable function $f:{\mathbb {R}}^n\rightarrow {\mathbb {R}}$ , we have

$$ \begin{align*} \int_{\mathcal F_F} \widetilde f(g) \,d\mu^{}_F(g) =\int_{{\mathbb{R}}^n} f(v) dv+ \sum_{m\in {\mathbb{Z}}-\{0\}} f(m\:k_0g_0^{-1}e_n), \end{align*} $$

where $k_0$ is determined by ${\mathbb {R}}.g_0^{-1}e_n \cap {\mathbb {Z}}^n={\mathbb {Z}}.k_0g_0^{-1}e_n$ .

Proof. By Lemma 3.1 and Theorem 3.3, the integral

$$ \begin{align*}\int_{F/\Gamma_F} \widetilde f(g) \,d\mu^{}_{F}(g)\end{align*} $$

is finite, and the map sending f to $ \int _{F/\Gamma _F} \widetilde f(g) \,d\mu ^{}_{F}(g)$ is a continuous positive linear functional on the space of compactly supported continuous functions and is hence given by a finite measure.

Note that the set of F-fixed vectors in ${\mathbb {R}}^n$ is ${\mathbb {R}}.g_0^{-1}e_n$ which is defined over ${\mathbb {Q}}$ , and F acts transitively on ${\mathbb {R}}^n-{\mathbb {R}}.g_0^{-1}e_n$ . Since ${\mathbb {R}}.g_0^{-1}e_n \cap g{\mathbb {Z}}^n$ is ${\mathbb {Z}}$ -span of $k_0g_0^{-1}e_n$ for some $0\neq k_0\in {\mathbb {R}}$ , it follows from the usual argument of Siegel’s integration formula combined with Proposition 3.3 (see [12, §3]) that

$$ \begin{align*} \int_{\mathcal F_F} \tilde f (g) \,d\mu(g)= \int_{{\mathbb{R}}^n} f(v) \,dv + \sum_{m\in {\mathbb{Z}}-\{0\}} f(m\:k_0g_0^{-1}e_n).\\[-49pt]\end{align*} $$

4 Upper bounds for spherical averages of the $\alpha $ -function

In this section, we will prove the following theorem, which is an analogue of [9, Theorem 3.2].

Theorem 4.1

1. (1) For $p\ge 3$ , $q\ge 2$ and $0<s<2$ . Then for every $g \in \mathrm {SL}_n({\mathbb {R}})$ , we have

   $$ \begin{align*} \sup_{t>0} \int_K \alpha(a_tk.g{\mathbb{Z}}^n)^s \,dm(k) < \infty. \end{align*} $$

2. (2) For $p=2$ , $q=3$ , there is $0<s<1$ such that

   $$ \begin{align*} \sup_{t>0} \int_K \alpha(a_tk.g{\mathbb{Z}}^n)^s \,dm(k) < \infty. \end{align*} $$

The proof is based on the following proposition, which is [9, Proposition 5.12].

Proposition 4.2. Consider a self-adjoint reductive subgroup H of $\mathrm {GL}_n({\mathbb {R}})$ . Let ${K=\mathrm {O}_n({\mathbb {R}})\cap H}$ and let m be the normalized Haar measure of K. Let $A=\{a_t : t\in {\mathbb {R}}\}$ be a self-adjoint one-parameter subgroup of H and let $\mathcal F$ be a family of strictly positive functions on H having the following properties.

1. (a) For any $\varepsilon>0$ , there is a neighbourhood $V(\varepsilon )$ of $\mathrm {Id}$ in H such that for any $f\in \mathcal F$ ,

   $$ \begin{align*} (1-\varepsilon)f(h) < f(uh) < (1+\varepsilon)f(h)\quad \text{for all } h\in H\quad \text{for all } u\in V(\varepsilon). \end{align*} $$

2. (b) For any $f\in \mathcal F$ , $f(Kh)=f(h)$ for all $h \in H$ .

3. (c) $\sup _{f\in \mathcal F} f(\mathrm {Id})<\infty $ .

Then there exists a positive constant $c=c(\mathcal F) <1$ such that for all $t_0>0, b>0$ , there exists $B=B(t_0,b)<\infty $ with the following property: if $f \in \mathcal F$ and

(4.1) $$ \begin{align} \int_K f(a_{t_0} k h) \,dm(k) < c f(h) + b \end{align} $$

$\text { for all } h \in KAK \subset H,$ then

$$ \begin{align*}\int_K f(a_\tau k) \,dm(k) <B\end{align*} $$

for any $\tau>0.$

4.1 Reduction to an $f_\varepsilon $ -function

Let us start by recalling the definition of a Benoist–Quint function. Write $\bigwedge ({\mathbb {R}}^n)=\bigoplus _{i=1}^{n-1} \bigwedge ^i({\mathbb {R}}^n)$ and consider the representation $\rho : H \rightarrow \mathrm {GL}(\bigwedge ({\mathbb {R}}^n))$ induced by the linear representation of H on ${\mathbb {R}}^n$ . Since H is semisimple, $\rho $ decomposes into a direct sum of irreducible representations. For each highest weight $\unicode{x3bb} $ , denote by $V^{\unicode{x3bb} }$ the direct sum of all irreducible components with highest weight $\unicode{x3bb} $ and by $\tau _{\unicode{x3bb} }$ the orthogonal projection on $V^{\unicode{x3bb} }$ .

For $\varepsilon>0$ and $0<i<n$ , we define the Benoist–Quint $\varphi $ -function $\varphi _{\varepsilon } : \bigwedge ({\mathbb {R}}^n)\rightarrow [0, \infty ]$ as in [4, 19]:

$$ \begin{align*}\varphi_{\varepsilon} (v) = \begin{cases} \min_{\unicode{x3bb}\neq 0} \varepsilon^{(n-i)i}\|\tau_{\unicode{x3bb}}(v)\|^{-1} & \text{if}\ \|\tau_0(v)\|\leq \varepsilon^{(n-i)i}, \\ 0 & \text{otherwise.} \end{cases} \end{align*} $$

Note that $V^0= \{v \in \bigwedge ({\mathbb {R}}^d) : Hv=v \}$ by definition. Denote by $(V^0)^\perp $ its orthogonal complement in $\bigwedge ({\mathbb {R}}^d).$

Remark 4.3

1. (1) Since $\tau _{\unicode{x3bb} }$ is defined in terms of projection of v onto $V^{\unicode{x3bb} }$ , for every ${v \in \bigwedge ({\mathbb {R}}^d)}$ and $\unicode{x3bb} \neq 0$ , we have

   $$ \begin{align*}\tau_{\unicode{x3bb}}(v)= \tau_{\unicode{x3bb}}(v- \tau_0(v)). \end{align*} $$

2. (2) Since $\max _{\unicode{x3bb} \neq 0} \| \tau _\unicode{x3bb} (v) \|$ defines a norm on $(V^0)^\perp $ , there exists $c_1>1$ such that for all $v \in (V^0)^\perp ,$

   (4.2) $$ \begin{align} \frac{1}{c_1\|v \|} \leq \frac 1 {\max_{\unicode{x3bb}\neq 0} \|\tau_\unicode{x3bb}(v)\|} \leq c_1 \frac{1}{\|v \|}. \end{align} $$

4.2 The function $f_{\varepsilon }$ and associated inequalities

Recall that $\Omega (\Lambda )=\bigcup _{i=1}^n \Omega ^i(\Lambda )$ , where $\Omega ^i(\Lambda )$ is defined by

$$ \begin{align*} \Omega^i(\Lambda)=\{v=v_1\wedge \cdots \wedge v_i: v_1, \ldots, v_i \in \Lambda\} \setminus \{ 0 \}. \end{align*} $$

For $ \varepsilon>0$ , define $f_{\varepsilon }: \mathrm {SL}_n({\mathbb {R}})/\mathrm {SL}_n({\mathbb {Z}}) \to [0, \infty ]$ by

$$ \begin{align*}f_\varepsilon(\Lambda)= \max_{v\in \Omega(\Lambda)} \varphi_{\varepsilon} (v).\end{align*} $$

We will first show that although $f_\varepsilon $ is not finite on its entire domain, its restriction to each H-orbit $H.\Lambda $ is finite for sufficiently small $\varepsilon .$

Lemma 4.4. For a given $g\in \mathrm {SL}_n({\mathbb {R}})$ , there is $\varepsilon _0>0$ such that if $0< \varepsilon < \varepsilon _0$ , the function

$$ \begin{align*}f_{g,\varepsilon}(h):=f_{\varepsilon}(hg{\mathbb{Z}}^n)\end{align*} $$

has a finite value for all $h\in H$ .

Proof. Observe that $f_{\varepsilon }(hg{\mathbb {Z}}^n)=\infty $ if and only if there is $1<i<n$ and $0\neq v \in \Omega ^i(g{\mathbb {Z}}^n)\cap V^0$ for which $\|v\|\le \varepsilon ^{i(n-i)}$ . Since any element in H is of the form $\mathrm {diag}(M, 1)$ with $M \in {\mathrm {SO}}(p,q-1)^\circ $ , H acts on $\bigoplus _{i=1}^{n-1} {\mathbb {R}}.e_i$ irreducibly. This implies that any non-zero H-fixed elements $v\in \Omega (g{\mathbb {Z}}^n)$ are scalar multiples of $e_n$ , $e_1\wedge \cdots \wedge e_{n-1}$ , or ${e_1\wedge \cdots \wedge e_n}$ . If $ \Omega (g{\mathbb {Z}}^n)$ does not contain any such vectors other than $e_1\wedge \cdots \wedge e_n$ , any value of $ \varepsilon>0$ will work. Otherwise, there exists a non-empty set S of vectors $v\in \Omega (g{\mathbb {Z}}^n)$ which are of the form

$$ \begin{align*} v= a(v) e_n \quad \textrm{or} \quad v= a(v)(e_1\wedge \cdots \wedge e_{n-1}) \end{align*} $$

for some $a(v)>0$ . Since $g{\mathbb {Z}}^n$ is discrete, $ \varepsilon _0:= \min \{ a(v)^{1/(n-1)}: v \in S \}>0$ . If $\varepsilon <\varepsilon _0$ , there are no vectors in $\Omega (g{\mathbb {Z}}^n)\cap V^0$ or norm at most $\varepsilon ^{i(n-i)}$ . It follows that the restriction of $f_\varepsilon $ to $Hg{\mathbb {Z}}^n$ is finite.

Lemma 4.5. Let $s>0$ and $g\in \mathrm {SL}_n({\mathbb {R}})$ . Let $\varepsilon>0$ be such that $f_{g,\varepsilon }(h)<\infty $ for all ${h \in H}$ . Then there exist $c_{s,\varepsilon }>0$ and $C_{s,\varepsilon }>0$ depending on s and $\varepsilon $ such that for all $ h \in H$ , we have

$$ \begin{align*} \alpha(hg{\mathbb{Z}}^n)^s \le c_{s,\varepsilon} f_{g,\varepsilon}(h)^s + C_{s,\varepsilon}. \end{align*} $$

Proof. Write $ \varepsilon _1=\min _{1\le i\le n-1} \varepsilon ^{i(n-i)}$ and $\varepsilon _2= \max _{1\le i\le n-1} \varepsilon ^{i(n-i)}$ , and define ${c_{s,\varepsilon }=({c_1}/ \varepsilon _1)^s}$ and $C_{s,\varepsilon }= \varepsilon _2^s+1$ , where $c_1$ is chosen as in equation (4.2). In view of equation (4.2), for all $v \in \bigoplus _{\unicode{x3bb} \neq 0} V^\unicode{x3bb} $ , we have

$$ \begin{align*} \frac { \varepsilon_1} {c_1 \|v\|} \le \varphi_{\varepsilon}(v) \le \frac {c_1 \varepsilon_2}{\|v\|}. \end{align*} $$

Let $v\in \Omega ^i(hg{\mathbb {Z}}^n)$ be the vector at which $\alpha (hg{\mathbb {Z}}^n)$ is attained. We will consider two cases.

If $\|\tau _0(v)\|> \varepsilon ^{i(n-i)}$ , then we have

$$ \begin{align*} \alpha(hg{\mathbb{Z}}^n) = \frac 1 {\|v\|} \le \frac 1 {\|\tau_0(v)\|} \le \varepsilon^{-i(n-i)} \le C_{s, \varepsilon}. \end{align*} $$

Otherwise, we have $\|\tau _0(v)\| \le \varepsilon ^{i(n-i)}$ . In this case, by the choice of $\varepsilon $ , we must have ${v\neq \tau _0(v)}$ . This implies that

$$ \begin{align*} \alpha(hg{\mathbb{Z}}^n)= \frac 1 {\|v\|} \le \frac 1 {\|v-\tau_0(v)\|} \le \frac{c_1}{ \varepsilon_1} \varphi_{\varepsilon} (v-\tau_0(v))=\frac{c_1}{ \varepsilon_1} \varphi_{\varepsilon}(v) \le c_{s,\varepsilon} f_{g, \varepsilon}(h). \end{align*} $$

The claim follows by combining these two cases.

Lemma 4.6. Suppose $p\ge 3$ , $q\ge 2$ and $ s \in (0,2)$ or $p=2$ , $q=2, 3$ and $s \in (0,1)$ . Then, for every $c>0$ , there exists $t_0>0$ such that for every $t>t_0$ and $v\in \bigwedge ^i({\mathbb {R}}^n)-V^0$ , the following holds:

$$ \begin{align*} \int_K \frac 1 {\max_{\unicode{x3bb}\neq0} \|\tau_\unicode{x3bb}(a_tkv)\|^s} \, dm(k) \le \frac {c} {\max_{\unicode{x3bb}\neq0} \|\tau_\unicode{x3bb}(v)\|^s}. \end{align*} $$

Proof. Let $v \in \bigwedge ^i({\mathbb {R}}^n)- V^0$ . By part (1) of Remark 4.3, we may assume that ${v\in \bigoplus _{\unicode{x3bb} \neq 0} V^\unicode{x3bb} }$ . It follows from [9, Proposition 5.4] and the inequality in equation (4.2) that

$$ \begin{align*} \int_K \frac 1 {\max_{\unicode{x3bb}\neq0} \|\tau_\unicode{x3bb}(a_tkv)\|^s} \,dm(k) &\le c_1\int_K \frac{1}{\| a_t k v \|^s} \,dm(k) < c_1c' \frac{1}{\|v\|^s}\\ &\le c_1^2c' \frac 1 {\max_{\unicode{x3bb}\neq0} \|\tau_\unicode{x3bb}(v)\|^s}.\end{align*} $$

Indeed, one can use [9, Proposition 5.4] as follows: let $W^-, W^0, W^+$ be the eigenspaces corresponding to eigenvalues $e^{-t}, 1, e^t$ (of $a_t$ ) in $\bigwedge ^i ({\mathbb {R}}^n)$ , respectively. From $v \not \in V^0$ , it follows that $Kv \nsubseteq W^0$ . Since $p\ge 3$ and $q\ge 2$ , we deduce that conditions (a), (b), (c) of [9, Lemma 5.2] are satisfied. For $p=2$ and $q=2, \; 3$ , one can directly show that conditions (a), (b) of [9, Lemma 5.1] are satisfied.

Proposition 4.7. Let $g\in \mathrm {SL}_d({\mathbb {R}})$ . Suppose $p\ge 3$ , $q\ge 2$ and $ s \in (0,2)$ or $p=2$ , $q=2, 3$ and $s \in (0,1)$ . One can find $\varepsilon _1>0$ for which for any $\varepsilon \in (0, \varepsilon _1)$ and for any $c>0$ , there are $t_0$ and $b>0$ such that for every $h \in H$ , the following inequality holds:

$$ \begin{align*} \int_K f_{g, \varepsilon} (a_{t_0} kh)^s \,dm(k) < c f_{g, \varepsilon}(h)^s + b.\end{align*} $$

Proof. Let $\Omega ^i$ be the set of monomials in $\bigwedge ^i({\mathbb {R}}^n)$ for $0\le i\le n$ . By [19], there exists $C>0$ such that for all $0<\varepsilon <1/C,$ and $u \in \Omega ^{i_1}$ , $v \in \Omega ^{i_2}, w \in \Omega ^{i_3}$ with $i_1 \geq 0, i_2>0, i_3>0$ and

$$ \begin{align*}\varphi_\varepsilon ( u \wedge v ) \geq 1, \varphi_\varepsilon (u \wedge w ) \geq 1,\end{align*} $$

we have following.

1. (1) If $i_1>0$ and $i_1 + i_2 + i_3 <d,$ then

   $$ \begin{align*} \min \{ \varphi_\varepsilon ( u \wedge v), \varphi_\varepsilon ( u \wedge w) \} \leq (C \varepsilon )^{{1}/{2}} \max \{\varphi_\varepsilon(u), \varphi_\varepsilon ( u \wedge v \wedge w) \}.\end{align*} $$

2. (2) If $i_1=0$ and $i_1 + i_2+ i_3 < d$ , then

   $$ \begin{align*}\min \{ \varphi_\varepsilon ( v ) , \varphi_\varepsilon ( w ) \} \leq (C \varepsilon)^{{1}/{2}} \varphi_\varepsilon (v \wedge w).\end{align*} $$

3. (3) If $i_1>0, i_1 + i_2 + i_3 = d$ and $\| u \wedge v \wedge w \| \geq 1,$ then

   $$ \begin{align*}\min \{ \varphi_\varepsilon ( u \wedge v) , \varphi_\varepsilon ( u \wedge w) \} \leq (C\varepsilon)^{{1}/{2}} \varphi_\varepsilon(u).\end{align*} $$

4. (4) If $i =0, i_1 + i_2 + i_3 =d$ and $\|v \wedge w \| \geq 1,$ then

   $$ \begin{align*}\min \{ \varphi_\varepsilon(v), \varphi_\varepsilon (w) \} \leq b_1,\end{align*} $$
   where $b_1 = \sup \{ \varphi _\varepsilon (v) : v \in \bigwedge ({\mathbb {R}}^n): \|v\| \geq 1 \}.$

By Lemma 4.6, there exists $t_0>0$ , independent of the choice of $\varepsilon>0$ , such that for any $v\in \bigwedge ({\mathbb {R}}^n)$ with $\varphi _\varepsilon (v)\neq 0$ , we have

(4.3) $$ \begin{align} \int_K \varphi_\varepsilon (a_{t_0}kv)^s \,dm(k) \le \frac c {2n} \varphi_\varepsilon (v)^s. \end{align} $$

Let $m_0=e^{t_0s}\ge 1$ so that

$$ \begin{align*} \frac 1 {m_0} \varphi_\varepsilon (v) \le \varphi_\varepsilon (a_{t_0}v) \le m_0\varphi_\varepsilon (v). \end{align*} $$

Define the set

$$ \begin{align*} \Psi(hg{\mathbb{Z}}^n)=\{v\in \Omega(hg{\mathbb{Z}}^n): f_{\varepsilon,g}(h) \le m_0^2 \varphi_\varepsilon(v)\}. \end{align*} $$

Note that

$$ \begin{align*}f_{\varepsilon,g}(h) =\max_{v\in \Omega(hg{\mathbb{Z}}^n)} \varphi_\varepsilon(v) =\max_{v\in \Psi(hg{\mathbb{Z}}^n)} \varphi_\varepsilon(v) \end{align*} $$

and if $v\in \Omega (hg{\mathbb {Z}}^n)$ is such that $f_{g, \varepsilon }(h)=\varphi _\varepsilon (v)$ , then $v\in \Psi (hg{\mathbb {Z}}^n)$ . Choose $\varepsilon>0$ small enough so that

(4.4) $$ \begin{align} m_0^4 C \varepsilon <1. \end{align} $$

Case 1. $f_{\varepsilon ,g}(h)=f_{\varepsilon }(hg{\mathbb {Z}}^n)\le \max \{b_1, m_0^2\}$ . For any $k\in K$ , since $f_\varepsilon $ is left K-invariant,

$$ \begin{align*} f_\varepsilon(a_{t_0}khg{\mathbb{Z}}^n) \le m_0 f_\varepsilon(khg{\mathbb{Z}}^n) =m_0 f_\varepsilon(hg{\mathbb{Z}}^n), \end{align*} $$

and hence it follows that

(4.5) $$ \begin{align} \int_K f_{g, \varepsilon}(a_{t_0}kh)^s \,dm(k) \le ( m_0\max\{b_1, m_0^2\})^s. \end{align} $$

Case 2. $f_{\varepsilon ,g}(h)> \max \{b_1, m_0^2\}$ . One can deduce that $\Psi (hg{\mathbb {Z}}^n)$ contains at most one element up to sign change in each degree from exactly the same argument for [19, Claim 3.9] with the assumption $m^4_0C\varepsilon <1$ .

Note that for any $v\in \Omega (hg{\mathbb {Z}}^n)$ ,

$$ \begin{align*}\varphi_\varepsilon(a_{t_0}kv) \le \max_{\psi\in \Psi(hg{\mathbb{Z}}^n)} \varphi_\varepsilon(a_{t_0}k\psi)\end{align*} $$

since if $v\in \Psi (hg{\mathbb {Z}}^n)$ , it is obvious and if $v\notin \Psi (hg{\mathbb {Z}}^n)$ , by the definition of $\Psi (hg{\mathbb {Z}}^n)$ and  $m_0$ ,

$$ \begin{align*} \varphi_\varepsilon(a_{t_0}kv) &\le m_0 \varphi_\varepsilon(v) \le m_0^{-1} f_\varepsilon(hg{\mathbb{Z}}^n)\\ &\le m_0^{-1} \max_{\psi\in \Psi(hg{\mathbb{Z}}^n)} \varphi_\varepsilon(\psi) \le \max_{\psi\in \Psi(hg{\mathbb{Z}}^n)} \varphi_\varepsilon(a_{t_0}k\psi). \end{align*} $$

Hence,

$$ \begin{align*} \int_{K} f_{\varepsilon} (a_{t_0} khg{\mathbb{Z}}^n)^s \,dm(k) \le \sum_{\psi \in \Psi(hg{\mathbb{Z}}^n)} \int_K \varphi_\varepsilon(a_{t_0}k\psi)^s \,dm(k). \end{align*} $$

For any $\psi \in \Psi (hg{\mathbb {Z}}^n)$ , $0<f_{\varepsilon ,g}(h)/m_0^2\le \varphi _\varepsilon (v)$ , we have $\psi \notin V^0$ , and hence by equation (4.3),

$$ \begin{align*} \int_K \varphi_\varepsilon(a_{t_0}k\psi)^s \,dm(k) \le \frac c {2n} \varphi_\varepsilon (\psi)^s. \end{align*} $$

Since there is at most $2n$ elements in $\Psi (hg{\mathbb {Z}}^n)$ ,

(4.6) $$ \begin{align} \int_{K} f_{\varepsilon} (a_{t_0} khg{\mathbb{Z}}^n)^s \,dm(k) \le c\max_{\psi\in \Psi(hg{\mathbb{Z}}^n)} \varphi_\varepsilon(\psi)^s=cf_{\varepsilon,g}(h)^s. \end{align} $$

Therefore, by equations (4.5) and (4.6), it follows that

$$ \begin{align*} \int_K f_{\varepsilon,g}(a_{t_0}kh)^s \,dm(k) \le c f_{\varepsilon,g}(h)^s+(m_0\max\{b_1, m_0^2\})^s.\\[-42pt] \end{align*} $$

Proof of Theorem 4.1

By Lemma 4.5, it suffices to show that

$$ \begin{align*} \sup_{t>0} \int_K f_{g,\varepsilon}(a_tk.g{\mathbb{Z}}^n)^s \,dm(k)<\infty \end{align*} $$

for an appropriate $\varepsilon>0$ , using Proposition 4.2. The assumptions of Proposition 4.2 are obvious except the condition (c) and the inequality in equation (4.1). Choose $\varepsilon>0$ such that Lemma 4.4 and the inequality in equation (4.4) holds. Note that $m_0\ge 1$ in equation (4.4) is determined once $0<c<1$ in Proposition 4.2 is given. Then Lemma 4.4 shows the condition (c) and Proposition 4.7 shows the inequality in equation (4.1).

Proof of Theorem 2.4

The proof works exactly as the proof of in [9, Theorem 3.4]. Instead of using in [9, Theorem 3.2], one needs to use Theorem 4.1 and one of Theorems 2.13 and 2.14 depending on the orbit closure.

5 Passage to dynamics on the space $\mathscr {X}_n$ of unimodular lattices in ${\mathbb {R}}^n$

In this section, we will show how to use the equidistribution results of previous sections to prove Theorem 1.2. The methods used here are analogous to those in [9, §3]. Our assumption that $({\mathbf q}, {\mathbf l}) \in \mathscr {S}_n$ will be used in this section as well. Throughout the proof, we will assume that $n \ge 4$ . We denote by ${\mathbb {R}}^n_+$ the set of vectors $v \in {\mathbb {R}}^n$ with $ \langle v_, e_1 \rangle>0$ . The volume of the unit sphere in ${\mathbb {R}}^m$ is denoted by $\gamma _{m-1}$ . Finally, for $p+q=n$ , we write $c_{p,q}= {2^{(n-2)/2}}/{\gamma _{p-1} \gamma _{q-1}}.$

We will start by setting some notation. For $t \in {\mathbb {R}}$ , recall the one-parameter subgroup of H defined by

$$ \begin{align*}a_t=\mathrm{diag}(e^{-t}, e^{t}, 1, \ldots, 1).\end{align*} $$

Let $f: {\mathbb {R}}^{n}_+ \to {\mathbb {R}}$ be continuous of compact support. We set

$$ \begin{align*}J_f(r, \zeta, s )=\frac 1 {r^{n-3}}\int_{{\mathbb{R}}^{n-3}} f(r, x_2, x_3, \ldots, x_{n-1}, s) \, dx_3 \cdots dx_{n-1}, \end{align*} $$

where $x_{2}$ is uniquely determined so that ${\mathbf q}_0(r, x_2, \ldots , x_{n-1}, s) = \zeta $ .

Proposition 5.1. For every $\varepsilon> 0$ , there exists $t_0>0$ so that if $t>t_0$ ,

$$ \begin{align*}\bigg| c_{p, q-1} e^{(n-3)t} \int_K f(a_t kv) \,dk - J_f (\|v\|e^{-t}, {\mathbf q}_0(v), {\mathbf l}_0(v))\bigg|< \varepsilon \end{align*} $$

for any $v\in {\mathbb {R}}^n$ .

Proof. This proposition is analogous to [9, Lemma 3.6] and a special case of [19, Lemma 5.1], where the number of linear forms is set to be one and the matrix g to the identity. Let us point out that the function $J_f$ in [19, Lemma 5.1] also depends on the value of quadratic form ( $\zeta $ for us), but is not part of the notation.

Proposition 5.2. Let f be a continuous bounded function on ${\mathbb {R}}^{n}_+$ with compact support. For every $ \epsilon>0$ and $ g_0 \in G$ , the following inequality holds for sufficiently large values of t:

$$ \begin{align*} \bigg| e^{-(n-3)t} \sum_{ v \in {\mathbb{Z}}^n} J_f (\|g_0 v\|e^{-t}, {\mathbf q}_0(g_0 v), {\mathbf l}_0(g_0 v)) - c_{p, q-1} \int_K \widetilde{ f}(a_t kg_0)\,dk \bigg|< \varepsilon. \end{align*} $$

Proof. It follows from Proposition 5.1 that the number of the terms involved in the sum over vectors in ${\mathbb {Z}}^n$ is $O( e^{(n-3) t})$ . Now, the desired inequality follows by applying the conclusion of Proposition 5.1 to the vectors $g_0 v $ with $v \in {\mathbb {Z}}^n$ and summing over all these vectors.

The next proposition is similar to [9, Lemma 3.8], with the difference that the last variable s is fixed.

Proposition 5.3. Let $h=h(v, \zeta , s ): ({\mathbb {R}}^{n} \setminus \{ 0\})\times {\mathbb {R}}\times {\mathbb {R}} \to {\mathbb {R}}$ be a continuous function of compact support. Then

(5.1) $$ \begin{align} &\lim_{T\rightarrow \infty}\frac 1 {T^{n-3}}\int_{{\mathbb{R}}^n} h\bigg(\frac v T, {\mathbf l}_0(v), {\mathbf q}_0(v)\bigg)\,dv\nonumber\\ &\quad= c_{p,q-1} \int_{K}\int_{{\mathbb{R}}}\int_{{\mathbb{R}}}\int_0^\infty h(rk^{-1}e_1, \zeta, s)r^{n-3}\frac {dr}{2r} \,ds \, d\zeta \,dm(k). \end{align} $$

Proof. We start from the left-hand side of equation (5.1). Decompose the vector v as ${v=v'+v_ne_n}$ and denote by ${\mathbf q}^{\prime }_0$ the restriction of ${\mathbf q}_0$ to the hyperplane ${v_n=0}$ . Since h is compactly supported,

$$ \begin{align*} &\lim_{T\to \infty} \int_{v_n}\int_{{\mathbb{R}}^{n-1}} h\bigg(\frac {v'+v_ne_n} T, v_n, {\mathbf q}^{\prime}_0(v')-v_n^2\bigg)\,dv'\,dv_n\\&\quad=\lim_{T\to \infty}\int_{v_n}\int_{{\mathbb{R}}^{n-1}} h\bigg(\frac {v'} T, v_n, {\mathbf q}^{\prime}_0(v')-v_n^2\bigg)\,dv'\,dv_n\\ &\quad=\int_{v_n}\int_{{\mathbb{R}}^{n-1}} h_{v_n}\bigg(\frac {v'} T, {\mathbf q}^{\prime}_0(v')\bigg)\,\,dv'\,dv_n, \end{align*} $$

where $h_a(v', \xi )=h(v, a, \xi -a^2)$ . Note that $h_a$ is a function on $({\mathbb {R}}^{n-1}-\{0\})\times {\mathbb {R}}$ . By [9, Lemma 3.6],

$$ \begin{align*} &\int_{v_n}\int_{{\mathbb{R}}^{n-1}} h_{v_n}\bigg(\frac {v'} T, {\mathbf q}^{\prime}_0(v')\bigg)\,dv'\,dv_n\\ &\quad=\int_{v_n}\int_K\int_{{\mathbb{R}}}\int_0^{\infty} h_{v_n}(rk^{-1}e_1, \xi-v_n^2)r^{n-3}\frac {dr}{2r} \,d\zeta \,dm(k)\\ &\quad=\int_{{\mathbb{R}}}\int_K\int_{{\mathbb{R}}}\int_0^{\infty} h(rk^{-1}e_1, \eta, \xi)r^{n-3}\frac {dr} {2r} \,d\zeta \,d\eta, \end{align*} $$

after appropriate changing of variables.

Corollary 5.4. Let f be a continuous bounded function on ${\mathbb {R}}^{n}_+$ with compact support. Set $h(v, \xi , s)= J_f( \| v \|, \xi , s)$ . Then we have

$$ \begin{align*} \lim_{T\rightarrow \infty}\frac 1 {T^{n-3}}\int_{{\mathbb{R}}^n} h \bigg( \frac{v}{T}, {\mathbf q}_0(v), {\mathbf l}_0(v) \bigg) \ dv = c_{p, q-1} \int_{G/\Gamma} \widetilde{ f}(g) \,d\mu(g). \end{align*} $$

Proof. Using the change of variable

$$ \begin{align*} v=(v_1, \ldots, v_n) \mapsto ( v_1, \zeta, v_3, \ldots, v_{n}), \end{align*} $$

where $ \zeta = {\mathbf q}_0( x_1, \ldots , x_n)$ , the desired claim will follow.

Corollary 5.5. Let ${\mathcal {V} }_{T, I, J}({\mathbf q}, {\mathbf l})$ denote the volume of the subset of ${\mathbb {R}}^n$ consisting of vectors v for which $\|v\| < T$ , $ {\mathbf q}(v) \in I $ , and $ {\mathbf l}(v) \in J$ . Then

$$ \begin{align*} \lim_{T \to \infty} \frac{{\mathcal{V} }_{T, I, J}({\mathbf q}, {\mathbf l})}{T^{n-3}} = C({\mathbf q}, {\mathbf l}) |I| \ |J|, \end{align*} $$

where $C({\mathbf q}, {\mathbf l})$ is a constant depending only on ${\mathbf q}, {\mathbf l}$ , and $| \cdot |$ denotes the length of an interval.

Proof. This follows from Corollary 5.4. For details, see [2].

Let us now turn to the proof of the main theorem. Let $g_0 \in G$ be such that $ {\mathbf q}= {\mathbf q}_0^{g_0}$ and $ {\mathbf l}= {\mathbf l}_0^{g_0}$ . Consider the space $ \mathcal {C}$ of all functions on $ ({\mathbb {R}}^n \setminus \{ 0 \} ) \times {\mathbb {R}} \times {\mathbb {R}}$ that vanish outside of a fixed compact set and equip it with the topology of uniform convergence. It follows from Proposition 5.3 that the functional $L: \mathcal {C} \to {\mathbb {R}}$ defined by

$$ \begin{align*} L(h)= \lim_{T \to \infty} \frac{1}{T^{n-3}} \int_{{\mathbb{R}}^n} h\bigg( \frac{v}{T}, {\mathbf q}_0(v), {\mathbf l}_0(v) \bigg) \, dv\end{align*} $$

is continuous. Let $\chi $ denote the characteristic function of $ \{ v \in {\mathbb {R}}^n: \| v \| \in (1/2,1) \} \times [a,b] \times [c,d]$ . Note that

$$ \begin{align*} \sum_{ v \in {\mathbb{Z}}^n } \chi( e^{-t} v, {\mathbf q}_0( g_0 v), {\mathbf l}_0(g_0 v) ) \end{align*} $$

counts the number of $v \in {\mathbb {Z}}^n$ satisfying $ e^{t}/2 \le \| v \| \le e^t$ , $ a \le {\mathbf q}_0( g_0 v) \le b$ and ${c \le {\mathbf l}_0(g_0 v) \le d}$ . Given $ \epsilon>0$ , there exists $h_+, h_- \in \mathcal {C}$ such that

$$ \begin{align*} h_-( g_0 v, \zeta, s) \le \chi( g_0v, \zeta,s ) \le h_+( g_0 v, \zeta, s) \quad\text{and} \quad | L( h_+)- L( h_-) | < \epsilon. \end{align*} $$

One can easily verify that every compactly supported radial function is of the form $J_f(\|v\|, \zeta , s)$ for some compactly supported function f defined on ${\mathbb {R}}^n_+$ with the similar arguments in [9, p. 109]. By Proposition 5.2, Theorem 2.3, two variations of Siegel’s integral formula (Theorems 3.4 and 3.5, depending on the orbit closures) and Proposition 5.3, there exists $t_0$ such that for $t>t_0$ , we have

(5.2) $$ \begin{align} \bigg| e^{- (n-3) t} \sum_{ v \in {\mathbb{Z}}^n } h_{\pm}( e^{-t} g_0 v, {\mathbf q}_0( g_0 v) , {\mathbf l}_0(g_0 v) ) - L(h_{\pm}) \bigg| < \epsilon. \end{align} $$

Clearly, for t sufficiently large, we have

(5.3) $$ \begin{align} \bigg| e^{- (n-3) t} \int_{{\mathbb{R}}^n} h_{\pm}( e^{-t} g_0 v, {\mathbf q}_0( g_0 v) , {\mathbf l}_0(g_0 v) ) - L(h_{\pm}) \bigg| < \epsilon. \end{align} $$

We note that when we apply Theorem 3.5, since we are considering $J_f$ functions for f supported on ${\mathbb {R}}^n_+$ , we have that $f(xe_n)=0$ for any $x\in {\mathbb {R}}$ . After applying Theorem 2.3, it follows that

$$ \begin{align*} &c_{p,q-1}e^{(n-3)t}\int_{F/\Gamma_{F}} \widetilde{f}(g_0g\Gamma)\,d\mu_F(g)\\ &\quad=c_{p,q-1}e^{(n-3)t} \int_{{\mathbb{R}}^n} f(g_0v)\,dv + \sum_{m\in {\mathbb{Z}}-\{0\}} f(g_0(mk_0 g_0^{-1}e_n))\\ &\quad=c_{p,q-1}e^{(n-3)t} \int_{{\mathbb{R}}^n} f(v) \,dv + \sum_{m\in {\mathbb{Z}}-\{0\}} f(mk_0e_n)\\ &\quad=c_{p,q-1}e^{(n-3)t} \int_{{\mathbb{R}}^n} f(v) \,dv \end{align*} $$

so that we can apply Proposition 5.3. It follows that for every $ \theta>0$ , for $t>t_0$ , we have

(5.4) $$ \begin{align} (1 - \theta) \int_{{\mathbb{R}}^n} h_-( e^{-t} v, {\mathbf q}_0( g_0 v), {\mathbf l}_0(g_0 v) ) \, dv & \le \sum_{ v \in {\mathbb{Z}}^n} \chi( e^{-t} v, {\mathbf q}_0( g_0 v), {\mathbf l}_0(g_0 v) ) \nonumber\\ & \le (1 + \theta) \!\int_{{\mathbb{R}}^n}\! h_+( e^{-t} v, {\mathbf q}_0( g_0 v), {\mathbf l}_0(g_0 v) ) \, dv, \end{align} $$

which implies that for sufficiently large $T\gg 0$ ,

$$ \begin{align*} (1-\theta)\operatorname{\mathrm{vol}}({\mathbf q}^{-1}(I) \cap {\mathbf l}^{-1}(J)\cap B_T) \kern1.5pt{\le}\kern1.5pt \mathcal{N}_{T, I, J}({\mathbf q}, {\mathbf l}) \kern1.5pt{\le}\kern1.5pt (1+\theta)\operatorname{\mathrm{vol}}({\mathbf q}^{-1}(I) \kern1.5pt{\cap}\kern1.5pt {\mathbf l}^{-1}(J)\kern1.4pt{\cap}\kern1.4pt B_T). \end{align*} $$

Now, the theorem follows from Corollary 5.5.

6 Counterexamples

In this section, we provide counterexamples showing that Theorem 1.2 does not generally hold when $(p,q)=(2,2)$ and $(2,3)$ . The construction is based on the existence of forms of signature $(2,1)$ and $(2,2)$ for which equation (1.1) fails, as proven in [9].

Let us first consider the $(2,2)$ -case. For an irrational positive real number $\beta $ , set

(6.1) $$ \begin{align}\begin{aligned} {\mathbf q}_{\beta}(x_1, x_2, x_3, x_4) & = (x_1^2+x_2^2)- \beta x_3^2 - ( \beta x_3+ x_4)^2, \\ {\mathbf l}_{\beta} (x_1, x_2, x_3, x_4) & = \beta x_3+ x_4. \end{aligned}\end{align} $$

We claim that $({\mathbf q}_\beta , {\mathbf l}_\beta )$ belongs to $\mathscr {S}_4$ . It is clear that both forms are irrational. Suppose that $ \unicode{x3bb} _1 {\mathbf q}_{\beta }+ \unicode{x3bb} _2 {\mathbf l}_{\beta }^2$ is a rational quadratic form. By considering the ratios of the coefficients of monomials $ x_4^2$ and $x_3x_4$ , and the term $x_1^2$ , we conclude that

must both be rational. This implies that $ \beta $ is rational, which is a contradiction. It is also clear that the restriction of ${\mathbf q}_{ \beta }$ to the kernel of ${\mathbf l}_{ \beta }$ is indefinite. This shows that the pair $( {\mathbf q}_{ \beta }, {\mathbf l}_{ \beta })$ is of type I.

Now, consider the quadratic form

$$ \begin{align*} {\mathbf q}^{\prime}_{\beta}= x_1^2+ x_2^2- \beta^2 x_3^2.\end{align*} $$

Given any $ \varepsilon>0$ and interval $I=(a, b) \subseteq {\mathbb {R}}$ , of [9, Theorem 2.2] provides a dense set of irrational values for $B \subseteq {\mathbb {R}}$ such that for every $ \beta \in B$ , there exists $c>0$ and a sequence $T_j \to \infty $ such that

$$ \begin{align*} \mathcal{N}_{{\mathbf q}, I} (T)> c T_j (\log T_j)^{1- \varepsilon} \end{align*} $$

holds for all $j \ge 1$ . Choose $ \beta \in (1/2,1)$ and $I= [ \beta ^{-1}, 2]$ . Then we can find a subset $L_j \subseteq {\mathbb {Z}}^3$ of cardinality at least $c T_j (\log T_j)^{1- \varepsilon } $ such that for every $x=(x_1, x_2, x_3) \in L_j$ , we have

$$ \begin{align*} x_1^2+ x_2^2 - \beta^2 x_3^2 \in [ \beta^{-1}, 2], \quad { x_1^2+ x_2^2 +x_2^2 \le T_j^2 }. \end{align*} $$

For every $(x_1, x_2, x_3) \in L_j$ , choose $x_4 \in {\mathbb {Z}}$ such that $ | \beta x_3+ x_4| \le 1$ . Note that this also implies that for j sufficiently large, we have

$$ \begin{align*} | x_4| \le 1+ |\ \beta x_3| \le 1+ \beta|T_j| \le |T_j|. \end{align*} $$

From here, we conclude the following inequalities:

(6.2) $$ \begin{align} \begin{aligned}{\mathbf q}_{ \beta} (x_1, x_2, x_3, x_4) &= q^{\prime}_{ \beta} (x_1, x_2, x_3) - ( \beta x_3+ x_4)^2 \in [-1, 2], \\ {\mathbf l}_{ \beta} (x_1, x_2, x_3, x_4) & = \beta x_3+ x_4 \in [-1, 1]. \end{aligned}\end{align} $$

Moreover,

$$ \begin{align*} \| (x_1, x_2, x_3, x_4) \| \le \sqrt{ 2} T_j. \end{align*} $$

Setting $I= [-1,2], J= [-1, 1]$ and adjusting the constant c slightly, the claim follows.

Forms of signature $(2,3)$ can be dealt with in a similar manner by considering the pair

$$ \begin{align*} {\mathbf q}_{\beta}(x_1, x_2, x_3, x_4,x_5) & = (x_1^2+x_2^2)- \beta (x_3^2+x_4^2) - ( \beta x_3+ \beta x_4+ x_5)^2, \\ {\mathbf l}_{\beta} (x_1, x_2, x_3, x_4,x_5) & = \beta x_3+ \beta x_4+ x_5. \end{align*} $$

We omit the details.