Local newforms for generic representations of unramified even unitary groups I: Even conductor case

Abstract

In this paper, we define compact open subgroups of quasi-split even unitary groups for each even non-negative integer and establish the theory of local newforms for irreducible tempered generic representations with a certain condition on the central characters. To do this, we use the local Gan–Gross–Prasad conjecture, the local Rankin–Selberg integrals and the local theta correspondence.

1 Introduction

In the 1970s, Atkin–Lehner [1] and Li [17] introduced the notion of newforms for elliptic modular forms and showed the multiplicity one theorem. Together with their results, Casselman’s theory of local newforms [5] is a bridge between modular forms and automorphic representations of $\mathrm {GL}_2/\mathbb {Q}$ . Since then, the theory of local newforms was developed for several groups. For example, for low rank cases, Roberts–Schmidt [22] and Lansky–Raghuram [16] established this theory for $\mathrm {GSp}_4$ and $\mathrm {U}(1,1)$ , respectively. Casselman’s result was extended to $\mathrm {GL}_n$ by Jacquet–Piatetski-Shapiro–Shalika [13] (see also [12]) and by Atobe–Kondo–Yasuda [2]. For other general rank cases,

• • Tsai [23] studied the local newforms of generic supercuspidal representations of $\mathrm {SO}_{2n+1}$ ; and

• • the author together with Oi and Yasuda [3] treated the case for unramified $\mathrm {U}_{2n+1}$ .

In this paper, for a bridge to hermitian modular forms, we try to establish the theory of local newforms for $\mathrm {U}(n,n)$ .

Let us describe our results. Let $E/F$ be an unramified quadratic extension of non-archimedean local fields of characteristic $0$ and of residue characteristic $p> 2$ . Fix a nontrivial additive character $\psi $ of F such that $\psi |_{\mathfrak {o}_F} = \mathbf {1}$ but $\psi |_{\mathfrak {p}_F^{-1}} \not = \mathbf {1}$ , and set $\psi _E(x) = \psi (\frac {x+\overline {x}}{2})$ for $x \in E$ . Consider a quasi-split unitary group of $2n$ variables given by

$$\begin{align*}\mathrm{U}_{2n} = \left\{ g \in \mathrm{GL}_{2n}(E) \;\middle|\; {}^t\overline{g} \begin{pmatrix} 0&w_n \\ -w_n &0 \end{pmatrix} g = \begin{pmatrix} 0&w_n \\ -w_n &0 \end{pmatrix} \right\} \end{align*}$$

with

$$\begin{align*}w_n = \begin{pmatrix} &&1 \\ &\unicode{x22F0}&\\ 1&& \end{pmatrix} \in \mathrm{GL}_n(E). \end{align*}$$

We denote by $W = E^{2n}$ the vector space where $\mathrm {U}_{2n}$ acts. The center of $\mathrm {U}_{2n}$ is identified with $E^1 = \{x \in E^\times \;|\; N_{E/F}(x) = 1\}$ . Define a compact subgroup $K_{2m}^W$ of $\mathrm {U}_{2n}$ by $K_{0}^W = \mathrm {U}_{2n} \cap \mathrm {GL}_{2n}(\mathfrak {o}_E)$ , and by

for $2m> 0$ . For an irreducible smooth representation $\pi $ of $\mathrm {U}_{2n}$ , we denote by $\pi _\psi $ the maximal quotient of $\pi $ on which the subgroup

$$\begin{align*}Z = \left\{ \begin{pmatrix} 1&0& z \\ 0&\mathbf{1}_{2n-2}&0\\ 0&0&1 \end{pmatrix} \in \mathrm{U}_{2n} \;\middle|\; z \in F \right\} \cong F \end{align*}$$

acts by $\psi $ . This is a local analogue of the Fourier–Jacobi expansions of hermitian modular forms and is called the Fourier–Jacobi module of $\pi $ . We write $\pi _\psi ^{K_{2m}^W}$ for the image of the subspace $\pi ^{K_{2m}^W}$ consisting of $K_{2m}^W$ -fixed vectors via the canonical surjection $\pi \twoheadrightarrow \pi _\psi $ .

The main theorem is stated as follows. For other notations, in particular for the notion of $\psi _E$ -generic, see Section 2 below.

Theorem 1.1 (Theorem 2.2).

Let $\pi $ be an irreducible tempered representation of $\mathrm {U}_{2n}$ with the L-parameter $\phi _\pi $ and the central character $\omega _\pi $ . We denote by $c(\phi _\pi )$ the conductor of $\phi _\pi $ .

1. (1) If $\pi $ is not $\psi _E$ -generic, then $\pi _{\psi }^{K_{2m}^W} = 0$ for any $2m \geq 0$ . Conversely, if $\pi $ is $\psi _E$ -generic, then there exists $2m \geq 0$ such that $\pi _{\psi }^{K_{2m}^W} \not = 0$ .

2. (2) Suppose that $\pi $ is $\psi _E$ -generic. If $2m < c(\phi _\pi )$ , then $\pi _{\psi }^{K_{2m}^W} = 0$ . If $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ , then

   $$\begin{align*}\dim_{\mathbb{C}}(\pi_{\psi}^{K_{2m}^W}) \leq 1. \end{align*}$$

3. (3) Set $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ . Suppose that $\pi $ is $\psi _E$ -generic and that $\omega _\pi $ is trivial on $E^1 \cap (1+\mathfrak {p}_E^m)$ . Then $\pi _\psi ^{K_{2m}^W} \not = 0$ .

If $2m = c(\phi _\pi )$ and if $\omega _\pi $ is trivial on $E^1 \cap (1+\mathfrak {p}_E^m)$ , we shall call an element in $\pi ^{K_{2m}^W}$ whose image in $\pi _\psi $ is nonzero a local newform of $\pi $ .

Remark 1.2.

1. (1) If $\pi ^{K_{2m}^W} \not = 0$ , then $\omega _\pi $ is trivial on $E^1 \cap (1+\mathfrak {p}_E^m)$ since $E^1 \cap (1+\mathfrak {p}_E^m) \subset K_{2m}^W$ .

2. (2) Even if $2m = c(\phi _\pi )$ or $c(\phi _\pi )+1$ , the dimension of $\pi ^{K_{2m}^W}$ can be greater than $1$ . A counterexample already appears in the case where $n=1$ , which was treated by Lansky and Raghuram. See [16, Theorem 4.2.1].

3. (3) As well as in [3], one might expect the existence of $K_m^W$ for all integers $m \geq 0$ such that Theorem 1.1 holds. Unfortunately, we do not know how to define $K_m^W$ for odd integers $m>0$ at this moment.

We expect that Theorem 1.1 has several applications such as a higher level generalization of a result of Chenevier–Renard [7]. We will try it as a next project.

A usual method to establish the theory of local newforms is to apply the Rankin–Selberg integrals, which are based on the multiplicity one theorem for several Gan–Gross–Prasad (GGP) pairs. For example, Tsai [23] and Cheng [8] used the pairs $(\mathrm {SO}_{2n+1}(F), \mathrm {SO}_{2n}(F))$ and $(\mathrm {U}_{2n+1},\mathrm {U}_{2n})$ to obtain knowledge about newforms. In this paper, we will also use this method as well. However, in our case, one needs the GGP pair $(\mathrm {U}_{2n}, \mathrm {U}_{2n-2})$ , which is not a ‘basic’ case. More precisely, we have to consider the restrictions of irreducible representations of $\mathrm {U}_{2n}$ to the Jacobi group. Since the Jacobi group is not reductive, several arguments in [23] would not work.

For example, to prove an analogue of Theorem 1.1 (1) in [23], Tsai used a lemma of Moy–Prasad ([23, Lemma 3.4.1]). We do not know whether this lemma can be extended to our case. Instead of this lemma, we use the local period integrals for the refined GGP conjecture. Using the absolutely convergence of these integrals, the argument of Gan–Savin [11, Lemma 12.5] can show Theorem 1.1 (1). See Section 3.2 below. This is the same idea as in the previous paper [3, Theorem 4.5].

The proof of Theorem 1.1 (2) is the same as usual. Namely, it is an application of the Rankin–Selberg integrals for $\mathrm {U}_{2n} \times \mathrm {GL}_{n-1}(E)$ . This theory in this case was established by Ben-Artzi–Soudry [4] and Morimoto [21], and is recalled in Theorem 4.2. Especially, the multiplicativity of the gamma factors is included in [21, Theorem 3.1]. Using the Rankin–Selberg integrals, we will define certain formal power series. Lemma 4.4 is a key computation to give lower bounds of the degrees. Using the functional equations of the Rankin–Selberg integrals, we would obtain an upper bound of the dimension of $\pi ^{K_{2m}^W}_\psi $ . However, since the Rankin–Selberg integrals for $\mathrm {U}_{2n} \times \mathrm {GL}_{n-1}(E)$ factors through $\pi \twoheadrightarrow \pi _\psi $ , we cannot estimate the dimension of $\pi ^{K_{2m}^W}$ itself.

For the proof of Theorem 1.1 (3), the fact that we have to deal with the Jacobi group complicates the situation. Indeed, the arguments in [23, Chapter 8] and in the previous paper [3, Theorem 4.3] might not work. In this paper, we give a new, or rather old, idea.

Recall that the theory of newforms was initiated by Atkin–Lehner [1] and Li [17] for elliptic modular forms of integral weights. Kohnen [14] established a similar theory to the half-integral weights case. Moreover, he proved that the newforms of integral weights and the ones of half-integral weights are related to each other by the Shimura correspondence. Since the theta correspondence is a generalization of the Shimura correspondence, the local newforms will be compatible with the local theta correspondence in the future. Instead, the local theta correspondence would be useful to show the existence of the local newforms. This is our idea.

In fact, if we let $\sigma = \theta _\psi (\pi )$ be the theta lift of $\pi $ to $\mathrm {U}_{2n+1}$ , then $\sigma $ is nonzero irreducible tempered and generic, and its conductor and central character are the same as the ones of $\pi $ . By the definition of the theta lifting, we have a surjective $\mathrm {U}_{2n+1} \times \mathrm {U}_{2n}$ -equivariant map

$$\begin{align*}\omega_\psi \rightarrow \sigma \boxtimes \pi, \end{align*}$$

where $\omega _\psi $ is the Weil representation of $\mathrm {U}_{2n+1} \times \mathrm {U}_{2n}$ . Let $K_{2m}^V$ be a conjugate of the compact subgroup of $\mathrm {U}_{2n+1}$ defined in [3], where $V = E^{2n+1}$ is the vector space on which $\mathrm {U}_{2n+1}$ acts. Set $J_{2m}^V$ to be the subgroup of $\mathrm {U}_{2n+1}$ generated by $K_{2m}^V$ and the central subgroup $E^1 \cap (1+\mathfrak {p}_E^m)$ . Then by using a lattice model and Waldspurger’s result (Proposition 5.3), one can show that $\omega _\psi ^{J_{2m}^V}$ is generated by $\omega _\psi ^{J_{2m}^V \times K_{2m}^W}$ as a representation of $\mathrm {U}_{2n}$ . Hence, if $2m \geq c(\phi _\pi )$ and $\omega _\pi |_{E^1 \cap (1+\mathfrak {p}_E^m)} = \mathbf {1}$ , then $\pi ^{K_{2m}^W} \not = 0$ since $\sigma ^{J_{2m}^V} \not = 0$ . See Proposition 5.6 for the details.

However, it is much harder to show $\pi ^{K_{2m}^W}_\psi \not = 0$ when $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ . Let $l_\sigma \colon \sigma \rightarrow \mathbb {C}$ be a nonzero Whittaker functional. Then the composition

$$\begin{align*}\omega_\psi \rightarrow \sigma \boxtimes \pi \xrightarrow{l_\sigma \otimes \mathrm{id}} \pi \end{align*}$$

factors through a twisted Jacquet module of $\omega _\psi $ along a maximal unipotent subgroup of $\mathrm {U}_{2n+1}$ . By the same argument as Mao–Rallis [18, Proposition 2.3], this twisted Jacquet module is isomorphic to the compact induction $\mathrm {ind}_{N_{2n}'}^{\mathrm {U}_{2n}}(\mu )$ , where $N_{2n}'$ is a maximal unipotent subgroup of $\mathrm {U}_{2n}$ and $\mu $ is a generic character of $N_{2n}'$ . By Cheng’s result [8, Theorem 1.4, Lemma 7.5], $l_\sigma $ is nonzero on the one-dimensional subspace $\sigma ^{J_{2m}^V}$ if $l_\sigma $ is suitably chosen. Hence, there is $\phi \in \omega _\psi ^{J_{2m}^V \times K_{2m}^W}$ such that it is nonzero under the all maps in the following diagram:

Lemma 5.7 asserts that the support of the image of $\phi $ in $\mathrm {ind}_{N_{2n}'}^{\mathrm {U}_{2n}}(\mu )$ is small enough. It implies that $\pi ^{K_{2m}^W}_\psi \not = 0$ immediately. See Section 5.5 for the details. Finally, to prove Lemma 5.7, we need to change models of the Weil representation and review the argument of Mao–Rallis [18, Proposition 2.3].

This paper is organized as follows. In Section 2, we introduce several notations and state our main theorem. Using the local Fourier–Jacobi periods, we show Theorem 1.1 (1) in Section 3. Theorem 1.1 (2) is obtained as an application of the Rankin–Selberg integrals in Section 4. Finally, we study theta liftings to prove Theorem 1.1 (3) in Section 5.

Notation

Let $E/F$ be an unramified quadratic extension of non-archimedean local fields of characteristic $0$ and of residue characteristic $p> 2$ . The nontrivial element in $\mathrm {Gal}(E/F)$ is denoted by $x \mapsto \overline {x}$ . Set $\mathfrak {o}_E$ (resp. $\mathfrak {o}_F$ ) to be the ring of integers of E (resp. F), and $\mathfrak {p}_E$ (resp. $\mathfrak {p}_F$ ) to be its maximal ideal. Let $E^1 = \{x \in E^\times \;|\; x\overline {x} = 1\}$ denote the kernel of the norm map $N_{E/F} \colon E^\times \rightarrow F^\times $ . Fix a uniformizer $\varpi $ of F, which is also a uniformizer of E. When $x \in E^\times $ can be written as $x = u \varpi ^l$ for some $u \in \mathfrak {o}_E^\times $ , we write $\mathrm {ord}(x) = l$ . Set $q = |\mathfrak {o}_F/\mathfrak {p}_F|$ so that $q^2 = |\mathfrak {o}_E/\mathfrak {p}_E|$ . Let $|\cdot |_E$ be the normalized absolute value of E so that $|x|_E = q^{-2 \mathrm {ord}(x)}$ for $x \in E^\times $ .

We fix $\delta \in \mathfrak {o}_E^\times $ such that $\overline {\delta } = -\delta $ , and a nontrivial additive character $\psi \colon F \rightarrow \mathbb {C}^\times $ such that $\psi |_{\mathfrak {o}_F} = \mathbf {1}$ but $\psi |_{\mathfrak {p}_F^{-1}} \not = \mathbf {1}$ . Set $\psi _E(x) = \psi (\frac {1}{2}\mathrm {tr}_{E/F}(x)) = \psi (\frac {x+\overline {x}}{2})$ and $\psi _E^\delta (x) = \psi _E(x/\delta )$ . Then $\psi _E$ and $\psi _E^\delta $ are nontrivial additive characters of E such that $\psi _E|_F = \psi $ and $\psi _E^\delta |_F = \mathbf {1}$ . The unique nontrivial quadratic unramified character of $E^\times $ is denoted by $\chi $ . Namely, $\chi |_{\mathfrak {o}_E^\times } = \mathbf {1}$ and $\chi (\varpi ) = -1$ . In particular, if we write $\chi = |\cdot |_E^{s_0}$ , we have $q^{-2s_0} = -1$ .

A representation $\pi $ of a p-adic group G means a smooth representation over a complex vector space. When K is a compact open subgroup of G, we write $\pi ^K$ for the subspace of $\pi $ consisting of K-fixed vectors. Let $\mathrm {Irr}(G)$ be the set of equivalence classes of irreducible representations of G, and $\mathrm {Irr}_{\mathrm {temp}}(G)$ be its subset consisting of tempered representations.

2 Statement of the main theorem

In this section, we define families of compact open subgroups of unitary groups, and we state our main theorem.

2.1 Unitary groups

Let $V = V_{2n+1}$ (resp. $W = W_{2n}$ ) be a hermitian (resp. skew-hermitian) space over E of dimension $2n+1$ (resp. $2n$ ) equipped with a nondegenerate hermitian form $\left \langle \cdot , \cdot \right \rangle _V$ (resp. skew-hermitian form $\left \langle \cdot ,\cdot \right \rangle _W$ ). Assume that there are bases $\{e_n, \dots , e_1, e_0, e_{-1}, \dots , e_{-n}\}$ of V and $\{f_n,\dots ,f_1, f_{-1}, \dots , f_{-n}\}$ of W, respectively, such that

$$\begin{align*}\left\langle e_i, e_j \right\rangle_V = \left\langle f_i, f_j \right\rangle_W = 0 \end{align*}$$

unless $j = -i$ , and

$$\begin{align*}\left\langle e_0,e_0 \right\rangle_V = \left\langle e_i,e_{-i} \right\rangle_V = \left\langle f_i,f_{-i} \right\rangle_W = 1 \end{align*}$$

for $1 \leq i \leq n$ .

Using these bases, we often identify the associated unitary groups $\mathrm {U}(V)$ and $\mathrm {U}(W)$ with

$$ \begin{align*} \mathrm{U}_{2n+1} &= \left\{ h \in \mathrm{GL}_{2n+1}(E) \;\middle|\; {}^t\overline{h} w_{2n+1} h = w_{2n+1} \right\}, \\ \mathrm{U}_{2n} &= \left\{ g \in \mathrm{GL}_{2n}(E) \;\middle|\; {}^t\overline{g} J_{2n} g = J_{2n} \right\}, \end{align*} $$

respectively, where we set

$$\begin{align*}w_n = \begin{pmatrix} &&1\\ &\unicode{x22F0}&\\ 1&& \end{pmatrix} \in \mathrm{GL}_n(E), \quad J_{2n} = \begin{pmatrix} 0&w_n\\ -w_n&0 \end{pmatrix} \in \mathrm{GL}_{2n}(E). \end{align*}$$

2.2 Representations of unitary groups

Let $N_{2n+1}$ (resp. $N_{2n}$ ) be the upper triangular unipotent subgroup of $\mathrm {U}_{2n+1}$ (resp. $\mathrm {U}_{2n}$ ). We define generic characters of $N_{2n+1}$ and $N_{2n}$ by the same formula

$$\begin{align*}u \mapsto \psi_E\left(\sum_{k=1}^{n}u_{k,k+1}\right). \end{align*}$$

By abuse of notation, we denote these characters by $\psi _E$ . We say that an irreducible representation $\sigma $ of $\mathrm {U}_{2n+1}$ (resp. $\pi $ of $\mathrm {U}_{2n}$ ) is generic (resp. $\psi _E$ -generic) if $\mathrm {Hom}_{N_{2n+1}}(\sigma , \psi _E) \not = 0$ (resp. $\mathrm {Hom}_{N_{2n}}(\pi , \psi _E) \not = 0$ ).

For an irreducible representation $\pi $ of $\mathrm {U}_{2n}$ , we denote by $\pi ^\vee $ the contragredient representation of $\pi $ . By a result in [19, Chapter 4. II. 1], we know $\pi ^\vee \cong \pi ^\theta $ , where $\pi ^\theta (g) = \pi (\theta (g))$ with

$$\begin{align*}\theta \colon \mathrm{U}_{2n} \rightarrow \mathrm{U}_{2n},\; g \mapsto \begin{pmatrix} \mathbf{1}_n & 0 \\ 0 & -\mathbf{1}_n \end{pmatrix} \overline{g} \begin{pmatrix} \mathbf{1}_n & 0 \\ 0 & -\mathbf{1}_n \end{pmatrix}^{-1}. \end{align*}$$

In particular, $\pi $ is $\psi _E$ -generic if and only if $\pi ^\vee $ is $\psi _E^{-1}$ -generic.

By the local Langlands correspondence established by Mok [20], to an irreducible representation $\sigma $ of $\mathrm {U}_{2n+1}$ (resp. $\pi $ of $\mathrm {U}_{2n}$ ), one can attach a conjugate self-dual representation $\phi _\sigma $ (resp. $\phi _\pi $ ) of $W_E \times \mathrm {SL}_2(\mathbb {C})$ of dimension $2n+1$ (resp. $2n$ ), where $W_E$ is the Weil group of E. We call $\phi _\sigma $ (resp. $\phi _\pi $ ) the L-parameter for $\sigma $ (resp. $\pi $ ). Then we define the conductor $c(\phi _\sigma )$ of $\phi _\sigma $ by the non-negative integer satisfying

$$\begin{align*}\varepsilon(s,\phi_\sigma,\psi_E) = \varepsilon(0,\phi_\sigma,\psi_E)q^{-2c(\phi_\sigma)s}. \end{align*}$$

Similarly, the conductor $c(\phi _\pi )$ of $\phi _\pi $ is defined.

The center of $\mathrm {U}_{2n+1}$ (resp. $\mathrm {U}_{2n}$ ) is $\mathrm {U}_1$ which is identified with $E^1$ . For an irreducible representation $\sigma $ (resp. $\pi $ ) of $\mathrm {U}_{2n+1}$ (resp. $\mathrm {U}_{2n}$ ), we denote its central character by $\omega _\sigma $ (resp. $\omega _\pi $ ). If $\sigma $ (resp. $\pi $ ) corresponds to $\phi _\sigma $ (resp. $\phi _\pi $ ), then the L-parameter of $\omega _\sigma $ (resp. $\omega _\pi $ ) is given by $\det (\phi _\sigma )$ (resp. $\det (\phi _\pi )$ ).

2.3 Jacobi group

Set

$$\begin{align*}\mathbf{v}(x,y;z) = \begin{pmatrix} 1 & x & y & z+\frac{1}{2}(x w_{n-1}{}^t\overline{y} - y w_{n-1} {}^t\overline{x})\\ 0 & \mathbf{1}_{n-1} & 0 & w_{n-1}{}^t\overline{y} \\ 0 & 0& \mathbf{1}_{n-1} & -w_{n-1}{}^t\overline{x} \\ 0&0&0& 1 \end{pmatrix} \in \mathrm{U}_{2n} \end{align*}$$

for $x,y \in E^{n-1}$ and $z \in F$ . Here, $E^{n-1}$ is the space of row vectors. Let $H_{n-1} = \{\mathbf {v}(x,y; z) \;|\; x,y \in E^{n-1}, z \in F\} \cong E^{2n-2} \oplus F$ be a Heisenberg group in $4n-3$ variables over F with the multiplication law

$$\begin{align*}\mathbf{v}(x,y;z) \mathbf{v}(x',y';z') = \mathbf{v}\left(x+x', y+y'; z+z'+\frac{1}{2}\mathrm{tr}_{E/F}(x w_{n-1}{}^t\overline{y} - y w_{n-1} {}^t\overline{x}) \right). \end{align*}$$

We write

$$ \begin{align*} X_{n-1} &= \{\mathbf{v}(x,0;0) \;|\; x \in E^{n-1}\}, \\ Y_{n-1} &= \{\mathbf{v}(0,y;0) \;|\; y \in E^{n-1}\}, \\ Z &= \{\mathbf{v}(0,0;z) \;|\; z \in F\}. \end{align*} $$

By abuse of notation, we denote the character $Z \ni \mathbf {v}(0,0;z) \mapsto \psi (z)$ by $\psi $ .

We identify $\mathrm {U}_{2n-2}$ as a subgroup of $\mathrm {U}_{2n}$ by the inclusion

$$\begin{align*}\mathrm{U}_{2n-2} \ni g' \mapsto \begin{pmatrix} 1 && \\ &g'& \\ &&1 \end{pmatrix} \in \mathrm{U}_{2n}. \end{align*}$$

Then $\mathrm {U}_{2n-2}$ normalizes $H_{n-1}$ . We call $J_{n-1} = H_{n-1} \rtimes \mathrm {U}_{2n-2}$ the Jacobi group. Note that Z is the center of $J_{n-1}$ .

For an irreducible representation $\pi $ of $\mathrm {U}_{2n}$ , we denote by $\pi _{\psi }$ the maximal quotient of $\pi $ on which Z acts by $\psi $ . We call $\pi _\psi $ the Fourier–Jacobi module of $\pi $ . For a compact open subgroup K of $\mathrm {U}_{2n}$ , we denote by $\pi ^K_{\psi }$ the image of $\pi ^K$ via the canonical surjection $\pi \twoheadrightarrow \pi _{\psi }$ . Note that $\pi _{\psi }$ is a smooth representation of $J_{n-1}$ so that K does not act on $\pi _{\psi }$ itself.

For $t \in E^\times $ , if we put $\psi '(x) = \psi (N_{E/F}(t)x)$ and

$$\begin{align*}K' = \begin{pmatrix} t && \\ &\mathbf{1}_{2n-2}& \\ && \overline{t}^{-1} \end{pmatrix}^{-1} K \begin{pmatrix} t && \\ &\mathbf{1}_{2n-2}& \\ && \overline{t}^{-1} \end{pmatrix}, \end{align*}$$

then $\pi (\mathrm {diag}(t,\mathbf {1}_{2n-2}, \overline {t}^{-1}))$ induces isomorphisms

$$\begin{align*}\pi^{K'} \xrightarrow{\sim} \pi^K,\quad \pi_{\psi'} \xrightarrow{\sim} \pi_\psi. \end{align*}$$

Hence, we have $\pi _{\psi '}^{K'} \cong \pi _{\psi }^K$ .

2.4 Compact subgroups

For each non-negative even integer $2m \geq 0$ , we define compact subgroups $K_{2m}^V \subset \mathrm {U}(V) \cong \mathrm {U}_{2n+1}$ and $K_{2m}^W \subset \mathrm {U}(W) \cong \mathrm {U}_{2n}$ as follows. When $2m = 0$ , we set $K_0^V = \mathrm {U}_{2n+1} \cap \mathrm {GL}_{2n+1}(\mathfrak {o}_E)$ and $K_0^W = \mathrm {U}_{2n} \cap \mathrm {GL}_{2n}(\mathfrak {o}_E)$ . If $2m> 0$ , we set

Note that

which is denoted by $\mathbb {K}_{2m,\mathrm {U}(V)}$ in [3], and by $K_{n,2m}$ in [8]. If we set ${}^tK_{2m}^W = \{{}^tk \;|\; k \in K_{2m}^W\}$ to be the transpose of $K_{2m}^W$ , then

$$ \begin{align*} K_{2m}^W = \begin{pmatrix} \varpi^{-m} && \\ &\mathbf{1}_{2n-2}& \\ && \varpi^m \end{pmatrix} {}^tK_{2m}^W \begin{pmatrix} \varpi^{-m} && \\ &\mathbf{1}_{2n-2}& \\ && \varpi^m \end{pmatrix}^{-1}. \end{align*} $$

The theory of local newforms for $\mathrm {U}_{2n+1}$ is established by the author together with Oi and Yasuda [3, Theorem 1.1] and by Cheng [8, Theorem 1.2] as follows.

Theorem 2.1. Let $\sigma $ be an irreducible tempered representation of $\mathrm {U}_{2n+1}$ with the L-parameter $\phi _\sigma $ .

1. (1) If $\sigma $ is not generic, then $\sigma ^{K_{2m}^V} = 0$ for any $2m \geq 0$ .

2. (2) If $\sigma $ is generic, then

   $$\begin{align*}\dim_{\mathbb{C}}(\sigma^{K_{2m}^V}) = \left\{ \begin{aligned} &0 &\quad&\text{if}\ \ 2m < c(\phi_\sigma), \\ &1 &\quad&\text{if}\ \ 2m = c(\phi_\sigma) \text{ or } c(\phi_\sigma)+1. \end{aligned} \right. \end{align*}$$
   Moreover, if $2m> c(\phi _\sigma )$ , then $\sigma ^{K_{2m}^V} \not = 0$ .

In this paper, we will prove an analogue of this theorem for $\mathrm {U}_{2n}$ as follows.

Theorem 2.2. Let $\pi $ be an irreducible tempered representation of $\mathrm {U}_{2n}$ with the L-parameter $\phi _\pi $ and the central character $\omega _\pi $ .

1. (1) If $\pi $ is not $\psi _E$ -generic, then $\pi _{\psi }^{K_{2m}^W} = 0$ for any $2m \geq 0$ . Conversely, if $\pi $ is $\psi _E$ -generic, then there exists $2m \geq 0$ such that $\pi _{\psi }^{K_{2m}^W} \not = 0$ .

2. (2) Suppose that $\pi $ is $\psi _E$ -generic. If $2m < c(\phi _\pi )$ , then $\pi _{\psi }^{K_{2m}^W} = 0$ . If $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ , then

   $$\begin{align*}\dim_{\mathbb{C}}(\pi_{\psi}^{K_{2m}^W}) \leq 1. \end{align*}$$

3. (3) Set $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ . Suppose that $\pi $ is $\psi _E$ -generic and that $\omega _\pi $ is trivial on $E^1 \cap (1+\mathfrak {p}_E^m)$ . Then $\pi _\psi ^{K_{2m}^W} \not = 0$ .

When $2m = c(\phi _\pi )$ , we shall call an element in $\pi ^{K_{2m}^W}$ whose image in $\pi _\psi $ is nonzero a local newform of $\pi $ .

3 Local Fourier–Jacobi periods

In this section, we will prove Theorem 2.2 (1). To do this, we use the local Gan–Gross–Prasad conjecture for $(\mathrm {U}_{2n}, \mathrm {U}_{2n-2})$ .

3.1 Weil representation

Let $W_0$ be the subspace of W generated by $\{f_{n-1},\dots ,f_1, f_{-1}, \dots , f_{-n+1}\}$ . We write $G_n = \mathrm {U}(W)$ and $G_{n-1} = \mathrm {U}(W_0)$ in this section. Hence, the Jacobi group $J_{n-1}$ is written as $J_{n-1} = H_{n-1} \rtimes G_{n-1}$ .

Recall that we have a compact subgroup $K_{2m}^W$ of $G_n = \mathrm {U}(W)$ . Note that the intersections

$$\begin{align*}K^J = K_{2m}^{W} \cap J_{n-1}, \quad K^H = K_{2m}^{W} \cap H_{n-1}, \quad K^{W_0} = K_{2m}^{W} \cap \mathrm{U}(W_0) \end{align*}$$

are independent of $2m$ . Moreover, $K^{W_0}$ is a hyperspecial maximal compact subgroup of $G_{n-1} = \mathrm {U}(W_0)$ .

We consider the Weil representation $\omega _\psi $ of $J_{n-1}$ associated to $\psi $ and $\chi $ . It is realized on the Schwartz space $\mathcal {S}(E^{n-1})$ as follows. For $\phi \in \mathcal {S}(E^{n-1})$ and $\xi \in E^{n-1}$ ,

$$ \begin{align*} &\omega_\psi(\mathbf{v}(x,0;0))\phi(\xi) = \phi(\xi+x), \quad x \in E^{n-1}, \\ &\omega_\psi(\mathbf{v}(0,y;0))\phi(\xi) = \psi_E(2\xi w_{n-1} {}^t\overline{y})\phi(\xi), \quad y \in E^{n-1}, \\ &\omega_\psi(\mathbf{v}(0,0;z))\phi(\xi) = \psi(z) \phi(\xi), \quad z \in F, \\ &\omega_\psi(\mathbf{m}(a))\phi(\xi) = \chi(\det(a)) |\det(a)|^{\frac{1}{2}} \phi(\xi a), \quad a \in \mathrm{GL}_{n-1}(E), \\ &\omega_\psi(\mathbf{n}(b))\phi(\xi) = \psi_E\left(\overline{\xi} \overline{b} w_{n-1} {}^t \xi \right)\phi(\xi), \quad b \in \mathrm{M}_{n-1}(E), {}^t(w_{n-1}\overline{b}) = w_{n-1}b, \\ &\omega_\psi (J_{2n-2})\phi(\xi) = \int_{E^{n-1}} \phi(x) \psi_E(2\overline{x} \cdot {}^t\xi) dx, \end{align*} $$

where we set

$$\begin{align*}\mathbf{m}(a) = \begin{pmatrix} a & 0 \\ 0 & w_{n-1}{}^t\overline{a}^{-1}w_{n-1}^{-1} \end{pmatrix}, \quad \mathbf{n}(b) = \begin{pmatrix} 1 & b \\ 0 & 1 \end{pmatrix} \in G_{n-1}, \end{align*}$$

and the measure $dx$ on $E^{n-1}$ is the self-dual Haar measure with respect to $\psi _E$ . The Weil representation $\omega _\psi $ is unitary with respect to the pairing

$$\begin{align*}(\phi_1,\phi_2) = \int_{E^{n-1}}\phi_1(\xi) \overline{\phi_2(\xi)} d\xi. \end{align*}$$

Set $\phi _0 \in \mathcal {S}(E^{n-1})$ to be the characteristic function on $\mathfrak {o}_E^{n-1}$ . Note that $\phi_0 $ is fixed by $\omega _{\psi }(K^J)$ . Moreover, the subspace $\omega _{\psi }^{K^H}$ is one-dimensional spanned by $\phi _0$ .

3.2 Proof of Theorem 2.2 (1)

Let $\pi \in \mathrm {Irr}_{\mathrm {temp}}(G_n)$ and $\pi ' \in \mathrm {Irr}_{\mathrm {temp}}(G_{n-1})$ . Fix a nonzero $G_n$ -invariant (resp. $G_{n-1}$ -invariant) bilinear pairing $(\cdot , \cdot )_{\pi } \colon \pi \times \pi ^\vee \rightarrow \mathbb {C}$ (resp. $(\cdot , \cdot )_{\pi '} \colon \pi ' \times \pi ^{\prime \vee } \rightarrow \mathbb {C}$ ). For $\varphi \in \pi , \varphi ^\vee \in \pi ^\vee , \varphi ' \in \pi ', \varphi ^{\prime \vee } \in \pi ^{\prime \vee }$ and $\phi , \phi ^\vee \in \mathcal {S}(E^{n-1})$ , we define the local Fourier–Jacobi period by

$$ \begin{align*} &\alpha(\varphi, \varphi^\vee, \varphi', \varphi^{\prime\vee}, \phi, \phi^\vee) \\&= \int_{G_{n-1}}\int_{H_{n-1}} (\pi(hg)\varphi, \varphi^\vee)_{\pi} (\pi'(g)\varphi', \varphi^{\prime\vee})_{\pi'} \overline{(\omega_\psi(hg) \phi, \phi^\vee)} dhdg. \end{align*} $$

Proposition 3.1. The integral $\alpha (\varphi , \varphi ^\vee , \varphi ', \varphi ^{\prime \vee }, \phi , \phi ^\vee )$ is absolutely convergent.

Proof. This is exactly the same as the symplectic-metaplectic case ([25, Proposition 2.2.1]). We omit the details.

Since the central character of $\omega _\psi $ is $\psi $ , if $\alpha (\varphi , \varphi ^\vee , \varphi ', \varphi ^{\prime \vee }, \phi , \phi ^\vee ) \not = 0$ , then

$$\begin{align*}\int_{F}(\pi(hg \cdot \mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \not= 0 \end{align*}$$

for some $h \in H_{n-1}$ and $g \in G_{n-1}$ . This means that the image of $\varphi $ in $\pi _\psi $ is nonzero. The converse holds in the following sense.

Lemma 3.2. Let $\varphi \in \pi $ . Assume that the image of $\varphi $ in $\pi _\psi $ is nonzero. Then there exists $\varphi ^\vee \in \pi ^\vee $ such that

$$\begin{align*}\int_{F}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \not= 0. \end{align*}$$

Proof. Note that by Proposition 3.1, the integral

$$\begin{align*}\int_{F}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \end{align*}$$

converges absolutely. Suppose that this integral is equal to zero for all $\varphi ^\vee \in \pi ^\vee $ . We will show that the image of $\varphi $ in $\pi _\psi $ is zero.

For an integer $j> 0$ , set

$$\begin{align*}T_j = \left\{ t(1+a) = \begin{pmatrix} 1+a && \\ &\mathbf{1}_{2n-2}& \\ && (1+a)^{-1} \end{pmatrix} \;\middle|\; a \in \mathfrak{p}_F^j \right\} \subset \mathrm{U}_{2n}. \end{align*}$$

Recall that $\psi |_{\mathfrak {o}_F} = \mathbf {1}$ but $\psi |_{\mathfrak {p}_F^{-1}} \not = \mathbf {1}$ . Hence, for fixed $z \in F$ with $-k = \mathrm {ord}(z)$ , the map

$$\begin{align*}T_j \ni t(1+a) \mapsto \frac{\psi((1+a)^2z)}{\psi(z)} \in \mathbb{C}^\times \end{align*}$$

is a character if $k \leq 2j$ . Moreover, it is trivial if $k \leq j$ . Hence,

$$\begin{align*}q^{-j} \int_{\mathfrak{p}_F^{j}} \psi((1+a)^2z) da = \left\{ \begin{aligned} &\psi(z) &\quad&\text{if } k \leq j, \\ &0 &\quad&\text{if } j < k \leq 2j. \end{aligned} \right. \end{align*}$$

Since $\pi $ is smooth, there is an integer $j> 0$ such that $\varphi $ is $T_j$ -fixed. To show that the image of $\varphi $ in $\pi _\psi $ is zero, it suffices to prove that

$$\begin{align*}\int_{\mathfrak{p}_F^{-j}} \pi(\mathbf{v}(0,0;z)\varphi) \overline{\psi(z)} dz = 0. \end{align*}$$

This is equivalent to saying that

(†) $$ \begin{align} \int_{\mathfrak{p}_F^{-j}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz = 0 \end{align} $$

for all $\varphi ^\vee \in \pi ^\vee $ . We claim that we may assume that $\varphi ^\vee $ is $T_j$ -fixed. Indeed, if $z \in \mathfrak {p}_F^{-j}$ , since $k = -\mathrm {ord}(z) \leq j$ , we have

$$ \begin{align*} \int_{\mathfrak{p}_F^{-j}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz &= \int_{\mathfrak{p}_F^{-j}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \left(q^{-j}\int_{\mathfrak{p}_F^j}\overline{\psi((1+a)^2z)} da\right)dz \\&= q^{-j} \int_{\mathfrak{p}_F^{-j}} \int_{\mathfrak{p}_F^j} (\pi(t(1+a)^{-1}\mathbf{v}(0,0;z)t(1+a)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} da dz \\&= q^{-j} \int_{\mathfrak{p}_F^{-j}} \int_{\mathfrak{p}_F^j} (\pi(\mathbf{v}(0,0;z)) \varphi, \pi^\vee(t(1+a))\varphi^\vee)_\pi \overline{\psi(z)} da dz. \end{align*} $$

Hence, $(\dagger )$ holds for $\varphi ^\vee $ if it holds for

$$\begin{align*}q^{-j} \int_{\mathfrak{p}_F^j} \pi^\vee(t(1+a))\varphi^\vee da \end{align*}$$

which is $T_j$ -fixed.

Now assume that $\varphi ^\vee $ is $T_j$ -fixed. Then we claim that

$$\begin{align*}\int_{\mathfrak{p}_F^{-j}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz = \int_{F}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz, \end{align*}$$

and hence, the left-hand side is zero by assumption. Indeed, for $k> j > 0$ , since $k \geq 2$ so that $k-1 < k \leq 2(k-1)$ , we have

$$ \begin{align*} &\int_{\mathfrak{p}_F^{-k} \setminus \mathfrak{p}_F^{-k+1}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \\&= q^{-k+1} \int_{\mathfrak{p}_F^{-k} \setminus \mathfrak{p}_F^{-k+1}} \int_{\mathfrak{p}_F^{k-1}} (\pi(t(1+a)^{-1}\mathbf{v}(0,0;z)t(1+a))\varphi, \varphi^\vee)_\pi \overline{\psi(z)} da dz \\&= \int_{\mathfrak{p}_F^{-k} \setminus \mathfrak{p}_F^{-k+1}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \left(q^{-k+1}\int_{\mathfrak{p}_F^{k-1}} \overline{\psi((1+a)^2z)} da \right)dz \\&=0. \end{align*} $$

This completes the proof of the lemma.

Now, we prove Theorem 2.2 (1).

Proof of Theorem 2.2 (1).

Let $\pi $ be an irreducible tempered representation of $G_n = \mathrm {U}_{2n}$ . Suppose that $\pi _{\psi }^{K_{2m}^W} \not = 0$ for some $2m \geq 0$ . We will show that $\pi $ must be $\psi _E$ -generic.

Fix $\varphi \in \pi ^{K_{2m}^W}$ such that the image of $\pi _{\psi }$ is nonzero. By Lemma 3.2, one can find $\varphi ^\vee \in \pi ^\vee $ such that

$$\begin{align*}\int_{F}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \not= 0. \end{align*}$$

Since Z is the center of $H_{n-1}$ , we may assume that $\varphi ^\vee $ is fixed by $K^H$ . Hence, the matrix coefficient $H_{n-1} \ni h \mapsto (\pi (h) \varphi , \varphi ^\vee )_\pi $ is bi- $K^H$ -invariant. Since $\omega _{\psi }$ is the unique irreducible representation of $H_{n-1}$ whose central character is $\psi $ , there are $\phi , \phi ^\vee \in \mathcal {S}(E^{n-1})$ such that

$$\begin{align*}\int_{H_{n-1}} (\pi(h) \varphi, \varphi^\vee)_\pi \overline{(\omega_{\psi}(h)\phi, \phi^\vee)} dh \not= 0. \end{align*}$$

We may also assume that both $\phi $ and $\phi ^\vee $ are fixed by $K^H$ . Since $\omega _{\psi }^{K^H} = \mathbb {C} \phi _0$ , we can take $\phi = \phi ^\vee = \phi _0$ . Hence,

$$\begin{align*}\int_{H_{n-1}} (\pi(h) \varphi, \varphi^\vee)_\pi \overline{(\omega_{\psi}(h)\phi_0, \phi_0)} dh \not= 0. \end{align*}$$

Now by applying the same argument as [11, Lemma 12.5] to the integral on $G_{n-1}$ , one can find $\pi ' \in \mathrm {Irr}_{\mathrm {temp}}(G_{n-1})$ and $(\varphi ', \varphi ^{\prime \vee }) \in \pi ' \times \pi ^{\prime \vee }$ such that

$$\begin{align*}\alpha(\varphi, \varphi^\vee, \varphi', \varphi^{\prime\vee}, \phi_0, \phi_0) \not= 0. \end{align*}$$

We may assume that $\varphi '$ is fixed by $K^{W_0}$ since so are $\varphi $ and $\phi _0$ . This means that $\pi '$ is unramified. By the local Gan–Gross–Prasad conjecture ([9, Conjecture 17.3, Theorem 19.1]), whose basic case is proven by Gan–Ichino [10, Theorem 1.3], we can deduce that $\pi $ is $\psi _E$ -generic.

Conversely, if $\pi $ is $\psi _E$ -generic, by the local Gan–Gross–Prasad conjecture, one can find an irreducible tempered unramified representation $\pi '$ of $G_{n-1}$ such that $\mathrm {Hom}_{J_{n-1}}(\pi \otimes \pi ' \otimes \overline {\omega _{\psi }}, \mathbb {C}) \not = 0$ . Since $\pi '$ and $\omega _\psi $ are irreducible as representations of $G_{n-1}$ and $H_{n-1}$ , respectively, for any nonzero unramified vector $\varphi ^{\prime }_0 \in \pi '$ and for any nonzero element $\mathcal {L} \in \mathrm {Hom}_{J_{n-1}}(\pi \otimes \pi ' \otimes \overline {\omega _{\psi }}, \mathbb {C})$ , one can take $\varphi \in \pi $ such that $\mathcal {L}(\varphi \otimes \varphi _0' \otimes \overline {\phi _0}) \not = 0$ . We may assume that $\varphi $ is fixed by $K^{J}$ . Since $\pi $ is smooth, $\varphi $ is fixed by $K_{2m}^{W}$ for $2m \gg 0$ . In this case, $\varphi $ gives a nonzero element in $\pi _\psi ^{K_{2m}^W}$ .

This completes the proof of Theorem 2.2 (1).

Recall in [9, Corollary 16.3] that for $\pi \in \mathrm {Irr}(G_n)$ and $\pi ' \in \mathrm {Irr}(G_{n-1})$ , we have

$$\begin{align*}\dim_{\mathbb{C}} \mathrm{Hom}_{J_{n-1}}(\pi \otimes \pi' \otimes \overline{\omega_\psi}, \mathbb{C}) \leq 1. \end{align*}$$

It is worth to state the following result which was obtained by the above argument.

Proposition 3.3. Let $\pi $ be an irreducible tempered representation of $G_n$ . Suppose that there is $\varphi \in \pi ^{K_{2m}^W}$ whose image in $\pi _{\psi }$ is nonzero for some $2m \geq 0$ . Then there exists an irreducible tempered unramified representation $\pi '$ of $G_{n-1}$ together with an unramified vector $\varphi ^{\prime }_0 \in \pi '$ such that $\mathcal {L}(\varphi \otimes \varphi ^{\prime }_0 \otimes \overline {\phi _0}) \not = 0$ for any nonzero $\mathcal {L} \in \mathrm {Hom}_{J_{n-1}}(\pi \otimes \pi ' \otimes \overline {\omega _{\psi }}, \mathbb {C})$ .

4 Uniqueness

In this section, we will prove Theorem 2.2 (2). As usual, this is an application of Rankin–Selberg integrals.

4.1 Rankin–Selberg integrals

Let $\tau $ be an irreducible generic representation of $\mathrm {GL}_{n-1}(E)$ which is realized on the Whittaker space $\mathcal {W}(\tau ,\psi _E^{-1})$ with respect to the inverse of $\psi _E$ . For $s \in \mathbb {C}$ , we consider the normalized parabolically induced representation

$$\begin{align*}\mathrm{Ind}_{Q_{n-1}}^{G_{n-1}}\left(\tau |\det|^{s-\frac{1}{2}}\right) \end{align*}$$

of $G_{n-1}$ , where $Q_{n-1} = M_{n-1}U_{n-1}$ denotes the standard Siegel parabolic subgroup so that

$$ \begin{align*} M_{n-1} &= \{\mathbf{m}(a) \;|\; a \in \mathrm{GL}_{n-1}(E)\}, \\ U_{n-1} &= \{\mathbf{n}(b) \;|\; b \in \mathrm{M}_{n-1}(E), {}^t(w_{n-1}\overline{b}) = w_{n-1}b\}. \end{align*} $$

We realize it on the space $V_{Q_{n-1}}^{G_{n-1}}(\mathcal {W}(\tau ,\psi _E^{-1}), s)$ of smooth functions $f_s \colon G_{n-1} \times \mathrm {GL}_{n-1}(E) \rightarrow \mathbb {C}$ such that

• • $f_s(\mathbf {n}(b) \mathbf {m}(a) g,a') = |\det a|_E^{s+\frac {n}{2}-1} f_s(g,a'a)$ for $g \in G_{n-1}$ , $a,a' \in \mathrm {GL}_{n-1}(E)$ and $\mathbf {n}(b) \in U_{n-1}$ ;

• • the function $a \mapsto f_s(g,a)$ belongs to $\mathcal {W}(\tau ,\psi _E^{-1})$ for any $g \in G_{n-1}$ .

Define a new representation $\tau ^*$ by $\tau ^*(a) = \tau (a^*)$ , where $a^* = w_{n-1}{}^t\overline {a}^{-1}w_{n-1}^{-1}$ . Note that $\tau ^* \cong \overline {\tau }^\vee $ , where $\overline {\tau }(a) = \tau (\overline {a})$ . As in [21, Section 2.3], one can define a normalized intertwining operator

$$\begin{align*}M^*(\tau,s) \colon V_{Q_{n-1}}^{G_{n-1}}(\mathcal{W}(\tau,\psi_E^{-1}), s) \rightarrow V_{Q_{n-1}}^{G_{n-1}}(\mathcal{W}(\tau^*,\psi_E^{-1}), 1-s). \end{align*}$$

Let $\pi $ be an irreducible $\psi _E$ -generic representation of $G_n$ realized on the Whittaker space $\mathcal {W}(\pi ,\psi _E)$ . For $W \in \mathcal {W}(\pi ,\psi _E)$ , $f_s \in V_{Q_{n-1}}^{G_{n-1}}(\mathcal {W}(\tau ,\psi _E^{-1}), s)$ and $\phi \in \mathcal {S}(E^{n-1})$ , we define the Rankin–Selberg integral $\mathcal {L}(W,f_s,\overline {\phi })$ by

$$\begin{align*}\int_{N_{n-1} \backslash G_{n-1}} \int_{E^{n-1}} W(w_{1,n-1} \mathbf{v}(x,0;0)g ) f_s(g,\mathbf{1}_{n-1}) \overline{\omega_\psi(g) \phi(x)} dx dg, \end{align*}$$

where we set

$$\begin{align*}w_{1,n-1} = \left(\begin{array}{cc|cc} &\mathbf{1}_{n-1}&&\\ 1&&&\\ \hline &&&1\\ &&\mathbf{1}_{n-1}& \end{array}\right) \in G_n. \end{align*}$$

Remark 4.1. Note that

$$\begin{align*}W(w_{1,n-1} \mathbf{v}(x,0;0)g \cdot \mathbf{v}(0,0;z)) = \psi(z)W(w_{1,n-1}\mathbf{v}(x,0;0)g) \end{align*}$$

for $W \in \mathcal {W}(\pi , \psi _E)$ . Hence, the restriction map $W \mapsto W(w_{1,n-1} \mathbf {v}(x,0;0)g)$ factors through $\pi \twoheadrightarrow \pi _\psi $ . In particular, if $\pi $ is $\psi _E$ -generic, then $\pi _\psi $ is nonzero.

Theorem 4.2. Keep the notations.

1. (1) The integral $\mathcal {L}(W,f_s,\overline {\phi })$ converges absolutely for $\mathrm {Re}(s) \gg 0$ . It is a rational function in $q^{-s}$ so that it admits a meromorphic continuation to the whole s-plane.

2. (2) Let $I(\pi \times \tau \times \chi )$ be the fractional ideal of $\mathbb {C}[q^{-s}, q^{s}]$ generated by $\mathcal {L}(W,f_s,\overline {\phi })$ for $W \in \mathcal {W}(\pi ,\psi _E)$ , $f_s \in V_{Q_{n-1}}^{G_{n-1}}(\mathcal {W}(\tau ,\psi _E^{-1}), s)$ and $\phi \in \mathcal {S}(E^{n-1})$ . Then there is a unique polynomial $P(X) \in \mathbb {C}[X]$ with $P(0)=1$ such that $I(\pi \times \tau \times \chi ) = (P(q^{-s})^{-1})$ . We define the L-function attached to $\pi \times \tau $ and $\chi $ by

   $$\begin{align*}L(s, \pi \times \tau, \chi) = P(q^{-s})^{-1}. \end{align*}$$

3. (3) There is a meromorphic function $\Gamma (s, \pi \times \tau , \psi )$ such that

   $$\begin{align*}\mathcal{L}(W, M^*(\tau, s)f_s, \overline{\phi}) = \omega_\pi(-1)^{n-1} \omega_\tau(-1)^n \Gamma(s, \pi \times \tau, \chi, \psi) \mathcal{L}(W,f_s,\overline{\phi}) \end{align*}$$
   for any $W \in \mathcal {W}(\pi ,\psi _E)$ , $f_s \in V_{Q_{n-1}}^{G_{n-1}}(\mathcal {W}(\tau ,\psi _E^{-1}), s)$ and $\phi \in \mathcal {S}(E^{n-1})$ . We call $\Gamma (s, \pi \times \tau , \chi , \psi )$ the gamma factor attached to $\pi \times \tau $ , $\chi $ and $\psi $ .

4. (4) The gamma factor $\Gamma (s, \pi \times \tau , \chi , \psi )$ satisfies several properties (including the multiplicativity), which determine $\Gamma (s, \pi \times \tau , \chi , \psi )$ uniquely.

5. (5) Define the $\varepsilon $ -factor attached to $\pi \times \tau $ , $\chi $ and $\psi $ by

   $$\begin{align*}\varepsilon(s, \pi \times \tau, \chi, \psi) = \Gamma(s, \pi \times \tau, \chi, \psi) \frac{L(s, \pi \times \tau, \chi)}{L(1-s, \pi^\vee \times \tau^\vee, \chi)}. \end{align*}$$

   Then it satisfies that

   $$\begin{align*}\varepsilon(1-s, \pi \times \tau^*, \chi, \psi)\varepsilon(s, \pi \times \tau, \chi, \psi) = 1. \end{align*}$$

   In particular, $\varepsilon (s,\pi \times \tau , \chi , \psi ) \in \mathbb {C}^\times (q^{-s})^{\mathbb {Z}}$ .

Proof. (1) is [4, Proposition 6.4]. By [4, Proposition 6.5], we see that $1 \in I(\pi \times \tau \times \chi )$ , which implies (2). The assertion (3) follows from the multiplicity one theorem proven in [9, Corollary 16.3]. (4) is proven by Morimoto [21, Theorem 3.1]. Since $M^*(\tau ^*, 1-s) \circ M^*(\tau , s) = \mathrm {id}$ , using $\omega _{\tau ^*}(-1) = \omega _{\tau }(-1)$ , we have

$$ \begin{align*} \mathcal{L}(W, f_s, \overline{\phi}) &= \mathcal{L}(W, M^*(\tau^*, 1-s) \circ M^*(\tau, s)f_s, \overline{\phi}) \\&= \omega_\pi(-1)^{n-1}\omega_{\tau^*}(-1)^n \Gamma(1-s, \pi \times \tau^*, \chi, \psi) \mathcal{L}(W, M^*(\tau, s)f_s, \overline{\phi}) \\&= \Gamma(1-s, \pi \times \tau^*, \chi, \psi)\Gamma(s, \pi \times \tau, \chi, \psi) \mathcal{L}(W, f_s, \overline{\phi}) \end{align*} $$

for any W, $f_s$ and $\phi $ . It means that

$$\begin{align*}\Gamma(1-s, \pi \times \tau^*, \chi, \psi)\Gamma(s, \pi \times \tau, \chi, \psi) = 1, \end{align*}$$

which is equivalent to saying that

$$\begin{align*}\varepsilon(1-s, \pi \times \tau^*, \chi, \psi)\varepsilon(s, \pi \times \tau, \chi, \psi) = 1. \end{align*}$$

Hence, $\varepsilon (s,\pi \times \tau , \chi , \psi ) \in \mathbb {C}[q^{-s}, q^{s}]^\times = \mathbb {C}^\times (q^{-s})^{\mathbb {Z}}$ .

4.2 Unramified representations

In this subsection, we consider the Rankin–Selberg integrals when $\tau $ varies over irreducible unramified representations of $\mathrm {GL}_{n-1}(E)$ .

Recall that $K^{W_0} = K_{0}^W \cap G_{n-1}$ . It is a hyperspecial maximal compact subgroup of $G_{n-1}$ , and the Iwasawa decomposition $G_{n-1} = Q_{n-1} K^{W_0}$ holds.

Irreducible unramified representations of $\mathrm {GL}_{n-1}(E)$ are parametrized by the Satake parameters $\underline {x} = (x_1,\dots ,x_{n-1}) \in (\mathbb {C}^\times )^{n-1}/S_{n-1}$ . We write the unramified representation associated to $\underline {x}$ by $\tau _{\underline {x}}$ . Then for almost all $\underline {x}$ , since $\tau _{\underline {x}}$ is generic, there exists a unique function $f_{s}(\underline {x}) \in V_{Q_{n-1}}^{G_{n-1}}(\mathcal {W}(\tau_{\underline{x}} ,\psi _E^{-1}), s)$ such that

• • $f_{s}(gk, a; \underline {x}) = f_{s}(g, a; \underline {x})$ for any $g \in G_{n-1}$ , $k \in K^{W_0}$ and $a \in \mathrm {GL}_{n-1}(E)$ ; and

• • the function $W(a; \underline {x}) = f_{s}(\mathbf {1}_{2(n-1)}, a; \underline {x})$ is right $\mathrm {GL}_{n-1}(\mathfrak {o}_E)$ -invariant with $W(\mathbf {1}_{n-1}; \underline {x}) = 1$ .

Lemma 4.3. For $\underline {x} = (x_1, \dots , x_{n-1})$ , we write $\underline {x}^{-1} = (x_1^{-1}, \dots , x_{n-1}^{-1})$ . Then we have

$$ \begin{align*} &\frac{M^*(\tau_{\underline{x}}, s)f_{s}(\underline{x})} {\prod_{i=1}^{n-1}(1-q^{-s}x_i)\prod_{1 \leq i < j \leq n-1}(1-q^{-2s}x_ix_j)} \\ &\quad = \frac{f_{1-s}(\underline{x}^{-1})} {\prod_{i=1}^{n-1}(1-q^{-(1-s)}x_i^{-1})\prod_{1 \leq i < j \leq n-1}(1-q^{-2(1-s)}x_i^{-1}x_j^{-1})}. \end{align*} $$

Proof. The assertion follows from [4, Theorem 8.1] and [21, Theorem 3.1].

Let $\pi $ be an irreducible $\psi _E$ -generic tempered representation of $G_n$ with L-parameter $\phi _\pi $ . Then by the uniqueness of the gamma factor (Theorem 4.2 (4)), we have

$$\begin{align*}\Gamma(s, \pi \times \tau_{\underline{x}}, \chi, \psi) = \prod_{i=1}^{n-1} \varepsilon(s+s_i+s_0, \phi_\pi, \psi_E)\frac{L(1-s-s_i-s_0, \phi_\pi^\vee)}{L(s+s_i+s_0, \phi_\pi)} \end{align*}$$

for almost all $\underline {x} = (x_1,\dots ,x_{n-1})$ , where $s_0, s_1, \dots , s_{n-1} \in \mathbb {C}$ are such that $q^{-2s_0} = -1$ and $x_i = q^{-2s_i}$ for $1 \leq i \leq n-1$ . Since $\phi _\pi $ is tempered, two meromorphic functions $\prod _{i=1}^{n-1}L(1-s-s_i-s_0, \phi _\pi ^\vee )$ and $\prod _{i=1}^{n-1}L(s+s_i+s_0, \phi _\pi )$ have no common pole for almost all $\underline {x}$ . In particular, in this case, we have

$$ \begin{align*} L(s, \pi \times \tau_{\underline{x}}, \chi) &= \prod_{i=1}^{n-1}L(s+s_i+s_0, \phi_\pi), \\ \varepsilon(s, \pi \times \tau_{\underline{x}}, \chi, \psi) &= \prod_{i=1}^{n-1} \varepsilon(s+s_i+s_0, \phi_\pi, \psi_E). \end{align*} $$

If we write $L(s, \phi _\pi ) = P_\pi (q^{-2s})$ and $\varepsilon (s, \phi _\pi , \psi _E) = \varepsilon q^{c(\phi _\pi )(1-2s)}$ , then

$$ \begin{align*} L(s, \pi \times \tau_{\underline{x}}, \chi) &= \prod_{i=1}^{n-1}P_\pi(-x_iq^{-2s}), \\ \varepsilon(s, \pi \times \tau_{\underline{x}}, \chi, \psi) &= \varepsilon^{n-1} (-q^{1-2s})^{c(\phi_\pi)(n-1)} \prod_{i=1}^{n-1}x_i^{c(\phi_\pi)}. \end{align*} $$

4.3 Proof of Theorem 2.2 (2)

The symmetric group $S_{n-1}$ acts on $\mathbb {C}[X_1^{\pm 1},\dots , X_{n-1}^{\pm 1}]$ canonically. Set

$$\begin{align*}\mathcal{T} = \mathbb{C}[X_1^{\pm1},\dots,X_{n-1}^{\pm1}]^{S_{n-1}}. \end{align*}$$

Note that

$$\begin{align*}\mathcal{T} = \mathbb{C}[T_1,\dots,T_{n-2}, T_{n-1}, T_{n-1}^{-1}] \end{align*}$$

with

$$\begin{align*}T_i = \sum_{\sigma \in S_{n-1}} X_{\sigma(1)} \cdots X_{\sigma(i)}. \end{align*}$$

The degree with respect to $T_{n-1}$ gives a $\mathbb {Z}$ -grading on $\mathcal {T}$ ; that is, $\mathcal {T} = \oplus _{d \in \mathbb {Z}} \mathcal {T}_d$ with

$$\begin{align*}\mathcal{T}_d = \mathbb{C}[T_1,\dots,T_{n-2}] T_{n-1}^d. \end{align*}$$

Write $\underline {X} = (X_1,\dots ,X_{n-1})$ and $q^{1-2s}\underline {X} = (q^{1-2s}X_1, \dots , q^{1-2s}X_{n-1})$ . There is a function

$$\begin{align*}W(\underline{X}) \colon \mathrm{GL}_{n-1}(E) \rightarrow \mathcal{T} \end{align*}$$

such that $W(\underline {X})|_{\underline {X} = \underline {x}} = W(\underline {x})$ for almost all $\underline {x} \in (\mathbb {C}^\times )^{n-1}$ . Similarly, we consider the function $f_{s}(\underline {X}) \colon G_{n-1} \times \mathrm {GL}_{n-1}(E) \rightarrow \mathcal {T}$ so that $f_{s}(\underline {X})|_{\underline {X} = \underline {x}} = f_{s}(\underline {x})$ for almost all $\underline {x} \in (\mathbb {C}^\times )^{n-1}$ . In particular, $f_{s}(\mathbf {1}_{2(n-1)}, a; \underline {X}) = W(a; q^{1-2s}\underline {X})$ .

We regard $\mathcal {L}(W, f_{1/2}(\underline {X}), \overline {\phi _0})$ as a formal power series of $X_1^{\pm 1}, \dots , X_{n-1}^{\pm 1}$ , or an element of $\mathbb {C}[T_1,\dots ,T_{n-2}] [[T_{n-1}^{\pm 1}]]$ . For $\lambda = (\lambda _1,\dots , \lambda _{n-1}) \in \mathbb {Z}^{n-1}$ , we set $|\lambda | = \lambda _1+\dots +\lambda _{n-1}$ . The following is a key lemma.

Lemma 4.4. Let $W \in \mathcal {W}(\pi , \psi _E)^{K_{2m}^W}$ . Write

$$ \begin{align*} \mathcal{L}(W, f_{1/2}(\underline{X}), \overline{\phi_0}) &= \sum_{\lambda \in \mathbb{Z}^{n-1}} a_\lambda(W) X_1^{\lambda_1}\cdots X_{n-1}^{\lambda_{n-1}} = \sum_{d \in \mathbb{Z}} \mathcal{L}_d(W) T_{n-1}^d \end{align*} $$

with $a_\lambda (W) \in \mathbb {C}$ and $\mathcal {L}_d(W) \in \mathbb {C}[T_1, \dots , T_{n-2}]$ . Then

• • $a_\lambda (W) = 0$ unless $|\lambda | \geq -(n-1)m$ ; and

• • $\mathcal {L}_d(W) = 0$ unless $d \geq -m$ .

Proof. For row vectors $x, u \in E^{n-1}$ and $a \in \mathrm {GL}_{n-1}(E)$ , we put $k(x,a,u)$ to be the matrix

$$\begin{align*}(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a))^{-1} \left( \begin{array}{cc|cc} \mathbf{1}_{n-1} & {}^tu & 0 & 0 \\ 0 & 1 & 0 & 0 \\ \hline 0 & 0 & 1 & -\overline{u}w_{n-1} \\ 0 & 0 & 0 & \mathbf{1}_{n-1} \end{array} \right) w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a). \end{align*}$$

By an easy calculation, $k(x,a,u)$ is equal to

$$\begin{align*}\left( \begin{array}{cc|cc} 1-x{}^tu & -x{}^tuxa & 0 & 0 \\ a^{-1}{}^tu & \mathbf{1}_{n-1} + a^{-1}{}^tuxa & 0 & 0 \\ \hline 0 & 0 & \mathbf{1}_{n-1} -w_{n-1}{}^t\overline{a}{}^t\overline{x} \overline{u} {}^t\overline{a}^{-1}w_{n-1} & w_{n-1}{}^t\overline{a}{}^t\overline{x} \overline{u}{}^t\overline{x}\\ 0 & 0 & -\overline{u}{}^t\overline{a}^{-1}w_{n-1} & 1+\overline{u}{}^t\overline{x} \end{array} \right). \end{align*}$$

In particular, if $xa \in \mathfrak {o}_E^{n-1}$ and $u{}^ta^{-1} \in (\mathfrak {p}_E^{m})^{n-1}$ , then $x{}^tu \in \mathfrak {p}_E^{m}$ so that $k(x,a,u) \in K_{2m}^W$ .

As functions on $g \in G_{n-1}$ , all of $W(w_{1,n-1}\mathbf {v}(x,0;0)g)$ , $f_{s}(g,\mathbf {1}_{n-1}; \underline {X})$ and $\overline {\omega _{\psi }(g)\phi _0}$ are right $K^{W_0}$ -invariant. Hence, by the integral formula with respect to the Iwasawa decomposition, we can write $\mathcal {L}(W,f_{s}(\underline {X}), \overline {\phi _0})$ as

$$\begin{align*}\int_{T_{n-1}}\int_{E^{n-1}} W(w_{1,n-1}\mathbf{v}(x,0;0)t) f_{s}(t, \mathbf{1}_{n-1}; \underline{X}) \overline{\omega_{\psi}(t)\phi_0(x)} \delta_{B_{n-1}}^{-1}(t)dxdt, \end{align*}$$

where $B_{n-1} = T_{n-1}N_{n-1}$ is the upper triangular Borel subgroup of $G_{n-1}$ with the diagonal torus $T_{n-1}$ . Write $t = \mathbf {m}(a)$ with $a = \mathrm {diag}(a_1,\dots ,a_{n-1})$ being a diagonal matrix in $\mathrm {GL}_{n-1}(E)$ . Then $\omega _{\psi }(\mathbf {m}(a))\phi _0(x) \not = 0 \iff xa \in \mathfrak {o}_E^{n-1}$ . In this case, if $W(w_{1,n-1}\mathbf {v}(x,0;0)\mathbf {m}(a)) \not = 0$ , then for $u = (u_1,\dots ,u_{n-1}) \in E^{n-1}$ such that $u{}^ta^{-1} \in (\mathfrak {p}_E^{m})^{n-1}$ , we have

$$ \begin{align*} 0 &\not= W(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a)) \\&= W(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a) \cdot k(x,a,u)) \\&= W\left( \left( \begin{array}{cc|cc} \mathbf{1}_{n-1} & {}^tu & 0 & 0 \\ 0 & 1 & 0 & 0 \\ \hline 0 & 0 & 1 & -\overline{u}w_{n-1} \\ 0 & 0 & 0 & \mathbf{1}_{n-1} \end{array} \right) w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a) \right) \\&= \psi_E(u_{n-1})W(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a)). \end{align*} $$

This shows that

$$\begin{align*}u_{n-1} \in \mathfrak{p}_E^{\mathrm{ord}(a_{n-1})+m} \implies \psi_E(u_{n-1}) = 1. \end{align*}$$

This means that $\mathrm {ord}(a_{n-1})+m \geq 0$ .

Recall that $f_{s}(\mathbf {m}(a), \mathbf {1}_{n-1}; \underline {X}) = \delta _{Q_{n-1}}^{\frac {1}{2}}(\mathbf {m}(a)) W(a; q^{1-2s}\underline {X})$ . By a similar (and well-known) argument, if $W(a; q^{1-2s}\underline {X}) \not = 0$ , then $\mathrm {ord}(a_1) \geq \dots \geq \mathrm {ord}(a_{n-1})$ . Hence, we conclude that if

$$\begin{align*}W(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a)) W(a; \underline{X}) \overline{\omega_{\psi}(\mathbf{m}(a))\phi_0(x)} \not= 0, \end{align*}$$

then

$$\begin{align*}\mathrm{ord}(a_1) \geq \dots \geq \mathrm{ord}(a_{n-1}) \geq -m \end{align*}$$

so that

$$\begin{align*}\mathrm{ord}(\det(a)) = \sum_{i=1}^{n-1}\mathrm{ord}(a_i) \geq -(n-1)m. \end{align*}$$

Since the Casselman–Shalika formula [6] tells us that

$$\begin{align*}W(a; \underline{X}) \in \left( \bigoplus_{\substack{\lambda \in \mathbb{Z}^{n-1} \\ |\lambda| = \mathrm{ord}(\det(a))}} \mathbb{C} X_1^{\lambda_1}\cdots X_{n-1}^{\lambda_{n-1}} \right) \cap \mathbb{C}[T_1,\dots,T_{n-2}]T_{n-1}^{\mathrm{ord}(a_{n-1})}, \end{align*}$$

we obtain the assertions.

For $W \in \mathcal {W}(\pi , \psi _E)^{K_{2m}^W}$ , we define $\Psi (W; \underline {X})$ by

$$\begin{align*}\Psi(W; \underline{X}) = \frac{\prod_{i=1}^{n-1}P_\pi(-q^{-1}X_i)\mathcal{L}(W, f_{1/2}(\underline{X}), \phi_0)} {\prod_{i=1}^{n-1}(1-q^{-1}X_i)\prod_{1\leq i < j \leq n-1}(1-q^{-2}X_iX_j)}. \end{align*}$$

Proposition 4.5. If $2m < c(\phi _\pi )$ , then $\Psi (W; \underline {X}) = 0$ for $W \in \mathcal {W}(\pi , \psi _E)^{K_{2m}^W}$ . If $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ , then

$$\begin{align*}\dim_{\mathbb{C}} \left\{ \Psi(W; \underline{X}) \;\middle|\; W \in \mathcal{W}(\pi, \psi_E)^{K_{2m}^W}\right\} \leq 1. \end{align*}$$

Proof. Since $P_\pi (X)$ is a polynomial of X with $P_\pi (0) = 1$ , and since $(1-q^{-1}X_i)^{-1} = \sum _{k=0}^\infty (q^{-1}X_i)^k$ and $(1-q^{-2}X_iX_j)^{-1} = \sum _{k=0}^\infty (q^{-2}X_iX_j)^k$ , if we write

$$ \begin{align*} \Psi(W; \underline{X}) &= \sum_{\lambda \in \mathbb{Z}^{n-1}} \alpha_\lambda(W) X_1^{\lambda_1}\cdots X_{n-1}^{\lambda_{n-1}} = \sum_{d \in \mathbb{Z}} \Psi_d(W; \underline{X}) T_{n-1}^d \end{align*} $$

with $\alpha _\lambda (W) \in \mathbb {C}$ and $\Psi _d(W; \underline {X}) \in \mathbb {C}[T_1, \dots , T_{n-2}]$ , by Lemma 4.4, we see that

• • $\alpha _\lambda (W) = 0$ unless $|\lambda | \geq -(n-1)m$ ; and

• • $\Psi _d(W; \underline {X}) = 0$ unless $d \geq -m$ .

Write $\underline {X}^{-1} = (X_1^{-1}, \dots , X_{n-1}^{-1})$ . By the functional equation (Theorem 4.2 (3), (5)) together with Lemma 4.3, we see that

(*) $$ \begin{align} T_{n-1}^{-c(\phi_\pi)} \Psi(W; \underline{X}^{-1}) = \varepsilon_0 \Psi(W; \underline{X}) \end{align} $$

with

$$\begin{align*}\varepsilon_0 = ((-1)^{c(\phi_\pi)} \varepsilon \cdot \omega_\pi(-1))^{n-1}. \end{align*}$$

The left-hand side and the right-hand side of $(\ast )$ belong to

$$\begin{align*}\bigoplus_{d \leq m-c(\phi_\pi)} \mathbb{C}[T_1, \dots, T_{n-2}] T_{n-1}^{d}, \quad \bigoplus_{d \geq -m} \mathbb{C}[T_1, \dots, T_{n-2}] T_{n-1}^{d}, \end{align*}$$

respectively. Hence, if $\Psi _d(W; \underline {X}) \not = 0$ , then $-m \leq d \leq m-c(\phi _\pi )$ so that $2m \geq c(\phi _\pi )$ . A similar argument shows that if $\alpha _{\lambda }(W) \not = 0$ , then

$$\begin{align*}-(n-1)m \leq |\lambda| \leq (n-1)(m-c(\phi_\pi)). \end{align*}$$

Now we assume that $2m = c(\phi _\pi )$ . Then $\Psi _d(W; \underline {X}) = 0$ unless $d = -m$ . Hence,

$$\begin{align*}T_{n-1}^{m} \Psi(W; \underline{X}) \in \mathbb{C}[T_1, \dots, T_{n-2}] \subset \mathbb{C}[X_1, \dots, X_{n-1}]. \end{align*}$$

This implies that $\alpha _{\lambda }(W) = 0$ unless $\lambda _i \geq -m$ for any $1 \leq i \leq n-1$ . However, since $\alpha _{\lambda }(W) = 0$ unless $|\lambda | = -(n-1)m$ , we see that $\alpha _{\lambda }(W) = 0$ unless $\lambda _1 = \dots = \lambda _{n-1} = -m$ . This means that

$$\begin{align*}\Psi(W; \underline{X}) \in \mathbb{C} T_{n-1}^{-m} \end{align*}$$

so that

$$\begin{align*}\dim_{\mathbb{C}} \left\{ \Psi(W; \underline{X}) \;\middle|\; W \in \mathcal{W}(\pi, \psi_E)^{K_{c(\phi_\pi)}^W}\right\} \leq 1. \end{align*}$$

Next, we assume that $2m = c(\phi _\pi )+1$ . Then $\Psi _d(W; \underline {X}) = 0$ unless $d = -m, -m+1$ , and $\alpha _\lambda (W) = 0$ unless $|\lambda | = -(n-1)m, -(n-1)(m-1)$ . In particular, $\Psi _{-m+1}(W; \underline {X})$ is a scalar so that

$$\begin{align*}\Psi_{-m+1}(W; \underline{X}^{-1}) = \Psi_{-m+1}(W; \underline{X}). \end{align*}$$

By the functional equation $(\ast )$ , we have

$$ \begin{align*} \Psi_{-m+1}(W; \underline{X}^{-1}) &= \varepsilon_0 \Psi_{-m}(W; \underline{X}), \\ \Psi_{-m}(W; \underline{X}^{-1}) &= \varepsilon_0 \Psi_{-m+1}(W; \underline{X}). \end{align*} $$

Hence, $\Psi _{-m}(W; \underline {X})$ is also a scalar. Therefore,

$$\begin{align*}\Psi(W; \underline{X}) \in \mathbb{C}(T_{n-1}^{-m} + \varepsilon_0 T_{n-1}^{-m+1}) \end{align*}$$

so that

$$\begin{align*}\dim_{\mathbb{C}} \left\{ \Psi(W; \underline{X}) \;\middle|\; W \in \mathcal{W}(\pi, \psi_E)^{K_{c(\phi_\pi)+1}^W}\right\} \leq 1. \end{align*}$$

This completes the proof.

By Proposition 3.3, we see that $\mathcal {W}(\pi , \psi _E)^{K_{2m}^W} \ni W \mapsto \Psi (W; \underline {X})$ gives an injective linear map

$$\begin{align*}\Psi \colon \pi_{\psi}^{K_{2m}^W} \hookrightarrow \mathcal{T}. \end{align*}$$

Hence, by Proposition 4.5, we have

• • $\pi _{\psi }^{K_{2m}^W} = 0$ if $2m < c(\phi _\pi )$ ; and

• • $\dim _{\mathbb {C}}(\pi _{\psi }^{K_{2m}^W}) \leq 1$ if $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ .

This completes the proof of Theorem 2.2 (2).

5 Existence

In this section, we will prove Theorem 2.2 (3). To do this, we will use the theta correspondence for $(\mathrm {U}(V), \mathrm {U}(W))$ .

5.1 Theta correspondence

Recall that $V = V_{2n+1}$ (resp. $W = W_{2n}$ ) is a hermitian (resp. skew-hermitian) space over E of dimension $2n+1$ (resp. $2n$ ). Then $\mathbb {W} = V \otimes _E W$ forms a symplectic space of dimension $4n(2n+1)$ equipped with the symplectic form

$$\begin{align*}\left\langle v \otimes w, v' \otimes w' \right\rangle = \mathrm{tr}_{E/F}\left( \left\langle v,v' \right\rangle_V \cdot \left\langle w,w' \right\rangle_W \right). \end{align*}$$

Here, $\mathrm {U}(V)$ , $\mathrm {U}(W)$ and $\mathrm {Sp}(\mathbb {W})$ act on V, W and $\mathbb {W}$ , respectively, all from the left. We have a canonical map $\mathrm {U}(V) \times \mathrm {U}(W) \rightarrow \mathrm {Sp}(\mathbb {W})$ .

Recall that $\chi $ is the unique nontrivial quadratic unramified character of $E^\times $ . Note that $\chi |_{F^\times }$ is equal to the quadratic character corresponding to $E/F$ . Let $\widetilde {\mathrm {Sp}}(\mathbb {W})$ be the metaplectic $\mathbb {C}^\times $ -cover. Using the pair $(\chi _V, \chi _W) = (\chi ^{2n+1}, \chi ^{2n})$ , we have Kudla’s splitting [15]

$$\begin{align*}\mathrm{U}(V) \times \mathrm{U}(W) \rightarrow \widetilde{\mathrm{Sp}}(\mathbb{W}). \end{align*}$$

Let $\omega _\psi $ be the Weil representation of $\widetilde {\mathrm {Sp}}(\mathbb {W})$ associated to the additive character $\psi $ . By the pullback, we obtain the Weil representation $\omega _{\psi , V, W}$ of $\mathrm {U}(V) \times \mathrm {U}(W)$ . For an irreducible representation $\pi $ of $\mathrm {U}(W)$ , it is known that the maximal $\pi $ -isotypic quotient of $\omega _{\psi , V, W}$ is of the form

$$\begin{align*}\Theta_\psi(\pi) \boxtimes \pi \end{align*}$$

for a smooth representation $\Theta _\psi (\pi )$ of $\mathrm {U}(V)$ of finite length. The Howe duality conjecture, proven by Waldspurger [24], asserts that if $\Theta _\psi (\pi )$ is nonzero, then it has a unique irreducible quotient $\theta _\psi (\pi )$ . We call $\theta _\psi (\pi )$ the theta lift of $\pi $ .

The following is a special case of Prasad’s conjecture, which was proven by Gan–Ichino [10]. See also Theorem 4.4 in that paper.

Theorem 5.1. Let $\pi $ be an irreducible $\psi _E$ -generic representation of $\mathrm {U}(W)$ with L-parameter $\phi _\pi $ . Then $\Theta _\psi (\pi )$ is always nonzero. Moreover, $\sigma = \theta _\psi (\pi )$ is generic, and its L-parameter is given by

$$\begin{align*}\phi_\sigma = \phi_\pi\chi \oplus \mathbf{1}, \end{align*}$$

where $\phi _\pi \chi = \phi _\pi \otimes \chi $ .

In particular, if $\sigma = \theta _\psi (\pi )$ , then we have $c(\phi _\sigma ) = c(\phi _\pi )$ and $\omega _{\sigma } = \omega _{\pi }$ . Moreover, if $\pi $ is tempered, then so is $\sigma $ , so that we have $\sigma ^{K_{2m}^V} \not = 0$ for $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ by Theorem 2.1.

5.2 Lattice model

First, we will show that $\pi ^{K_{2m}^W} \not = 0$ . To do this, we use a lattice model $\mathcal {S} = \mathcal {S}(A)$ of the Weil representation $\omega _\psi $ of $\widetilde {\mathrm {Sp}}(\mathbb {W})$ . In this subsection, we recall this model.

Let $\mathbb {W}$ be a symplectic space over F of dimension $2N$ equipped with a symplectic form $\left \langle \cdot , \cdot \right \rangle $ . The group law of the Heisenberg group $H(\mathbb {W}) = \mathbb {W} \oplus F$ is given by

$$\begin{align*}(w_1,t_1) \cdot (w_2, t_2) = \left(w_1+w_2, t_1+t_2+\frac{1}{2}\left\langle w_1,w_2 \right\rangle\right), \end{align*}$$

whose center is $\{0\} \oplus F \cong F$ . By the Stone–von Neumann theorem, there is a unique (up to isomorphism) irreducible admissible representation $(\rho _\psi , \mathcal {S})$ of $H(\mathbb {W})$ whose central character is $\psi $ . The symplectic group $\mathrm {Sp}(\mathbb {W})$ acts on $H(\mathbb {W})$ by $g \cdot (w,t) = (gw,t)$ . By the uniqueness, for $g \in \mathrm {Sp}(\mathbb {W})$ , we have $M_g \in \mathrm {Aut}(\mathcal {S})$ such that

(⋆) $$ \begin{align} M_g \circ \rho_\psi(h) \circ M_g^{-1} = \rho_\psi(gh) \quad\text{for } h \in H(\mathbb{W}). \end{align} $$

By Schur’s lemma, such $M_g$ is determined uniquely up to a nonzero scalar. Define the metaplectic $\mathbb {C}^\times $ -cover $\widetilde {\mathrm {Sp}}(\mathbb {W})$ of $\mathrm {Sp}(\mathbb {W})$ by

$$\begin{align*}\widetilde{\mathrm{Sp}}(\mathbb{W}) = \{(g,M_g) \in \mathrm{Sp}(\mathbb{W}) \times \mathrm{Aut}(\mathcal{S}) \;|\; M_g \text{ satisfies } (\star)\}. \end{align*}$$

We have an exact sequence

given by $\alpha (z) = (\mathbf {1}_{\mathbb {W}}, z \cdot \mathrm {id}_{\mathcal {S}})$ and $\beta (g,M_g) = g$ . The Weil representation $\omega _\psi $ of $\widetilde {\mathrm {Sp}}(\mathbb {W})$ on the space $\mathcal {S}$ is defined by

$$\begin{align*}\omega_\psi(g,M_g) = M_g. \end{align*}$$

Now we shall give a realization of the space $\mathcal {S}$ . Let A be a lattice of $\mathbb {W}$ (i.e., a free $\mathfrak {o}_F$ -submodule of rank $2N$ ). The dual lattice $A^*$ is defined by

$$\begin{align*}A^* = \left\{w \in \mathbb{W} \;\middle|\; \left\langle w,a \right\rangle \in \mathfrak{o}_F \text{for any} a \in A \right\}. \end{align*}$$

Suppose that A is self-dual (i.e., $A^* = A$ ). Let $\mathcal {S}(A)$ be the space of locally constant, compactly supported functions $\phi \colon H(\mathbb {W}) \rightarrow \mathbb {C}$ such that

$$\begin{align*}\phi((a,t) \cdot h) = \psi(t) \phi(h) \end{align*}$$

for $(a,t) \in A \oplus F$ and $h \in H(\mathbb {W})$ . The group $H(\mathbb {W})$ acts on $\mathcal {S}(A)$ by the right translation $\rho _\psi $ . It is known that the representation $(\rho _\psi , \mathcal {S}(A))$ of $H(\mathbb {W})$ is irreducible with the central character $\psi $ . This gives a realization $(\omega _\psi , \mathcal {S}(A))$ of the Weil representation which is called a lattice model. Since $(a,0) \cdot (w,0) = (a+w, \frac {1}{2}\left \langle a,w \right \rangle )$ , by the restriction to $\mathbb {W} \oplus \{0\}$ , we can identify $\mathcal {S}(A)$ with the space of locally constant, compactly supported functions $\phi \colon \mathbb {W} \rightarrow \mathbb {C}$ such that

$$\begin{align*}\phi(a+w) = \psi\left(-\frac{1}{2}\left\langle a,w \right\rangle\right) \phi(w) \end{align*}$$

for $a \in A$ and $w \in \mathbb {W}$ .

For $g \in \mathrm {Sp}(\mathbb {W})$ , we define $M[g] \in \mathrm {Aut}(\mathcal {S}(A))$ by

$$\begin{align*}(M[g]\phi)(w) = \int_{A} \psi\left(\frac{1}{2}\left\langle a,w \right\rangle\right) \phi(g^{-1} \cdot (a+w)) da \end{align*}$$

for $\phi \in \mathcal {S}(A)$ and $w \in \mathbb {W}$ . Here, $da$ is the Haar measure on A normalized so that $\mathrm {vol}(A) = 1$ . It is easy to check that $(g,M[g]) \in \widetilde {\mathrm {Sp}}(\mathbb {W})$ .

Let $K_A$ be the stabilizer of A in $\mathrm {Sp}(\mathbb {W})$ . Then we have

$$\begin{align*}(M[k]\phi)(w) = \phi(k^{-1} \cdot w) \end{align*}$$

for $k \in K_A$ , $\phi \in \mathcal {S}(A)$ , and $w \in \mathbb {W}$ . The map $k \mapsto (k,M[k])$ gives a splitting $K_A \rightarrow \widetilde {\mathrm {Sp}}(\mathbb {W})$ . If we identify $K_A$ with the image, the restriction of the Weil representation $(\omega _\psi , \mathcal {S}(A))$ to $K_A$ is given by $\omega _\psi (k)\phi (w) = \phi (k^{-1} \cdot w)$ .

5.3 Families of lattices

Take bases $\{e_n, \dots , e_1, e_0, e_{-1}, \dots , e_{-n}\}$ of V and $\{f_n,\dots ,f_1, f_{-1},\dots ,f_{-n}\}$ of W, respectively, as in §2.1. Set

$$ \begin{align*} \Gamma_V &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_i\right) \oplus \mathfrak{o}_E e_0 \oplus \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_{-i} \right), \\ \Gamma_W &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E f_i\right) \oplus \left(\bigoplus_{i=1}^n \mathfrak{o}_E f_{-i} \right). \end{align*} $$

Then $\Gamma _V$ and $\Gamma _W$ are self-dual lattices (i.e., $\Gamma _V^* = \Gamma _V$ and $\Gamma _W^* = \Gamma _W$ ).

In this subsection, for two $\mathfrak {o}_E$ -modules $\Gamma _1$ and $\Gamma _2$ , we denote by $\Gamma _1 \otimes \Gamma _2$ the tensor product of $\mathfrak {o}_E$ -modules. We put

$$\begin{align*}A = \Gamma_V \otimes \Gamma_W. \end{align*}$$

This is a self-dual lattice of $\mathbb {W} = V \otimes _F W$ , (i.e., $A^* = A$ ). We will consider the lattice model $(\omega _\psi , \mathcal {S}(A))$ of the Weil representation of $\widetilde {\mathrm {Sp}}(\mathbb {W})$ .

Fix a non-negative even integer $2m \geq 0$ . We consider lattices

$$ \begin{align*} M_{2m} &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_i\right) \oplus \mathfrak{p}_E^{m} e_0 \oplus \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_{-i}\right),\\ N_{2m} &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E f_i \right) \oplus \left(\bigoplus_{i=1}^{n-1} \mathfrak{o}_E f_{-i} \right) \oplus \mathfrak{p}_E^{m} f_{-n} \end{align*} $$

of V and W, respectively. Then $M_{2m} \subset \Gamma _V$ and $N_{2m} \subset \Gamma _W$ . Moreover, the dual lattices are given by

$$ \begin{align*} M_{2m}^* &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_i\right) \oplus \mathfrak{p}_E^{-m} e_0 \oplus \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_{-i}\right), \\ N_{2m}^* &= \mathfrak{p}_E^{-m} f_n \oplus \left(\bigoplus_{i=1}^{n-1} \mathfrak{o}_E f_i \right) \oplus \left(\bigoplus_{i=1}^{n} \mathfrak{o}_E f_{-i} \right). \end{align*} $$

Recall that in Section 2.4, we defined compact subgroups $K_{2m}^V$ and $K_{2m}^W$ of $\mathrm {U}(V)$ and $\mathrm {U}(W)$ , respectively. The following lemma is easy to check.

Lemma 5.2. We have

$$ \begin{align*} K_{2m}^V = \{h \in \mathrm{U}(V) \;|\; (h-1) \cdot M_{2m}^* \subset M_{2m} \}, \\ K_{2m}^W = \{g \in \mathrm{U}(W) \;|\; (g-1) \cdot N_{2m}^* \subset N_{2m} \}. \end{align*} $$

In particular, $K_{2m}^V \times K_{2m}^W$ is contained in $K_A$ under the canonical map $\mathrm {U}(V) \times \mathrm {U}(W) \rightarrow \mathrm {Sp}(\mathbb {W})$ .

Let $\mathcal {S}(A)_{M_{2m}}$ be the subspace of $\mathcal {S}(A)$ consisting of functions $\phi \colon \mathbb {W} \rightarrow \mathbb {C}$ such that $\mathrm {Supp}(\phi ) \subset M_{2m}^* \otimes \Gamma _W$ . We will use the following result proven by Waldspurger.

Proposition 5.3 [24, Corollary III.2].

Let $J_{2m}^V$ be a compact subgroup of $\mathrm {U}(V)$ . Suppose that

• • $J_{2m}^V \supset K_{2m}^V$ ;

• • $\mathcal {S}(A)_{M_{2m}}$ is stable by $J_{2m}^V$ ;

• • $(\mathcal {S}(A)_{M_{2m}})^{J_m^V} \not = \{0\}$ .

Then $\mathcal {S}(A)^{J_{2m}^V}$ is generated by $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V}$ as a representation of $\mathrm {U}(W)$ .

We will apply this proposition to the compact subgroup $J_{2m}^V$ generated by $K_{2m}^V$ and $E^1 \cap (1+\mathfrak {p}_E^{m})$ , where the latter is regarded as a subgroup of the center of $\mathrm {U}(V)$ . Namely, $J_{0}^V = K_{0}^V$ , and

for $2m> 0$ . It is clear that $J_{2m}^V \supset K_{2m}^V$ .

We check the second and third conditions in Proposition 5.3.

Lemma 5.4. The space $\mathcal {S}(A)_{M_{2m}}$ is stable by $J_{2m}^V$ and fixed by $K_{2m}^V$ . Moreover, $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V} \not = \{0\}$ .

Proof. For $t \in \mathfrak {p}_E^{-m}$ and $w \in \Gamma _W$ , define $\phi _{t,w} \in \mathcal {S}(A)$ so that $\mathrm {Supp}(\phi _{t,w}) = A + te_0 \otimes w$ and $\phi _{t,w}(te_0 \otimes w) = 1$ . Then $\mathcal {S}(A)_{M_{2m}}$ is equal to the $\mathbb {C}$ -span of

$$\begin{align*}\left\{\phi_{t,w} \;\middle|\; t \in \mathfrak{p}_E^{-m},\; w \in \Gamma_W \right\}. \end{align*}$$

For $k \in J_{2m}^{V}$ , write $ke_0 = \sum _{i = -n}^n k_i e_i$ . Then

$$\begin{align*}k_i \in \left\{ \begin{aligned} &\mathfrak{p}_E^{m} &\quad&\text{if } i \not=0, \\ &1+\mathfrak{p}_E^{m} &\quad&\text{if } i = 0. \end{aligned} \right. \end{align*}$$

In particular, we see that $(k-1)te_0 \otimes w \in A$ . Hence,

$$ \begin{align*} \mathrm{Supp}(\omega_\psi(k)\phi_{t,w}) &= k(A + te_0 \otimes w) \\&= A + (k-1)te_0 \otimes w + te_0 \otimes w \\&= A + te_0 \otimes w = \mathrm{Supp}(\phi_{t,w}). \end{align*} $$

Moreover,

$$ \begin{align*} \omega_\psi(k)\phi_{t,w}(te_0 \otimes w) &= \omega_\psi(k)\phi_{t,w}\left(kte_0 \otimes w - (k-1)te_0 \otimes w\right) \\&= \psi\left( \frac{1}{2}\left\langle (k-1)te_0 \otimes w, kte_0 \otimes w \right\rangle \right) \omega_\psi(k)\phi_{t,w}(kte_0 \otimes w) \\&= \psi_E\left( \left\langle (k-1)te_0,kte_0 \right\rangle_V \cdot \left\langle w,w \right\rangle_W\right) \phi_{t,w}(te_0 \otimes w) \\&= \psi_E\left( N_{E/F}(t)(\left\langle ke_0,ke_0 \right\rangle_V-\left\langle e_0,ke_0 \right\rangle_V) \cdot \left\langle w,w \right\rangle_W\right) \\&= \psi_E\left( N_{E/F}(t)(1-\overline{k_0}) \left\langle w,w \right\rangle_W \right). \end{align*} $$

Hence, for $t \in \mathfrak {p}_E^{-m}$ , $w \in \Gamma _W$ and $k \in J_{2m}^V$ , there exists $c \in \mathbb {C}^\times $ such that $\omega _\psi (k)\phi _{t,w} = c\phi _{t,w}$ . This shows that $\mathcal {S}(A)_{M_{2m}}$ is stable by $J_{2m}^V$ . Moreover, if $k_0 \in \mathfrak {p}_E^{2m}$ or $\left \langle w,w \right \rangle _W = 0$ , then $c=1$ . Hence, we have $(\mathcal {S}(A)_{M_{2m}})^{K_{2m}^V} = \mathcal {S}(A)_{M_{2m}}$ and $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V} \not = \{0\}$ .

Therefore, by Proposition 5.3, we see that $\mathcal {S}(A)^{J_{2m}^V}$ is generated by $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V}$ as a representation of $\mathrm {U}(W)$ . If $2m> 0$ , then $\mathcal {S}(A)_{M_{2m}} \supset \mathcal {S}(A)_{M_{2m-2}}$ . Let $\mathcal {S}(A)_{M_{2m} \setminus M_{2m-2}}$ be the subspace spanned by

$$\begin{align*}\left\{\phi_{t,w} \;\middle|\; \mathrm{ord}(t) = -m,\; w \in \Gamma_W \setminus \varpi\Gamma_W \right\}. \end{align*}$$

Then we have

$$\begin{align*}\mathcal{S}(A)_{M_{2m}} = \mathcal{S}(A)_{M_{2m-2}} \oplus \mathcal{S}(A)_{M_{2m} \setminus M_{2m-2}}. \end{align*}$$

Lemma 5.5. Suppose that $2m> 0$ . The image $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V}$ under the projection $\mathcal {S}(A)_{M_{2m}} \twoheadrightarrow \mathcal {S}(A)_{M_{2m} \setminus M_{2m-2}}$ is equal to the one of the subspace spanned by

$$\begin{align*}\left\{ \omega_\psi(k')\phi_{t, f_n} \;\middle|\; \mathrm{ord}(t) = -m,\; k' \in K_0^W \right\}. \end{align*}$$

Moreover, $\phi _{t,f_n}$ is fixed by $K_{2m}^W$ , and $\phi _{t,f_{-n}}$ is fixed by ${}^tK_{2m}^W$ .

Proof. As we have seen in the proof of Lemma 5.4, $k \in J_{2m}^V$ acts on $\phi _{t,w}$ by the character

Hence, the image in question is equal to the one of the subspace spanned by $\phi _{t,w}$ with $\mathrm {ord}(t) = -m$ and $w \in \Gamma _W \setminus \varpi \Gamma _W$ such that $\left \langle w,w \right \rangle _W \in \mathfrak {p}_E^m$ . It means that

$$\begin{align*}\left\langle w,w \right\rangle_W \equiv \left\langle f_n, f_n \right\rangle_W \bmod \mathfrak{p}_E^{m}. \end{align*}$$

Note that

$$\begin{align*}K_0^W = \{g \in \mathrm{U}(V) \;|\; g \Gamma_W = \Gamma_W\} \end{align*}$$

is a hyperspecial maximal compact subgroup of $\mathrm {U}(W)$ . Hence, there exists $k' \in K_0^W$ such that $w \equiv k' \cdot f_n \bmod \varpi ^{m}\Gamma _W$ . In particular, we have

$$\begin{align*}te_0 \otimes w - te_0 \otimes k' \cdot f_n \in A. \end{align*}$$

Hence, we can find $c \in \mathbb {C}^\times $ such that $\phi _{t,w} = c\phi _{t, k' \cdot f_n} = c \cdot \omega _\psi (k') \phi _{t,f_n}$ . This shows the first assertion.

Fix $k' \in K_{2m}^W$ . Since $(k'-1)f_n \in \varpi ^{m} \Gamma _W$ , we have $(k'-1)(te_0 \otimes f_n) \in A$ for $t \in \mathfrak {p}_E^{-m}$ . Hence, $\mathrm {Supp}(\omega _\psi (k') \phi _{t,f_n}) = \mathrm {Supp}(\phi _{t,f_n})$ . Moreover,

$$ \begin{align*} \omega_\psi(k') \phi_{t,f_n}(te_0 \otimes f_n) &= \omega_\psi(k') \phi_{t,f_n}(k'(te_0 \otimes f_n) - (k'-1)(te_0 \otimes f_n)) \\&= \psi\left( \frac{1}{2}\left\langle (k'-1)(te_0 \otimes f_n), k'(te_0 \otimes f_n) \right\rangle \right) \omega_\psi(k') \phi_{t,f_n}(k'(te_0 \otimes f_n)) \\&= \psi_E\left( N_{E/F}(t) \left\langle (k'-1)f_n,k'f_n \right\rangle_W \right). \end{align*} $$

Since $\left \langle (k'-1)f_n,k'f_n \right \rangle _W = \left \langle -f_n,k'f_n \right \rangle _W \in \mathfrak {p}_E^{2m}$ , we have $\omega _\psi (k') \phi _{t,f_n}(te_0 \otimes f_n) = 1$ . Therefore, we conclude that $\omega _\psi (k') \phi _{t,f_n} = \phi _{t,f_n}$ for $k' \in K_{2m}^W$ . By a similar calculation, one can prove that $\omega _\psi (k') \phi _{t,f_{-n}} = \phi _{t,f_{-n}}$ for $k' \in {}^tK_{2m}^W$ . This completes the proof.

5.4 Existence of $K_{2m}^W$ -fixed vectors

Let $\pi $ be an irreducible $\psi _E$ -generic tempered representation of $\mathrm {U}(W)$ with the L-parameter $\phi _\pi $ and the central character $\omega _\pi $ . Consider its theta lift $\sigma = \theta _\psi (\pi )$ . It is an irreducible generic tempered representation of $\mathrm {U}(V)$ with L-parameter $\phi _\sigma = \phi _\pi \chi \oplus \mathbf {1}$ . In particular, $c(\phi _\sigma ) = c(\phi _\pi )$ so that $\sigma ^{K_{2m}^V} \not = 0$ for $2m \geq c(\phi _\pi )$ by Theorem 2.1. Since $\omega _\sigma = \omega _\pi $ , we see that $\sigma ^{J_{2m}^V} \not = 0$ if $2m \geq c(\phi _\pi )$ and $\omega _\pi |_{1+\mathfrak {p}_E^m} = \mathbf {1}$ .

Set $\omega _\psi = \omega _{\psi ,V,W}$ . By the definition of theta lifts, we have a $\mathrm {U}(V) \times \mathrm {U}(W)$ -equivariant surjective map

$$\begin{align*}\Phi \colon \omega_\psi \twoheadrightarrow \sigma \boxtimes \pi. \end{align*}$$

Proposition 5.6. Set $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ . Suppose that $2m>0$ and that $\omega _\pi $ is trivial on $1+\mathfrak {p}_E^m$ . For any sign $\epsilon \in \{\pm 1\}$ , there exists $t \in \mathfrak {p}_E^{-m}$ such that $\Phi (\phi _{t,f_{\epsilon n}}) \not = 0$ . In particular, $\pi ^{K_{2m}^W} \not = 0$ .

Proof. We realize $\omega _\psi $ on the lattice model $\mathcal {S}(A)$ . Since $\Sigma \mapsto \Sigma ^{J_{2m}^V}$ is an exact functor on the category of smooth representations $\Sigma $ of $\mathrm {U}(V)$ , we obtain a $\mathrm {U}(W)$ -equivariant surjective map

$$\begin{align*}\Phi \colon \mathcal{S}(A)^{J_{2m}^V} \twoheadrightarrow \sigma^{J_{2m}^V} \boxtimes \pi. \end{align*}$$

By Proposition 5.3 together with Lemma 5.4, its restriction to $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V}$ is still nonzero. Since $\sigma ^{K_{2m-2}^V} = 0$ , this map factors through the restriction of the projection $\mathcal {S}(A)_{M_{2m}} \twoheadrightarrow \mathcal {S}(A)_{M_{2m} \setminus M_{2m-2}}$ . Hence, by Lemma 5.5, there exists $t \in E^\times $ with $\mathrm {ord}(t) = -m$ such that $\Phi (\phi _{t,f_n}) \not = 0$ . Since $\phi _{t,f_n}$ is fixed by $J_{2m}^V \times K_{2m}^W$ , we have $\Phi (\phi _{t,f_n}) \in \sigma ^{J_{2m}^V} \boxtimes \pi ^{K_{2m}^W}$ so that $\pi ^{K_{2m}^W} \not = 0$ . By the same argument, one can show that $\Phi (\phi _{t,f_{-n}}) \not = 0$ for some $t \in \mathfrak {p}_E^{-m}$ .

5.5 Proof of Theorem 2.2 (3)

The goal of the rest of this section is to show that $\pi _\psi ^{K_{2m}^W} \not = 0$ if $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ and if $\omega _\pi $ is trivial on $1+\mathfrak {p}_E^m$ . If $2m = c(\phi _\pi ) = 0$ , then $\pi $ is unramified (with respect to the hyperspecial maximal compact subgroup $K_0^W$ ), and the Casselman–Shalika formula [6] shows that $\pi _\psi ^{K_{0}^W} \not = 0$ . See Remark 4.1. Hence, we may assume that $c(\phi _\pi )> 0$ so that $2m> 0$ .

We need further notations. Set

$$\begin{align*}X = \bigoplus_{i=1}^n Ee_i,\quad V_0 = E e_0, \quad X^* = \bigoplus_{i=1}^n Ee_{-i}. \end{align*}$$

Hence, $V = X \oplus V_0 \oplus X^*$ . For $a \in \mathrm {GL}(X)$ , $b \in \mathrm {Hom}(V_0,X)$ and $c \in \mathrm {Hom}(X^*,X)$ , we define $a^* \in \mathrm {GL}(X^*)$ , $b^* \in \mathrm {Hom}(X^*,V_0)$ and $c^* \in \mathrm {Hom}(X^*,X)$ so that

$$\begin{align*}\left\langle ax, x' \right\rangle_V = \left\langle x,a^*x' \right\rangle_V, \quad \left\langle be_0, x' \right\rangle_V = \left\langle e_0,b^*x' \right\rangle_V, \quad \left\langle cx', x" \right\rangle_V = \left\langle x', c^*x" \right\rangle_V \end{align*}$$

for $x \in X$ and $x',x" \in X^*$ . For $a \in \mathrm {GL}(X)$ , $b \in \mathrm {Hom}(V_0,X)$ and

$$\begin{align*}c \in \mathrm{Herm}(X^*,X) = \{c \in \mathrm{Hom}(X^*,X) \;|\; c^*=-c\}, \end{align*}$$

we put

$$ \begin{align*} \mathbf{m}_X(a) &= \begin{pmatrix} a && \\ &\mathbf{1}_{V_0}& \\ && (a^*)^{-1} \end{pmatrix}, \\ \quad \mathbf{n}_1(b) &= \begin{pmatrix} \mathbf{1}_X & b & -\frac{1}{2}bb^*\\ &\mathbf{1}_{V_0}& b^* \\ && \mathbf{1}_{X^*} \end{pmatrix}, \\ \mathbf{n}_2(c) &= \begin{pmatrix} \mathbf{1}_X & & c \\ &\mathbf{1}_{V_0}& \\ && \mathbf{1}_{X^*} \end{pmatrix}. \end{align*} $$

These are elements in $\mathrm {U}(V)$ .

Similarly, set

$$\begin{align*}Y = \bigoplus_{i=1}^n Ef_i, \quad Y^* = \bigoplus_{i=1}^n Ef_{-i} \end{align*}$$

so that $W = Y \oplus Y^*$ . For $a \in \mathrm {GL}(Y)$ and $c \in \mathrm {Hom}(Y^*,Y)$ , we define $a^* \in \mathrm {GL}(X^*)$ and $c^* \in \mathrm {Hom}(Y^*,Y)$ so that

$$\begin{align*}\left\langle ay, y' \right\rangle_W = \left\langle y,a^*y' \right\rangle_W, \quad \left\langle cy', y" \right\rangle_W = \left\langle y', c^*y" \right\rangle_W \end{align*}$$

for $y \in Y$ and $y',y" \in Y^*$ . For $a \in \mathrm {GL}(Y)$ and

$$\begin{align*}c \in \mathrm{Herm}(Y^*,Y) = \{c \in \mathrm{Hom}(Y^*,Y) \;|\; c^*=-c\}, \end{align*}$$

we put

$$\begin{align*}\mathbf{m}_Y(a) = \begin{pmatrix} a & \\ & (a^*)^{-1} \end{pmatrix}, \quad \mathbf{n}(c) = \begin{pmatrix} \mathbf{1}_Y & c \\ & \mathbf{1}_{Y^*} \end{pmatrix}. \end{align*}$$

These are elements in $\mathrm {U}(W)$ .

Define $a_\delta \in \mathrm {GL}(X)$ by

$$\begin{align*}a_\delta \colon e_i \mapsto \delta^{-i} e_i \end{align*}$$

for $-n \leq i \leq n$ . If we fix a nonzero Whittaker functional $l_\sigma \in \mathrm {Hom}_{N_{2n+1}}(\sigma , \psi _E)$ for $\sigma $ , then $l_{\sigma }' = l_\sigma \circ \sigma (\mathbf {m}_X(a_\delta ))$ is a nonzero Whittaker functional with respect to the character $\psi _{E}^\delta \colon N_{2n+1} \rightarrow \mathbb {C}^\times $ given by

$$\begin{align*}\psi_{E}^\delta(u) = \psi_E\left( \delta^{-1} \sum_{i=1}^{n} \left\langle u e_{i-1}, e_{-i} \right\rangle_V \right). \end{align*}$$

This is the generic character considered in [8].

Now we fix $t \in E^\times $ with $\mathrm {ord}(t) = -m$ such that $\Phi (\phi _{t, f_{-n}}) \not = 0$ . This belongs to $\sigma ^{J_{2m}^V} \boxtimes \pi $ . Note that $\mathbf {m}_X(t \cdot \mathbf {1}_X) K_{2m}^V \mathbf {m}_X(t \cdot \mathbf {1}_X)^{-1}$ is the compact group $K_{n,2m}$ considered in [8]. In particular, the Whittaker functional

$$\begin{align*}l^{\prime}_{\sigma,t} = l^{\prime}_\sigma \circ \sigma(\mathbf{m}_X(t \cdot \mathbf{1}_X)) = l_\sigma \circ \sigma(\mathbf{m}_X(a_\delta t)) \end{align*}$$

with respect to

$$\begin{align*}\psi_{E,t}^\delta \colon N_{2n+1} \ni u \mapsto \psi_E^{\delta}(\mathbf{m}_X(t \cdot \mathbf{1}_X) \cdot u \cdot \mathbf{m}_X(t \cdot \mathbf{1}_X)^{-1}) \in \mathbb{C}^\times \end{align*}$$

is nonzero on $\sigma ^{K_{2m}^V}$ by [8, Theorem 1.4, Lemma 7.5]. Therefore, the image $\phi _{t, f_{-n}}$ under the composition of $N_{2n+1} \times \mathrm {U}(W)$ -equivariant maps

is nonzero.

By the same argument as the proof of [18, Proposition 2.3], one can prove that the maximal quotient of $\omega _\psi $ on which $N_{2n+1}$ acts by $\psi _{E,t}^\delta $ is isomorphic to the compact induction $\mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ , where $N_{2n}'$ is the unipotent radical of the Borel subgroup of $\mathrm {U}(W)$ stabilizing the flag

$$\begin{align*}E f_1 \subset E f_1 \oplus E f_2 \subset \dots \subset E f_1 \oplus \dots \oplus E f_n = Y, \end{align*}$$

and $\mu $ is a character of $N_{2n}'$ given by

$$\begin{align*}\mu(u) = \psi_E \left( \sum_{i=1}^n \left\langle u f_{i+1}, f_{-i} \right\rangle + N_{E/F}(t)\left\langle u f_{-n}, f_{-n} \right\rangle \right). \end{align*}$$

Here, we note that $N_{2n}'$ differs from $N_{2n}$ defined in Section 2.2.

Hence, the map

factors through $\omega _\psi \rightarrow \mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ . Namely, we have a nonzero $\mathrm {U}(W)$ -equivariant map

$$\begin{align*}\mathrm{ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu) \rightarrow \pi. \end{align*}$$

The following is a key lemma, which will be proven in Section 5.7 below.

Lemma 5.7. Let $\widetilde {F}_{t,f_{-n}} \in \mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ be the image of $\phi _{t, f_{-n}} \in \mathcal {S}(A)$ . Then $\widetilde {F}_{t,f_{-n}}$ is right ${}^tK_{2m}^W$ -invariant and

$$\begin{align*}\mathrm{Supp}(\widetilde{F}_{t,f_{-n}}) = N_{2n}' \cdot {}^tK_{2m}^W. \end{align*}$$

Note that having a $\mathrm {U}(W)$ -equivariant map

$$\begin{align*}\mathrm{ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu) \rightarrow \pi \end{align*}$$

is equivalent to giving a $\mathrm {U}(W)$ -equivariant map

$$\begin{align*}\pi^\vee \rightarrow \mathrm{Ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu^{-1}). \end{align*}$$

These are related as follows. Suppose that $\mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu ) \ni \widetilde {F} \mapsto v \in \pi $ corresponds to $\pi ^\vee \ni v' \mapsto W \in \mathrm {Ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu ^{-1})$ . Then

$$\begin{align*}(v,v')_\pi = \int_{N_{2n}' \backslash \mathrm{U}(W)} \widetilde{F}(g) W(g) dg. \end{align*}$$

By Lemma 5.7, there exists $\widetilde {F} \in \mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ such that

• • its image v in $\pi $ is nonzero;

• • $\widetilde {F}$ is right ${}^tK_{2m}^W$ -invariant;

• • $\mathrm {Supp}(\widetilde {F}) = N_{2n}' \cdot {}^tK_{2m}^W$ .

Hence, $v \in \pi ^{{}^tK_{2m}^W}$ . One can take $v' \in (\pi ^\vee )^{{}^tK_{2m}^W}$ such that $(v,v')_\pi \not = 0$ . Let $W \in \mathrm {Ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu ^{-1})$ be the image of $v'$ . Then W is right ${}^tK_{2m}^W$ -invariant, and

$$\begin{align*}0 \not= (v,v')_\pi = \int_{N_{2n}' \backslash \mathrm{U}(W)} \widetilde{F}(g) W(g) dg = c\widetilde{F}(\mathbf{1}) W(\mathbf{1}) \end{align*}$$

for some constant $c> 0$ . Hence, $W(\mathbf {1}) \not = 0$ . Moreover, since $\mathbf {v}(0,0;z) \in N_{2n}'$ , we have

$$\begin{align*}W(\mathbf{v}(0,0;z)) = \mu^{-1}(\mathbf{v}(0,0;z)) W(\mathbf{1}) = \psi^{-1}(N_{E/F}(t)z) W(\mathbf{1}) \end{align*}$$

for $z \in F$ . Therefore, via $v' \mapsto W \mapsto W(\mathbf {1})$ , we conclude that

$$\begin{align*}(\pi^\vee)^{{}^tK_{2m}^W}_{\psi^{\prime-1}} \not= 0, \end{align*}$$

where we put $\psi '(z) = \psi (N_{E/F}(t)z)$ . Since

$$\begin{align*}{}^tK_{2m}^W = \begin{pmatrix} t && \\ &\mathbf{1}_{2n-2}& \\ &&\overline{t}^{-1} \end{pmatrix}^{-1} K_{2m}^W \begin{pmatrix} t && \\ &\mathbf{1}_{2n-2}& \\ && \overline{t}^{-1} \end{pmatrix}, \end{align*}$$

as in Section 2.3, we have

$$\begin{align*}(\pi^\vee)^{K_{2m}^W}_{\psi^{-1}} \cong (\pi^\vee)^{{}^tK_{2m}^W}_{\psi^{\prime-1}} \not= 0. \end{align*}$$

Since $\pi $ is $\psi _E$ -generic if and only if $\pi ^\vee $ is $\psi _E^{-1}$ -generic, by replacing $\pi $ and $\psi $ with $\pi ^\vee $ and $\psi ^{-1}$ , respectively, we conclude that

$$\begin{align*}\pi^{K_{2m}^W}_\psi \not= 0. \end{align*}$$

Therefore, Theorem 2.2 (3) is reduced to proving Lemma 5.7.

5.6 Mixed model

To show Lemma 5.7, we review the argument in the proof of [18, Proposition 2.3]. For this, we use another model of the Weil representation $\omega _\psi = \omega _{\psi ,V,W}$ of $\mathrm {U}(V) \times \mathrm {U}(W)$ . It is known that the Weil representation $\omega _\psi $ can be realized on the space $\mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ , which is called a mixed model. See, for example, [10, Section 7.4]. Let us recall some formulas for the action of $\mathrm {U}(V) \times \mathrm {U}(W)$ on this space.

For $\varphi _1 \otimes \varphi _2 \in \mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ and $(x,y) \in (X^* \otimes W) \times (V_0 \otimes Y^*)$ ,

$$ \begin{align*} &\omega_\psi(g)(\varphi_1 \otimes \varphi_2)(x,y) = \varphi_1(g^{-1}x) \cdot \omega_\psi^0(g)\varphi_2(y), \quad g \in \mathrm{U}(W), \\ &\omega_\psi(h_0)(\varphi_1 \otimes \varphi_2)(x,y) = \varphi_1(x) \cdot \omega_\psi^0(h_0)\varphi_2(y), \quad h_0 \in \mathrm{U}(V_0), \\ &\omega_\psi(\mathbf{m}_X(a))(\varphi_1 \otimes \varphi_2)(x,y) = \chi_W(\det a) |\det a|^n \varphi_1(a^* x) \cdot \varphi_2(y), \quad a \in \mathrm{GL}(X), \\ &\omega_\psi(\mathbf{n}_1(b))(\varphi_1 \otimes \varphi_2)(x,y) = \varphi_1(x) \cdot \rho_\psi^0(b^*x, 0)\varphi_2(y), \quad b \in \mathrm{Hom}(V_0,X), \\ &\omega_\psi(\mathbf{n}_2(c))(\varphi_1 \otimes \varphi_2)(x,y) = \psi\left(\frac{1}{2}\left\langle cx,x \right\rangle\right) \varphi_1(x) \cdot \varphi_2(y), \quad c \in \mathrm{Herm}(X^*,X). \end{align*} $$

Here, $\mathcal {S}(V_0 \otimes Y^*)$ is regarded as the Schrödinger model of

• • the irreducible representation $\rho _\psi ^0$ of the Heisenberg group $H(V_0 \otimes W)$ on $\mathcal {S}(V_0 \otimes Y^*)$ with the central character $\psi $ ; and

• • the Weil representation $\omega _\psi ^0$ of $\mathrm {U}(V_0) \times \mathrm {U}(W)$ .

Hence, for $\varphi _2 \in \mathcal {S}(V_0 \otimes Y^*)$ and $y \in V_0 \otimes Y^*$ , we have

$$\begin{align*}\rho_\psi^0((y_+ + y_-,t))\varphi_2(y) = \psi\left( t + \left\langle y,y_+ \right\rangle + \frac{1}{2}\left\langle y_-,y_+ \right\rangle \right) \varphi_2(y+y_-) \end{align*}$$

for $y_+ \in V_0 \otimes Y$ and $y_- \in V_0\otimes Y^*$ , and

$$ \begin{align*} &\omega_\psi^0(\mathbf{m}_Y(a))\varphi_2(y) = \chi(\det a)|\det a|^{\frac{1}{2}}\varphi(a^*y), \quad a \in \mathrm{GL}(Y), \\ &\omega_\psi^0(\mathbf{n}(c))\varphi_2(y) = \psi\left(\frac{1}{2} \left\langle cy,y \right\rangle\right) \varphi_2(y) \quad c \in \mathrm{Herm}(Y^*,Y). \end{align*} $$

Moreover, $\omega _\psi ^0(J_{2n})\varphi _2$ is given by a Fourier transform of $\varphi _2$ . For more precision, see [10, Section 7.4].

For $\varphi _1 \otimes \varphi _2 \in \mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ , define

$$\begin{align*}F_{\varphi_1 \otimes \varphi_2}(g) = \varphi_1(g^{-1} x_0) \cdot \omega_\psi^0(g)\varphi_2(y_0), \end{align*}$$

where we set

$$\begin{align*}x_0 = \sum_{i=1}^n \frac{1}{2\delta} e_{-i} \otimes f_{n+1-i}, \quad y_0 = t e_0 \otimes f_{-n}. \end{align*}$$

Let $Q_{2n} = M_{2n,S}N_{2n,S}$ be the Siegel parabolic subgroup of $\mathrm {U}(W)$ stabilizing Y, where $M_{2n,S} = \{\mathbf {m}_Y(a) \;|\; a \in \mathrm {GL}(Y)\}$ is its Levi subgroup, and $N_{2n,S}$ is its unipotent radical. Note that $N_{2n,S} \subset N_{2n}'$ . We regard $\mu $ as a character of $N_{2n,S}$ by the restriction. For $u \in N_{2n,S}$ , since $u^{-1}x_0 = x_0$ and

$$ \begin{align*} \psi\left(\frac{1}{2}\left\langle uy_0,y_0 \right\rangle\right) &=\psi_E\left(\left\langle te_0, te_0 \right\rangle_V \left\langle uf_{-n},f_{-n} \right\rangle_W \right) \\&= \psi_E\left(N_{E/F}(t) \left\langle uf_{-n},f_{-n} \right\rangle_W \right) = \mu(u), \end{align*} $$

we see that $F_{\varphi _1 \otimes \varphi _2}(g) \in \mathrm {ind}_{N_{2n,S}}^{\mathrm {U}(W)}(\mu )$ . Note that $\mathbf {n}_2(c)$ acts trivially on $F_{\varphi _1 \otimes \varphi _2}$ for $c \in \mathrm {Herm}(X^*,X)$ since Y is totally isotropic. However, for $b \in \mathrm {Hom}(V_0,X)$ , since $\mathbf {n}_1(b)$ commutes with $g \in \mathrm {U}(W)$ , we see that

$$ \begin{align*} F_{\omega_\psi(\mathbf{n}_1(b))(\varphi_1 \otimes \varphi_2)}(g) &= \omega_\psi(g) \circ \omega_\psi(\mathbf{n}_1(b)) (\varphi_1\otimes \varphi_2) (x_0,y_0) \\&= \omega_\psi(\mathbf{n}_1(b)) \circ \omega_\psi(g) (\varphi_1\otimes \varphi_2) (x_0,y_0) \\&= \rho_\psi^0(b^*x_0,0) \circ \omega_\psi(g) (\varphi_1\otimes \varphi_2) (x_0,y_0) \\&= \psi(\left\langle y_0,b^*x_0 \right\rangle) \omega_\psi(g) (\varphi_1\otimes \varphi_2) (x_0,y_0). \end{align*} $$

Since $\overline {\delta } = -\delta $ and $\left \langle f_{-n},f_n \right \rangle _W = -1$ , we have

$$ \begin{align*} \psi(\left\langle y_0,b^*x_0 \right\rangle) &= \psi_E\left( \sum_{i=1}^n \left\langle te_0,b^* \delta^{-1} e_{-i} \right\rangle_V \left\langle f_{-n}, f_{n+i-1} \right\rangle_W \right) \\&= \psi_E( \delta^{-1} t \left\langle be_0,e_{-1} \right\rangle_V) = \psi_{E,t}^\delta(\mathbf{n}_1(b)). \end{align*} $$

Hence, $\mathbf {n}_1(b)$ acts on $F_{\varphi _1 \otimes \varphi _2}$ by $\psi _{E,t}^\delta $ .

Define a map

$$\begin{align*}\mathrm{ind}_{N_{2n,S}}^{\mathrm{U}(W)}(\mu) \rightarrow \mathrm{ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu) \end{align*}$$

by

$$\begin{align*}F \mapsto \widetilde{F}(g) = \int_{N_{2n,S} \backslash N_{2n}'}F(ug) \mu(u)^{-1} du. \end{align*}$$

Then by the same argument as in [18, Proposition 2.3], one can prove that the map $\varphi _1 \otimes \varphi _2 \mapsto \widetilde {F}_{\varphi _1 \otimes \varphi _2}$ realizes an isomorphism between the maximal quotient of $\omega _\psi $ on which $N_{2n+1}$ acts by $\psi _{E,t}^\delta $ and $\mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ .

5.7 Proof of Lemma 5.7

In this subsection, we prove Lemma 5.7. To do this, we relate two models of the Weil representation.

Let $\varphi _1^0 \in \mathcal {S}(X^* \otimes W)$ and $\varphi _2^0 \in \mathcal {S}(V_0 \otimes Y^*)$ be the characteristic functions of

$$\begin{align*}\left(\bigoplus_{i=1}^n \mathfrak{o}_Ee_{-i} \right) \otimes \left( \bigoplus_{i=1}^n \mathfrak{o}_Ef_{i} \oplus \bigoplus_{i=1}^n \mathfrak{o}_Ef_{-i} \right), \quad \mathfrak{o}_E e_0 \otimes \left( \bigoplus_{i=1}^n \mathfrak{o}_Ef_{-i} \right), \end{align*}$$

respectively. Then the action $\rho = \rho _\psi $ of the Heisenberg group $H(\mathbb {W})$ on $\varphi _1^0 \otimes \varphi _2^0$ satisfies that

$$\begin{align*}\rho(a,t)(\varphi_1^0 \otimes \varphi_2^0) = \psi(t) \cdot \varphi_1^0 \otimes \varphi_2^0 \end{align*}$$

for $(a,t) \in A \oplus F$ . Moreover, the lattice model $\mathcal {S}(A)$ and the mixed model $\mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ are related by the isomorphism

$$ \begin{align*} \mathcal{S}(A) &\xrightarrow{\sim} \mathcal{S}(X^* \otimes W) \otimes \mathcal{S}(V_0 \otimes Y^*),\\ \phi &\mapsto \int_{(A \oplus F) \backslash H(\mathbb{W})} \phi(h) \rho(h)^{-1}(\varphi_1^0 \otimes \varphi_2^0)(x,y) dh. \end{align*} $$

In particular, $\phi _{t,f_{-n}} \in \mathcal {S}(A)$ corresponds to

$$\begin{align*}\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)(x,y) = \varphi_1^0(x) \cdot \rho_\psi^0(t e_0 \otimes f_{-n},0)^{-1} \varphi_2^0(y) \end{align*}$$

in $\mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ since $\mathrm {Supp}(\phi _{t,f_{-n}}) = (A + te_0 \otimes f_{-n}) \oplus F$ . Therefore, under the map

$$\begin{align*}\mathcal{S}(A) \rightarrow \mathrm{ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu) \end{align*}$$

obtained above, the image of $\phi _{t,f_{-n}}$ is $\widetilde {F}_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}$ .

Now we prove Lemma 5.7.

Proof of Lemma 5.7.

First, we consider $F_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}$ . Note that it is left $N_{2n,S}$ -invariant and right ${}^tK_{2m}^W$ -invariant. We claim that if

$$\begin{align*}F_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(g) \not= 0, \end{align*}$$

then

$$\begin{align*}g \in N_{2n,S} \cdot \mathbf{m}_Y(a) \cdot {}^tK_{2m}^W \end{align*}$$

for some $a \in \mathrm {GL}(Y) \cong \mathrm {GL}_n(E)$ such that $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ and

$$\begin{align*}a^*f_{-n}-f_{-n} \in \bigoplus_{i=1}^n \mathfrak{p}_E^m f_{-i}. \end{align*}$$

By the Iwasawa decomposition, we have $\mathrm {U}(W) = Q_{2n} K_0^W$ . Let $K_S$ and $K_M$ be subgroups of $K_0^W$ defined by

By the Bruhat decomposition for a finite unitary group over $\mathfrak {o}_F/\mathfrak {p}_F$ , we have

$$ \begin{align*} K_0^W &= K_S K_M \cup K_S J_{2n}^{-1} K_M \\&= (K_S \cap Q_{2n}) K_M \cup (K_S \cap Q_{2n}) J_{2n}^{-1} K_M. \end{align*} $$

Since $J_{2n} \in K_0^W$ and $J_{2n}^{-1} K_M J_{2n} = {}^t K_M$ , by the multiplication of $J_{2n}$ from the right, we have

$$\begin{align*}K_0^W = (K_S \cap Q_{2n}) J_{2n} {}^tK_M \cup (K_S \cap Q_{2n}) {}^tK_M. \end{align*}$$

Hence,

$$ \begin{align*} \mathrm{U}(W) &= Q_{2n} J_{2n} {}^tK_M \cup Q_{2n} {}^tK_M \\&= N_{2n,S}M_{2n,S} J_{2n} {}^tK_M \cup N_{2n,S}M_{2n,S} {}^tK_M. \end{align*} $$

Therefore, we may assume that $g = \mathbf {m}_Y(a)J_{2n}k$ or $g = \mathbf {m}_Y(a)k$ for some $a \in \mathrm {GL}(Y)$ and $k \in {}^tK_M$ .

Assume that $g = \mathbf {m}_Y(a)J_{2n}k$ is in the former case. Since $\varphi _1^0$ and $\varphi _2^0$ are fixed by $K_{0}^W$ , and since $\omega _\psi ^0(g) \circ \rho _\psi ^0(h) \circ \omega _\psi ^0(g)^{-1} = \rho _\psi ^0(gh)$ , we have

$$ \begin{align*} F_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(g) &= \omega_\psi(g) \circ \rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)(x_0,y_0) \\&= \varphi_1(g^{-1}x_0) \cdot \omega_\psi^0(g) \rho_\psi^0(t e_0 \otimes f_{-n},0)^{-1}\varphi_2^0(y_0) \\&= \varphi_1(a^{-1}x_0) \cdot \omega_\psi^0(\mathbf{m}_Y(a)) \rho_\psi^0(t e_0 \otimes J_{2n} k f_{-n},0)^{-1} \varphi_2^0(y_0). \end{align*} $$

Note that $\varphi _1(a^{-1}x_0) \not = 0$ if and only if $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ . However, since $k \in {}^tK_M$ , if we write $J_{2n} k f_{-n} = y+y^*$ with $y \in Y$ and $y^* \in Y^*$ , then $y^* \in \oplus _{i=1}^n \mathfrak {p}_E f_{-i}$ . Up to a nonzero constant, $\omega _\psi ^0(\mathbf {m}_Y(a)) \rho _\psi ^0(t e_0 \otimes J_{2n} k f_{-n},0)^{-1} \varphi _2^0(y_0)$ is equal to

$$\begin{align*}\varphi_2^0( te_0 \otimes (a^*f_{-n} - y^*) ). \end{align*}$$

If this is nonzero, then we must have $t(a^*f_{-n} - y^*) \in \oplus _{i=1}^n \mathfrak {o}_E f_{-i}$ . When $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ , this implies that

$$\begin{align*}f_{-n} \in (a^*)^{-1}y^* + \bigoplus_{i=1}^n \mathfrak{p}_E^m f_{-i} \subset \bigoplus_{i=1}^n \mathfrak{p}_E f_{-i}. \end{align*}$$

This is impossible. Hence, we have $F_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}(g) = 0$ .

Next, we assume that $g = \mathbf {m}_Y(a)k$ is in the latter case. By the Iwahori decomposition, we may further assume that $kf_{-n} = f_{-n}$ . Then

$$ \begin{align*} F_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(g) &= \omega_\psi(g) \circ \rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)(x_0,y_0) \\&= \varphi_1(g^{-1}x_0) \cdot \omega_\psi^0(g) \rho_\psi^0(t e_0 \otimes f_{-n})^{-1}\varphi_2^0(y_0) \\&= \varphi_1(a^{-1}x_0) \cdot \omega_\psi^0(\mathbf{m}_Y(a)) \rho_\psi^0(t e_0 \otimes kf_{-n})^{-1} \varphi_2^0(y_0) \\&= \varphi_1(a^{-1}x_0) \cdot \omega_\psi^0(\mathbf{m}_Y(a)) \rho_\psi^0(t e_0 \otimes f_{-n})^{-1} \varphi_2^0(y_0). \end{align*} $$

Up to a nonzero constant, it is equal to

$$\begin{align*}\varphi_1(a^{-1}x_0) \cdot \varphi_2^0( te_0 \otimes (a^*f_{-n}-f_{-n}) ). \end{align*}$$

If this is nonzero, then $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ and

$$\begin{align*}a^*f_{-n}-f_{-n} \in \bigoplus_{i=1}^n \mathfrak{p}_E^m f_{-i}. \end{align*}$$

This proves the claim.

Now we consider $\widetilde {F}_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}$ . Note that it is left $N_{2n}'$ -invariant and right ${}^tK_{2m}^W$ -invariant. Suppose that $\widetilde {F}_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}(g) \not = 0$ . By the claim, we may assume that $g = \mathbf {m}_Y(a)$ with $a \in \mathrm {GL}(Y)$ satisfying the conditions in the claim. By the Iwasawa decomposition, we may further assume that $a = a_d a_0$ such that

• • $\left \langle a_d f_i, f_{-j} \right \rangle _W = \varpi ^{\lambda _i} \delta _{i,j}$ for some $\lambda _i \in \mathbb {Z}$ ;

• • $a_0 \in \mathrm {GL}_n(\mathfrak {o}_E)$ via $\mathrm {GL}(Y) \cong \mathrm {GL}_n(E)$ .

Since $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ , we have $\lambda _i \leq 0$ for $1 \leq i \leq n$ . Note that

$$\begin{align*}a^* f_{-n}-f_{-n} = a_0^* a_d^* f_{-n}-f_{-n} = a_0^* \varpi^{\lambda_n}f_{-n}-f_{-n}. \end{align*}$$

Since this is in $\oplus _{i=1}^n \mathfrak {p}_E^m f_{-i}$ , we have $\lambda _n = 0$ and $\mathbf {m}_Y(a_0) \in {}^tK_{2m}^W$ . Hence, we may assume that $a_0 = \mathbf {1}_X$ (i.e., $g = \mathbf {m}_Y(a_d)$ ). For $2 \leq i \leq n$ and $x \in \mathfrak {o}_E$ , define $u_i \in N_{2n}'$ so that

$$\begin{align*}u_i f_j - f_j = \left\{ \begin{aligned} &x \cdot f_{i-1} &\quad&\text{if } j=i, \\ &0 &\quad&\text{if } j\not=i. \end{aligned} \right. \end{align*}$$

Then $u_i \in {}^tK_{2m}^W$ . Hence,

$$ \begin{align*} 0 &\not= \widetilde{F}_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(\mathbf{m}_Y(a_d)) \\&= \widetilde{F}_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(\mathbf{m}_Y(a_d)u_i) \\&= \mu(\mathbf{m}_Y(a_d) u_i \mathbf{m}_Y(a_d)^{-1}) \widetilde{F}_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(\mathbf{m}_Y(a_d)) \end{align*} $$

so that $\mu (\mathbf {m}_Y(a_d) u_i \mathbf {m}_Y(a_d)^{-1}) = 1$ for any $x \in \mathfrak {o}_E$ . Note that

$$ \begin{align*} \mu(\mathbf{m}_Y(a_d) u_i \mathbf{m}_Y(a_d)^{-1}) &= \psi_E( \left\langle \mathbf{m}_Y(a_d) u_i \mathbf{m}_Y(a_d)^{-1} f_{i}, f_{-i+1} \right\rangle ) \\&= \psi_E( \varpi^{\lambda_{i-1}-\lambda_i}x). \end{align*} $$

Hence, $\psi _E( \varpi ^{\lambda _{i-1}-\lambda _i}x) = 1$ for any $x \in \mathfrak {o}_E$ . This implies that $\lambda _{i-1} \geq \lambda _i$ . In conclusion, we have

$$\begin{align*}0 \geq \lambda_1 \geq \dots \geq \lambda_{n-1} \geq \lambda_n = 0 \end{align*}$$

so that $\lambda _1=\dots =\lambda _n=0$ . This means that $a_d = \mathbf {1}_X$ . This completes the proof of Lemma 5.7.