We provide a microlocal necessary condition for distinction of admissible representations of real reductive groups in the context of spherical pairs.
Let
${\mathbf {G}}$
be a complex algebraic reductive group and
${\mathbf {H}}\subset {\mathbf {G}}$
be a spherical algebraic subgroup. Let
${\mathfrak {g}},{\mathfrak {h}}$
denote the Lie algebras of
${\mathbf {G}}$
and
${\mathbf {H}}$
, and let
${\mathfrak {h}}^{\bot }$
denote the orthogonal complement to
${\mathfrak {h}}$
in
${\mathfrak {g}}^*$
. A
${\mathfrak {g}}$
-module is called
${\mathfrak {h}}$
-distinguished if it admits a nonzero
${\mathfrak {h}}$
-invariant functional. We show that the maximal
${\mathbf {G}}$
-orbit in the annihilator variety of any irreducible
${\mathfrak {h}}$
-distinguished
${\mathfrak {g}}$
-module intersects
${\mathfrak {h}}^{\bot }$
. This generalises a result of Vogan [Vog91].
We apply this to Casselman–Wallach representations of real reductive groups to obtain information on branching problems, translation functors and Jacquet modules. Further, we prove in many cases that – as suggested by [Pra19, Question 1] – when H is a symmetric subgroup of a real reductive group G, the existence of a tempered H-distinguished representation of G implies the existence of a generic H-distinguished representation of G.
Many of the models studied in the theory of automorphic forms involve an additive character on the unipotent radical of the subgroup
$\bf H$
, and we have devised a twisted version of our theorem that yields necessary conditions for the existence of those mixed models. Our method of proof here is inspired by the theory of modules over W-algebras. As an application of our theorem we derive necessary conditions for the existence of Rankin–Selberg, Bessel, Klyachko and Shalika models. Our results are compatible with the recent Gan–Gross–Prasad conjectures for nongeneric representations [GGP20].
Finally, we provide more general results that ease the sphericity assumption on the subgroups, and apply them to local theta correspondence in type II and to degenerate Whittaker models.