Flags of sheaves, quivers and symmetric polynomials

Abstract

We study a quiver description of the nested Hilbert scheme of points on the affine plane and its higher rank generalization – that is, the moduli space of flags of framed torsion-free sheaves on the projective plane. We show that stable representations of the quiver provide an ADHM-like construction for such moduli spaces. We introduce a natural torus action and use equivariant localization to compute some of their (virtual) topological invariants, including the case of compact toric surfaces. We conjecture that the generating function of holomorphic Euler characteristics for rank one is given in terms of polynomials in the equivariant weights, which, for specific numerical types, coincide with (modified) Macdonald polynomials. From the physics viewpoint, the quivers we study describe a class of surface defects in four-dimensional supersymmetric gauge theories in terms of nested instantons.

Introduction

In this work, we are interested in studying a quiver description of the local model for nested Hilbert schemes on complex surfaces – that is, nested Hilbert schemes of points on the affine plane $\mathbb A^2$ . We are also interested in its natural higher-rank generalization, which is given by the moduli space of flags of framed torsion-free sheaves on the projective plane ${\mathbb P}^2$ . For reasons which we will explain later in this introduction, we call the quiver modelling our moduli problem the nested instantons quiver, which is shown in Figure 1. Algebraic constructions arising from stable representations of these quivers appear to be generalizations of Nakajima’s ADHM presentation of framed torsion-free sheaves on ${\mathbb P}^2$ and, as a special case, of Hilbert schemes of points on the affine plane; [43]. The possibility of exploiting an ADHM-type construction is very powerful by itself in that it provides an explicit construction for an a priori very complicated moduli space of sheaves in terms of purely linear algebraic data. Other constructions of this kind can also be found in the literature for different moduli spaces of sheaves on algebraic varieties (cf. for instance, [1, 14, 43, 12, 28, 29, 27, 2]). Moreover, there has been some recent interest surrounding moduli spaces of flags of sheaves on surfaces, and in particular nested Hilbert schemes, due to their relevance to the context of Vafa-Witten theory; [50, 51]. It is indeed known that monopole contributions to Vafa-Witten invariants reduce under virtual localization to the computation of invariants of such moduli spaces. The interested reader can refer to [24, 34] for computations in the rank-one case, involving nested Hilbert schemes. The deformation-obstruction theory and virtual cycle for the components of the monopole branch in Vafa-Witten theory giving rise to flags of higher rank sheaves were explicitly constructed in [48]. Nested Hilbert schemes on surfaces were interpreted in terms of degeneracy loci in [20, 21], where they are also shown to be equipped with a perfect obstruction theory. Similarly nested Hilbert schemes of points were also studied in [19], and a perfect obstruction theory and virtual cycles are explicitly constructed. Their application to reduced Donaldson-Thomas and Pandharipande-Thomas invariants are also discussed in [19, 18, 13].

Figure 1 The nested instantons quiver. A vector space $V_i\cong {\mathbb C}^{n_i}$ (resp. $W_i\cong {\mathbb C}^{r_i}$ ) is intended to be attached to each node labelled by $n_i$ (resp. by $r_i$ ).

Content of the Paper

In this paper, we concentrate on the study of representations of the nested instantons quiver with a single framing; namely, we choose the dimension vector for the framing to be $\mathbf r=(r,0,\dots ,0)$ , where r is the dimension of the leftmost framing node in Figure 1. We also study its relation to flags of framed torsion-free sheaves on ${{\mathbb P}}^2$ and nested Hilbert schemes, and we compute some relevant virtual invariants via equivariant localization. In the following, we give a summary of the results we obtained in this paper.

In §1, we start our analysis by constructing the moduli space of stable representations of the nested instantons quiver, which is characterized by the following.

Theorem (Theorem 1.8).

The moduli space ${\mathcal N}(r,{\mathbf n})$ of stable representation of the nested instantons quiver of numerical type $(r,{\mathbf n})$ is a quasi-projective variety over ${\mathbb C}$ equipped with a perfect obstruction theory.

We also prove that ${\mathcal N}(r,{\mathbf n})$ embeds into a Nakajima quiver variety $\mathcal M(r,{\mathbf n})$ , which is a smooth hyperkähler variety; see § 1.3.

In §2, we construct the moduli space $\mathcal F(r,\boldsymbol \gamma )$ of flags of framed torsion-free sheaves on ${{\mathbb P}}^2$ and prove the existence of an isomorphism with ${\mathcal N}(r,{\mathbf n})$ . As a particular case, we have the following.

Theorem (Theorem 2.4).

The moduli space of nested instantons ${\mathcal N}(1,{\mathbf n})$ is isomorphic to the nested Hilbert scheme of points on ${\mathbb C}^2$ ; namely,

$$\begin{align*}{\mathcal N}(1,{\mathbf n})={\mathbb X}_0/\!\!/_\chi\mathcal G\cong\operatorname{\mathrm{Hilb}}^{\hat{{\mathbf n}}}({\mathbb C}^2). \end{align*}$$

The moduli space of flags of sheaves is constructed by means of a functor $\mathsf F_{(r,\boldsymbol \gamma )}:{\mathsf {Sch}}_{{\mathbb C}}^{\mathrm {op}}\to {\mathsf {Sets}}$ describing flags of torsion-free sheaves on ${\mathbb P}^2$ in the following.

Proposition (Proposition 2.7).

The moduli functor $\mathsf F_{(r,\boldsymbol \gamma )}$ is representable. The (quasi-projective) variety representing $\mathsf F_{(r,\boldsymbol \gamma )}$ is the moduli space of flags of framed (coherent) torsion-free sheaves on ${\mathbb P}^2$ , denoted by $\mathcal F(r,\boldsymbol \gamma )$ .

Its isomorphism with ${\mathcal N}(r,{\mathbf n})$ is proved in the following theorem.

Theorem (Theorem 2.9).

The moduli space of stable representations of the nested instantons quiver is a fine moduli space isomorphic to the moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ : $\mathcal F(r,\boldsymbol \gamma )\cong {\mathcal N}(r,{\mathbf n})$ , as schemes, where $n_i=\gamma _i+\cdots +\gamma _N$ .

The ADHM construction of a particular class of flags of sheaves on ${\mathbb P}^2$ was given in [44], where their connection to shuffle algebras on K-theory is also studied. Moreover, the construction of the functor $\mathsf F_{(r,\boldsymbol \gamma )}$ shows that the moduli space of nested instantons is isomorphic to a relative nested ${\mathsf {Quot}}$ -scheme. Perfect obstruction theories on ${\mathsf {Quot}}$ -schemes and the description of their local model in terms of a quiver are discussed in [2, 47], while nested quot schemes on curves appeared in [38], where their cohomology was studied, and in [39], where their motivic invariants where computed. More in general, the smoothness of nested quot schemes was studied in [40].

In §3, we proceed to the evaluation of the relevant virtual invariants via equivariant localization. Indeed, we show how on ${\mathcal N}(r,{\mathbf n})$ there is a natural action of an algebraic torus ${\mathbb T}=({\mathbb C}^{\ast })^2\times ({\mathbb C}^{\ast })^r$ and a ${\mathbb T}$ -equivariant lift of the perfect obstruction theory. The classification of the ${\mathbb T}$ -fixed locus of ${\mathcal N}(r,{\mathbf n})$ is presented in the following proposition.

Proposition (Proposition 3.4).

The ${\mathbb T}$ -fixed locus of ${\mathcal N}(r,{\mathbf n})$ is in bijection with $(N+1)$ -tuples of nested coloured partitions $\boldsymbol \mu _1\subseteq \cdots \subseteq \boldsymbol \mu _N\subseteq \boldsymbol \mu _0$ , with $|\boldsymbol \mu _0|=n_0$ and $|\boldsymbol \mu _{i>0}|=n_0-n_i$ .

In §3.2, we compute the generating function of the virtual Euler characteristics of ${\mathcal N}(1,{\mathbf n})$ ; see Equation (3.2.4) for the explicit combinatorial formula. We conjecture that, by summing over the nested partitions, this invariant is expressed in terms of polynomials:

Conjecture (Conjecture 3.6).

The equivariant virtual Euler characteristic

$$ \begin{align*}\chi^{\mathrm{vir}}({\mathcal N}(1,n_0,\dots,n_N);\mathfrak{q}_1^{-1},\mathfrak q_2^{-1})=\sum_{\mu_0}P_{\mu_0}(q,t)/N_{\mu_0}(q,t)\end{align*} $$

is such that

$$\begin{align*}P_{\mu_0}(q,t)=\frac{Q_{\mu_0}(q,t)}{(1-qt)^N}, \end{align*}$$

with $Q_{\mu _0}(q,t)\in \mathbb Z[q,t]$ and $N_{\mu _0}(q,t)$ as in (3.2.5).¹

For specific profiles of the nesting, these polynomials are conjectured to be determined by modified (or transformed) Macdonald polynomials, as defined in [25, Eq. 2.18]; cf. also (3.2.10).

Conjecture (Conjecture 3.8).

When $|\mu _0|=|\mu _N|+1=|\mu _{N-1}|+2=\cdots =|\mu _1|+N$ , we have

$$\begin{align*}\begin{aligned} Q_{\mu_0}(q,t)&=\left\langle h_{\mu_0}(\mathbf x),\widetilde{H}_{\mu_0}(\mathbf x;q,t)\right\rangle, \end{aligned} \end{align*}$$

where $h_\mu $ are the complete symmetric functions, the Hall pairing $\langle -,-\rangle $ is such that $\langle h_\mu ,m_\lambda \rangle =\delta _{\mu ,\lambda }$ (cf. [37]) and $\widetilde H_{\mu }(\mathbf x;q,t)$ are the modified Macdonald polynomials.

In §3.3, we compute the generating function of the virtual $\chi _{-y}$ -genus of ${\mathcal N}(1,{\mathbf n})$ (see Equation (3.3.3)) and of ${\mathcal N}(r,{\mathbf n})$ (see Equation (3.3.7)). We also conjecture, by specializing at $y=1$ , that the virtual Euler characteristic of ${\mathcal N}(1,{\mathbf n})$ reproduces the generating function of nested partitions of arbitrary length. These results are further generalized in §3.4, where we compute the generating function of the virtual elliptic genus of ${\mathcal N}(1,{\mathbf n})$ (see Equation (3.4.1)) and of ${\mathcal N}(r,{\mathbf n})$ (see Equation (3.4.2)).

Finally, in §4, we extend our results to ${{\mathbb P}}^2$ and ${{\mathbb P}}^1\times {{\mathbb P}}^1$ in the case of $\chi _{-y}$ -genera; see formulae (4.1.2) and (4.2.1), respectively. Notice that the choice of computing $\chi _{-y}$ -genera was due to the expected simple polynomial dependence in y. Everything which was done in this context is, however, completely general and holds for any complex genus.

Relation to String Theory

It is very well known that the interplay between string theory and geometry provided interesting results for both the parts at play during the last decades. A prototypical example of this phenomenon is given by the close relationship between BPS-bound states counting and enumerative geometry. On a threefold X, for instance, the problem of counting BPS-bound states in the context of D0-D2-D6 brane systems is translated in the computations of virtual invariants of Hilbert schemes of points and curves on X (i.e., the Donaldson-Thomas theory of X). The story goes in a quite similar way also in lower (and, up to a certain point, also higher) dimensions. There, instanton counting on complex surfaces is still closely related to the geometry and the distinguished algebraic structures of Hilbert schemes and moduli spaces of sheaves; [43]. As it turns out, these correspondences between physical theories and geometrical constructions can be generalized to different setups. For instance, one may want to study the geometry of instanton moduli spaces in presence of surface defects. In [5], we introduce the moduli space of nested instantons by studying surface defects in supersymmetric gauge theory on $T^2\times {\mathcal C}_{g,k}$ , where $T^2$ is a real two torus and ${\mathcal C}_{g,k}$ a genus g complex projective curve with k marked points.

The D-brane setup engineering the surface defect is described in [5], and its analysis naturally led to a description in terms of representations of a quiver in the category of vector spaces – the D-branes being the objects and the open strings being the morphisms. Let us briefly resume the D-brane geometry and its relation with the relevant mathematical problems. One considers type IIB supersymmetric background given by $T^2\times T^{*}{\mathcal C}_{g,k}\times {{\mathbb C}}^2$ , with r D7-branes located at points of the fiber of the cotangent bundle and n D3-branes along $T^2\times {\mathcal C}_{g,k}$ . The low energy effective theory of the D7-branes should correspond to a generalization of equivariant higher rank Donaldson-Thomas theory [15] on the non-CY four-fold $T^2\times {\mathcal C}_{g,k}\times {{\mathbb C}}^2$ , while the low energy effective theory of the D3-branes is equivariant Vafa-Witten theory on $T^2\times {\mathcal C}_{g,k}$ ; [53]. In the chamber of small volume of ${\mathcal C}_{g,k}$ , the effective theory describing the surface defect is encoded in the theory of maps from $T^2$ to the moduli space of stable representations of the comet-shaped quiver displayed in Figure 2. For $k=1$ , this is described by the total space of a bundle ${\mathcal V}_g$ over the nested instanton moduli space, which, in turn, is the moduli space of stable representations of the quiver displayed in Figure 1. Let us remark that virtual invariants of $\mathcal V_g$ have a connection to the cohomology of character varieties of punctured Riemann surfaces, and in particular to the conjecture proposed in [26] whose physical interpretation was provided in [11]. The interested reader can find the details in [5].

Figure 2 The comet-shaped quiver. As in Figure 1, to each node labelled by $n_i^{(j)}$ (resp. $r_i^{(j)}$ ) is associated a vector space $V_i^{(j)}\cong {\mathbb C}^{n_i^{(j)}}$ (resp. $W_i^{(j)}\cong {\mathbb C}^{r_i^{(j)}}$ ).

1 The nested instantons quiver

1.1 Quiver representations and stability

In the following, we will mainly be interested in studying the following quiver, which will be called the nested instantons quiver,

(1.1.1)

with relations

$$ \begin{align*} \begin{aligned} &[\alpha_0,\beta_0]+\xi\eta=0,\quad [\alpha_i,\beta_i]=0,\quad \alpha_{i-1}\phi_i-\phi_i\alpha_i=0=\beta_{i-1}\phi_i-\phi_i\beta_i\\ &\gamma_i\alpha_{i-1}-\alpha_i\gamma_i=0=\gamma_i\beta_{i-1}-\beta_i\gamma_i,\quad \phi_i\gamma_i=0,\quad \eta\phi_1=0,\quad \gamma_1\xi=0. \end{aligned} \end{align*} $$

Given a tuple of vector spaces $(W,V_0,\dots ,V_N)$ , one for each node of the quiver (1.1.1) and such that $\dim W=r$ , $\dim V_i=n_i$ , let ${\mathbb X}(r,{\mathbf n})$ be the linear space of representations of the quiver (1.1.1) with fixed dimension vector $(r,{\mathbf n})$ ; namely,

(1.1.2) $$ \begin{align} \begin{aligned} {\mathbb X}(r,{\mathbf n})=&\operatorname{\mathrm{End}} V_0^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(W,V_0)\oplus\operatorname{\mathrm{Hom}}(V_0,W)\oplus\operatorname{\mathrm{End}}(V_1)^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(V_1,V_0)\\ &\oplus\operatorname{\mathrm{Hom}}(V_0,V_1)\oplus\cdots\oplus\operatorname{\mathrm{End}}(V_{N})^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(V_{N},V_{N-1})\oplus\operatorname{\mathrm{Hom}}(V_{N-1},V_{N}). \end{aligned} \end{align} $$

A representation of numerical type $(r,{\mathbf n})$ of (1.1.1) with relations in the category of vector spaces will be given by the datum of a pair $X=(\mathbb W,h)$ , with $\mathbb W=(W,V_0,\dots ,V_{N})$ such that $\dim W=r$ and $\dim V_i=n_i$ , and

$$ \begin{align*}{\mathbb X}_0(r,{\mathbf n})\ni h=(B_1^0,B_2^0,I,J,B_1^1,B_2^1,F^1,G^1,\dots),\end{align*} $$

where ${\mathbb X}_0(r,{\mathbf n})\subset {\mathbb X}(r,{\mathbf n})$ is the closed subspace of ${\mathbb X}(r,{\mathbf n})$ whose morphisms satisfy the nested ADHM equations (1.1.3)

(1.1.3) $$ \begin{align} \begin{aligned} &[B_1^0,B_2^0]+IJ=0,\quad [B_1^i,B_2^i]=0,\quad B_1^{i-1}F^i-F^iB_1^i=0=B_2^{i-1}F^i-F^iB_2^i\\ &G^iB_1^{i-1}-B_1^iG^i=0=G^iB_2^{i-1}-B_2^iG^i,\quad F^iG^i=0,\quad JF^1=0,\quad G^1I=0. \end{aligned} \end{align} $$

In the following, we need to address the problem of King stability for representations of the nested instantons quiver. The definition of stability we will use follows from considering the moduli space of representations of the framed nested instantons quiver with a moduli space of representations of an auxiliary extended quiver with relations; [33, 12].

Definition 1.1. Let $\Theta =(\boldsymbol \theta ,\theta _\infty )\in \mathbb Q^{N+2}$ be such that $\Theta (X)={\mathbf n}\cdot \boldsymbol \theta +r\theta _{\infty }=0$ . We will say that a framed representation X of (1.1.1) is $\Theta $ -semistable if

• ○ $\forall 0\neq \tilde X\subset X$ of numerical type $(0,\tilde {{\mathbf n}})$ , we have $\Theta (\tilde X)=\boldsymbol \theta \cdot \tilde {{\mathbf n}}\le 0$ ;

• ○ $\forall 0\neq \tilde X\subset X$ of numerical type $(r,\tilde {{\mathbf n}})$ , we have $\Theta (\tilde X)=\boldsymbol \theta \cdot \tilde {{\mathbf n}}+r\theta _\infty \le 0$ .

If strict inequalities hold, X is said to be $\Theta $ -stable.

In [7, 54], the two node case (namely, $N=1$ ) was considered. We can here generalize their result to the more general nested instantons quiver (1.1.1).

Proposition 1.2. Let X be a representation of numerical type $(r,{\mathbf n})\in \mathbb N^{N+2}_{>0}$ of the quiver (1.1.1) with relations (1.1.3). Then choose $\theta _i>0$ , $\forall i>0$ and $\theta _0$ s.t. $\theta _0+\theta _1+\cdots +\theta _{N}<0$ . The following are equivalent:

1. (i) X is $\Theta $ -stable;

2. (ii) X is $\Theta $ -semistable;

3. (iii) X satisfies the following conditions:

   1. S1 $F^i\in \operatorname {\mathrm {Hom}}(V_{i+1},V_i)$ is injective, $\forall i\ge 1$ ;

   2. S2 the ADHM datum $\mathcal A=(W,V_0,B_1^0,B_2^0,I,J)$ is stable.

Proof. $\boldsymbol {(i)\Rightarrow (ii)}$ This is obvious, as a $\Theta $ -stable representation is also $\Theta $ -semistable.

$\boldsymbol {(ii)\Rightarrow (iii)}$ Let us first take a $\Theta $ -semistable representation X having at least one of the $F^i$ not injective. Without loss of generality, let $F^k$ be the only one to be such a map. Then $v_k\in \ker F^k\Rightarrow B_2^{k}v_k\in \ker F^k$ , due to the nested ADHM equations, and $B_2^{k}(\ker F^k)\subset \ker F^k$ (and similarly for $B_1^{k}$ ). Now

$$ \begin{align*}\tilde X=(0,\dots,0,\ker F^k,0,\dots,0,B_1^{k}|_{\ker F^k},B_2^{k}|_{\ker F^k},F^k,0,\dots,0)\end{align*} $$

is a subrepresentation of X of numerical type $(0,\dots ,0,\dim \ker F^k,0,\dots ,0)$ . Thus,

$$ \begin{align*}\tilde{{\mathbf n}}\cdot\boldsymbol\theta+\tilde r\theta_\infty=\theta_k\dim\ker F^k>0,\end{align*} $$

which contradicts the hypothesis of X being $\Theta $ -semistable. Let then X be a $\Theta $ -semistable representation satisfying S1. Let also $S\subseteq V_0$ be a $B^{0}_i$ -invariant subspace of $V_0$ , $i=1,2$ , such that $\operatorname {\mathrm {Im}}(I)\subseteq S$ , and define $\tilde V_0=S$ and $\tilde V_i=(F^1\circ \cdots \circ F^i)^{-1}(S)$ . Then $\tilde X=(W,\tilde V_0,\tilde V_1,\dots ,\tilde V_N)$ , with the morphisms induced by those of X, is a subrepresentation of X of numerical type $(r,\dim S,\dim \tilde V_1,\dots ,\dim \tilde V_N)$ . We have

$$ \begin{align*}\dim\tilde V_i=\dim(\operatorname{\mathrm{Im}}(F^1\circ\cdots\circ F^i)\cap S)=n_i+\dim S - \dim(\operatorname{\mathrm{Im}}(F^1\circ\cdots\circ F^i)+S), \end{align*} $$

and by semistability,

$$ \begin{align*}{\mathbf n}\cdot\boldsymbol{\theta}=n_0\theta_0+\cdots+n_N\theta_N\ge\dim S\theta_0+\sum_{k=1}^N\left(n_k+\dim S-\dim(\operatorname{\mathrm{Im}}(F^1\circ\cdots\circ F^k)+S)\right)\theta_k=\tilde{{\mathbf n}}\cdot\boldsymbol{\theta}, \end{align*} $$

whence

$$ \begin{align*}n_0(\theta_0+\cdots+\theta_N)\ge n_0\theta_0+\sum_{k=1}^N\dim(\operatorname{\mathrm{Im}}(F^1\circ\cdots\circ F^k)+S)\theta_k\ge\dim S(\theta_0+\cdots+\theta_N). \end{align*} $$

Since $\theta _0+\cdots +\theta _N<0$ , this implies $n_0\le \dim S$ ; thus, $S=V_0$ .

$\boldsymbol {(iii)\Rightarrow (i)}$ If we take a proper subrepresentation $\tilde X$ of numerical type $(\tilde r,\tilde {{\mathbf n}})$ , we just need to check the cases $\tilde r=0$ and $\tilde r=r$ .

• ○ If $\tilde r=r$ , then $\tilde W=W$ , which, in turn, implies that $I\neq 0$ ; otherwise, the ADHM datum $(B_1^0,B_2^0,I,J)$ would not be stable. Since $\tilde X$ is proper, the following diagram commutes:

  (1.1.4)

  
  so that $\tilde n_0>0$ ; otherwise, we would have $I=0$ . Moreover, the following diagram also commutes (and so does the analogous one for $B_2^0$ ):
  (1.1.5)

  
  leading to a contradiction with the stability of $(W,V_0,B_1^0,B_2^0,I,J)$ . Since we are interested in proper subrepresentations of X, at least one $\tilde n_{i>0}$ is not zero, and at least one of these nonzero $\tilde n_k<n_k$ , so that $\boldsymbol \theta \cdot \tilde {{\mathbf n}}+\theta _\infty r<0$ , and X is stable.

• ○ Let now $\tilde r=0$ . Since we are interested in proper subrepresentations, we must choose $\tilde n_0>0$ ; otherwise, $\tilde V_{k>0}=0$ by virtue of the injectivity of $F_k$ . Similarly, one has $\tilde n_i\le \tilde n_{i-1}$ , so that

  $$ \begin{align*}\boldsymbol\theta\cdot\tilde{{\mathbf n}}=\sum_{k=0}^N\theta_k\tilde n_k\le\tilde n_0\sum_{k=0}^N\theta_k<0, \end{align*} $$
  and X is stable.

Corollary 1.3. If X is a stable representation of the nested instantons quiver (1.1.1) with relations (1.1.3), $G^k=0,\ \forall k$ .

Proof. By the previous proposition, due to the injectivity of $F^k$ , $F^kG^k=0\Rightarrow G^k=0$ .

1.2 The nested instantons moduli space

We want now to discuss the construction of the moduli space of stable representations of the quiver (1.1.1) and its connection to GIT theory and stability. First, recall that we defined the space ${\mathbb X}(r,{\mathbf n})$ of nested ADHM data as in Equation (1.1.2), and an element $X\in {\mathbb X}(r,{\mathbf n})$ is called a nested ADHM datum. On ${\mathbb X}(r,{\mathbf n})$ , we have a natural action of $\mathcal G=\operatorname {\mathrm {GL}}(V_0)\times \cdots \times \operatorname {\mathrm {GL}}(V_{N})$ defined by

(1.2.1) $$ \begin{align} \begin{aligned} \Psi:(g_0,g_1,\dots,g_{N},X)\longmapsto(g_0B_1^0g_0^{-1},g_0B_2^0g_0^{-1},g_0I,Jg_0^{-1},\\ g_1B_1^1g_1^{-1},g_1B_2^1g_1^{-1},g_0F^1g_1^{-1},g_1G^1g_0^{-1},\\ \dots\\ g_{N}B_1^{N}g_{N}^{-1},g_{N}B_2^{N}g_{N}^{-1},g_{N-1}F^{N}g_{N}^{-1},g_{N}G^{N}g_{N-1}^{-1}). \end{aligned} \end{align} $$

Let now ${\mathbb X}^{\mathrm {st}}(r,{\mathbf n})$ be the open locus of $\Theta $ -stable representations in ${\mathbb X}(r,{\mathbf n})$ , where the stability chamber is chosen as in Proposition 1.2. The $\mathcal G$ -action on ${\mathbb X}(r,{\mathbf n})$ is free on the stable locus ${\mathbb X}^{\mathrm {st}}(r,{\mathbf n})$ . Indeed, it is known that the $\Theta $ -stable representations are exactly the simple objects in the category of linear $\Theta $ -semistable representations of a quiver Q, where we let be an algebraically closed field.² Then given a $\Theta $ -stable representation X, any $\boldsymbol g\in \mathcal G$ such that $\boldsymbol g\cdot X=X$ defines an endomorphism in of the simple object X. However, is an abelian cateogry, and by Schur’s lemma, is a finite-dimensional division -algebra for any simple object X. Hence, . Thus, we must have , for , and the only such $\boldsymbol g\in \mathcal G$ fixing X corresponds to $\lambda =1$ . Also, it is easy to prove that the $\mathcal G$ -action preserves the space ${\mathbb X}_0(r,{\mathbf n})$ of nested ADHM data satisfying the relations (1.1.3).

Now if $\chi :\mathcal G\to {\mathbb C}^{*}$ is an algebraic character for the algebraic reductive group $\mathcal G$ , we can produce the moduli space of $\chi $ -semistable orbits following a construction due to [33], ${\mathcal N}^{ss}_\chi (r,{\mathbf n})$ , which is a quasi-projective scheme over ${\mathbb C}$ and is defined as

$$ \begin{align*} {\mathcal N}_\chi^{ss}(r,{\mathbf n})={\mathbb X}_0(r,{\mathbf n})/\!\!/_\chi\mathcal G=\operatorname{\mathrm{Proj}}\left(\bigoplus_{n\ge 0}A({\mathbb X}_0(r,{\mathbf n}))^{\mathcal G,\chi^n}\right) \end{align*} $$

with $A({\mathbb X}_0(r,{\mathbf n}))^{\mathcal G,\chi ^n}$ the space of polynomials in the coordinate ring $A({\mathbb X}_0(r,{\mathbf n}))$ satisfying

$$ \begin{align*}A({\mathbb X}_0(r,{\mathbf n}))^{\mathcal G,\chi^n}=\{f\in A({\mathbb X}_0(r,{\mathbf n}))|f(\boldsymbol h\cdot X)=\chi(\boldsymbol h)^nf(X),\forall\boldsymbol h\in\mathcal G\}.\end{align*} $$

The scheme ${\mathcal N}_\chi ^{ss}(r,{\mathbf n})$ contains an open subscheme ${\mathcal N}^{s}_\chi (r,{\mathbf n})\subset {\mathcal N}^{ss}_\chi (r,{\mathbf n})$ encoding $\chi $ -stable orbits. As in [33], it turns out that the notions of $\chi $ -stability and of $\Theta $ -stability are closely related.

Proposition 1.4. Let $\Theta =(\theta _0,\theta _1,\dots ,\theta _{N},\theta _\infty )\in \mathbb Z^{N+2}$ and define $\chi _\Theta :\mathcal G\to {\mathbb C}^{*}$ the character

(1.2.2) $$ \begin{align} \chi_\Theta(\boldsymbol h)=\det(h_0)^{-\theta_0}\cdots\det(h_{N})^{-\theta_{N}}. \end{align} $$

A representation X of the nested ADHM quiver (1.1.1) is $\chi _\Theta $ -(semi)stable iff it is $\Theta $ -(semi)stable.

Since the proof for Proposition 1.4 deeply relies on some known results about equivalent characterizations of $\chi $ -stability, we will first recall them. In full generality, let V be a vector space over ${\mathbb C}$ equipped with the action of a connected subgroup G of $U(V)$ , whose complexification is denoted by $G^{{\mathbb C}}$ . Then if $\chi :G\to U(1)$ is a character of G, we can extend it to form its complexification $\chi :G^{{\mathbb C}}\to {\mathbb C}^{*}$ . We then form the trivial line bundle $V\times {\mathbb C}$ , which carries an action of $G^{{\mathbb C}}$ via $\chi $ :

$$ \begin{align*} g\cdot (x,z)=(g\cdot x,\chi(g)^{-1}z),\qquad g\in G,(x,z)\in V\times{\mathbb C}. \end{align*} $$

Definition 1.5. An element $x\in V$ is

1. 1. $\chi $ -semistable if there exists a polynomial $f\in A(V)^{G^{{\mathbb C}},\chi ^n}$ , with $n\ge 1$ such that $f(x)\neq 0$ ;

2. 2. $\chi $ -stable if it satisfies the previous condition and if

   1. (a) $\dim (G^{{\mathbb C}}\cdot x)=\dim (G^{{\mathbb C}}/\Delta )$ , where $\Delta \subseteq G^{{\mathbb C}}$ is the subgroup of $G^{{\mathbb C}}$ acting trivially on V;

   2. (b) the action of $G^{{\mathbb C}}$ on $\{x\in V:f(x)\neq 0\}$ is closed.

Given the previous definition, the next lemma due to King [33] gives an alternative characterization of $\chi $ -(semi)stable points under the $G^{{\mathbb C}}$ -action.

Lemma 1.6 (Lemma 2.2 and Proposition 2.5 in [33]).

Given the character $\chi :G^{{\mathbb C}}\to {\mathbb C}^{*}$ for the action of $G^{{\mathbb C}}$ on the vector space V, and the lift of this action to the trivial line bundle $V\times {\mathbb C}$ , a point $x\in V$ is

1. 1. $\chi $ -semistable iff $\overline {G^{{\mathbb C}}\cdot (x,z)}\cap (V\times \{0\})=\emptyset $ , for any $z\neq 0$ ;

2. 2. $\chi $ -stable iff $G^{{\mathbb C}}\cdot (x,z)$ is closed and the stabilizer of $(x,z)$ contains $\Delta $ with finite index.

Equivalently, a point $x\in V$ is

1. 1. $\chi $ -semistable iff $\chi (\Delta )=\{1\}$ and $\chi (\lambda )\ge 0$ for any $1$ -parameter subgroup $\lambda (t)\subseteq G^{{\mathbb C}}$ for which $\lim _{t\to 0}\lambda (t)\cdot x$ exists;

2. 2. $\chi $ -stable iff the only $\lambda (t)$ such that $\lim _{t\to 0}\lambda (t)\cdot x$ exists and $\chi (\lambda )=0$ are in $\Delta $ .

With these notations, if $V^{\mathrm {ss}}(\chi )$ denotes the set of $\chi $ -semistable points of V, $V/\!\!/_\chi G^{{\mathbb C}}$ can be identified with $V^{\mathrm {ss}}(\chi )/\sim $ , where $x\sim y$ in $V^{\mathrm {ss}}(\chi )$ iff $\overline {G^{{\mathbb C}}\cdot x}\cap \overline {G^{{\mathbb C}}\cdot y}\neq \emptyset $ in $V^{\mathrm {ss}}(\chi )$ .

Proof of Prop. 1.4.

Notice first that , as the $\mathcal G$ -action on the open dense ${\mathbb X}^{\mathrm {st}}(r,{\mathbf n})\subset {\mathbb X}(r,{\mathbf n})$ is free; thus, the $\mathcal G$ -action on ${\mathbb X}(r,{\mathbf n})$ is effective.

Take a $\theta $ -semistable representation $X\in {\mathbb X}^{\mathrm {st}}(r,{\mathbf n})$ and assume it does not satisfy $\chi _\theta $ -semistability. Then there exists a $1$ -parameter subgroup $\lambda (t)$ of $\mathcal G$ such that $\lim _{t\to 0}\lambda (t)\cdot X$ exists and $\chi (\lambda )<0$ . However, each such $1$ -parameter subgroup $\lambda $ determines a filtration $\cdots \supseteq X_n\supseteq X_{n+1}\supseteq \cdots $ of subrepresentations of X ([33]), and

(1.2.3) $$ \begin{align} \chi_\theta(\lambda)=-\sum_{n\in\mathbb Z}\theta(X_n)\ge 0, \end{align} $$

thus proving one side of the proposition, as the part concerning stability is obvious from the fact that trivial subrepresentations of X correspond to subgroups in $\Delta $ .

Conversely, if X is a $\chi _{\theta }$ -semistable representation, we want to show that it is also a $\theta $ -semistable one. We only need to consider two cases, corresponding to subrepresentations $\tilde X$ of X with $\tilde r=r$ or $\tilde r=0$ . Each vector space in X, say $V_i$ , will have then a direct sum decomposition $V_i=\widetilde V_i\oplus \widehat V_i$ . We will then take a $1$ -parameter subgroup $\lambda (t)$ such that

(1.2.4)

Then one can easily compute

(1.2.5) $$ \begin{align} \begin{aligned} \chi_{\theta}(\lambda(t))\cdot z&=\left[\det(\lambda_0(t))^{-\theta_0}\cdots\det(\lambda_N(t))^{-\theta_N}\right]^{-1}\cdot z\\ &=t^{\tilde{{\mathbf n}}\cdot\boldsymbol\theta}z. \end{aligned} \end{align} $$

It is then a matter of a simple computation to verify that if X was not $\theta $ -semistable, then one would have had $\lim _{t\to 0}\lambda (t)\cdot X\in {\mathbb X}\times \{0\}$ , thus contradicting the $\chi _\theta $ -semistability. A completely analogous computation can be carried over when $\tilde r=r$ , taking

(1.2.6)

and since $(\tilde {{\mathbf n}}-{\mathbf n})\cdot \boldsymbol \theta>0$ if X is supposed not to be $\theta $ -semistable, this would still lead to a contradiction.

Finally, if X was to be $\chi _\theta $ -stable but not $\theta $ -stable, the $1$ -parameter subgroups previously described would have stabilized the pair $(X,z)$ , $z\neq 0$ , in the two different cases $\tilde r=0$ and $\tilde r=r$ , respectively, thus again giving rise to a contradiction.

Corollary 1.7. Given a representation X of the nested instantons quiver (1.1.1) with relations (1.1.3) of numerical type $(r,{\mathbf n})$ , there exists a chamber in $\mathbb Q^{N+2}\ni (\boldsymbol \theta ,\theta _\infty )=\Theta $ in which $\theta _{i>0}>0$ and $\theta _0+\theta _1+\cdots +\theta _{N}<0$ such that the following are equivalent:

1. 1. X is $\Theta $ -semistable;

2. 2. X is $\Theta $ -stable;

3. 3. X is $\chi _\Theta $ -semistable;

4. 4. X is $\chi _\Theta $ -stable;

5. 5. X satisfies S1 and S2 in Proposition 1.2.

Because of the previous corollary, in the stability chamber defined by Proposition 1.2, all notions of stability are actually the same, so that a representation satisfying anyone of the conditions in Corollary 1.7 will be called stable, and the corresponding ${\mathcal N}^{ss}_{\chi _\Theta }(r,{\mathbf n})={\mathcal N}(r,{\mathbf n})$ will be referred to as the moduli space of stable representations of quiver (1.1.1) with relations (1.1.3) or, equivalently, as the moduli space of nested instantons. Altogether, the previous considerations prove the following theorem.³

Theorem 1.8. The moduli space ${\mathcal N}(r,{\mathbf n})$ of stable representation of the nested instantons quiver of numerical type $(r,{\mathbf n})$ is a quasi-projective variety equipped with a perfect obstruction theory, and it has virtual dimension $(2n_0-n_1)r$ .

Proof. We need to construct a perfect obstruction theory on ${\mathcal N}(r,{\mathbf n})$ . To this end, let $\mathbb Y(r,{\mathbf n})$ be the affine space of representations of the quiver obtained from (1.1.1) by neglecting all the $G^i\in \operatorname {\mathrm {Hom}}(V_{i-1},V_i)$ . Consider $\Theta =(\boldsymbol \theta ,\theta _\infty )\in \mathbb Q^{N+2}$ as in Proposition 1.2, and let $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})\subset \mathbb Y(r,{\mathbf n})$ be the open locus of $\Theta $ -stable points in $\mathbb Y(r,{\mathbf n})$ . Similarly to ${\mathbb X}(r,{\mathbf n})$ , on $\mathbb Y(r,{\mathbf n})$ , there is a natural $\mathcal G$ -action, which is free on the stable locus $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})$ . Thus, the GIT quotient $\mathbb Y(r,{\mathbf n})/\!/_{\chi _\Theta }\mathcal G$ , with $\chi _\Theta $ defined as in Proposition 1.4, exists as a smooth quasi-projective variety, which we will denote by $\mathcal Y(r,{\mathbf n})$ . The moduli space of nested instantons ${\mathcal N}(r,{\mathbf n})$ is then the closed subscheme of $\mathcal Y(r,{\mathbf n})$ cut by the nested ADHM equations (1.1.3), neglecting $G^i$ . We claim that there exists a vector bundle $\mathcal E$ over a smooth subvariety $\mathcal A$ of $\mathcal Y(r,{\mathbf n})$ , together with a section $\sigma \in H^0(\mathcal A,\mathcal E)$ , such that ${\mathcal N}(r,{\mathbf n})$ is isomorphic to the zero scheme $Z(\sigma )$ :

(1.2.7)

This induces a perfect obstruction theory on ${\mathcal N}(r,{\mathbf n})$ à la Behrend-Fantechi ([3]) – that is, a perfect complex $\mathbb E\in \mathbf D^{[-1,0]}(Z(\sigma ))$ , together with a morphism in the derived category

where $\mathbb L_{Z(\sigma )}$ is the truncation $\tau ^{\ge -1}L^{\bullet }_{Z(\sigma )}$ of Illusie’s cotangent complex $L^{\bullet }_{Z(\sigma )}\in \mathbf D^{[-\infty ,0]}(Z(\sigma ))$ ([32]), and ${\mathcal I}\subset \mathcal O_{\mathcal A}$ is the ideal sheaf of the inclusion $Z(\sigma )\hookrightarrow \mathcal A$ . In what follows, we will show how such a vector bundle $\mathcal E\longrightarrow \mathcal A$ is constructed.

On $\mathbb Y(r,{\mathbf n})$ , we introduce the vector bundles $C^i$ , $i=0,\dots ,3$ , defined as

$$ \begin{align*} C^0 &=\bigoplus_{i=0}^N\operatorname{\mathrm{End}}(V_i),\\ C^1 &=\operatorname{\mathrm{End}}(V_0)^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(W,V_0)\oplus\operatorname{\mathrm{Hom}}(V_0,W)\oplus\left[\bigoplus_{i=1}^N\left(\operatorname{\mathrm{End}}(V_i)^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(V_i,V_{i-1}\right)\right]\\ & \cong T_{\mathbb Y(r,{\mathbf n})},\\ C^2 &=\operatorname{\mathrm{End}}(V_0)\oplus\operatorname{\mathrm{Hom}}(V_1,W)\oplus\left[\bigoplus_{i=1}^N\left(\operatorname{\mathrm{Hom}}(V_i,V_{i-1})^{\oplus 2}\oplus\operatorname{\mathrm{End}}(V_i)\right)\right],\\ C^3 &=\bigoplus_{i=1}^N\operatorname{\mathrm{Hom}}(V_i,V_{i-1}). \end{align*} $$

Each vector bundle $C^i$ , $i=0,\dots ,3$ carries a natural equivariant structure under the $\mathcal G$ -action on $\mathbb Y(r,{\mathbf n})$ , so they restrict to the stable locus $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})$ and then descend to vector bundles $\mathcal C^i$ on $\mathcal Y(r,{\mathbf n})$ under the quotient map $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})\xrightarrow {\pi }\mathcal Y(r,{\mathbf n})$ .

On $\mathbb Y(r,{\mathbf n})$ , we also have vector bundle homomorphisms $d^i:C^i\to C^{i+1}$ , $i=0,1,2$ , defined along the fibres as

$$\begin{align*}\begin{aligned} &d^0\begin{pmatrix} h_0\\ h_1\\ \vdots\\ h_N \end{pmatrix} = \begin{pmatrix} [h_0,B_1^0]\\ [h_0,B_2^0]\\ h_0I\\ -Jh_0\\ [h_1,B_1^1]\\ [h_1,B_2^1]\\ h_0F^1-F^1h^1\\ \vdots \end{pmatrix},\qquad d^1\begin{pmatrix} b_1^0\\ b_2^0\\ i\\ j\\ b_1^1\\ b_2^1\\ f^1\\ \vdots \end{pmatrix}=\begin{pmatrix} [b_1^0,B_2^0]+[B_1^0,b_2^0]+iJ+Ij\\ jF^1+Jf^1\\ B_1^0f^1+b_1^0F^1-F^1b_1^1-f^1B_1^1\\ B_2^0f^1+b_2^0F^1-F^1b_2^1-f^1B_2^1\\ [b_1^1,B_2^1]+[B_1^1,b_2^1]\\ \vdots\\ B_1^{N-1}f^N+b_1^{N-1}F^N-F^Nb_1^N-f^NB_1^N\\ B_2^{N-1}f^N+b_2^{N-1}F^N-F^Nb_2^N-f^NB_2^N\\ [b_1^N,B_2^N]+[B_1^N,b_2^N] \end{pmatrix},\\ &d^2\begin{pmatrix} c_1\\ c_2\\ c_3\\ \vdots\\ c_{3N+2} \end{pmatrix}=\begin{pmatrix} c_1F^1+B_2^0c_3-c_3B_2^1+c_4B_1^1-B_1^0c_4-Ic_2-F^1c_5\\ c_5F^2+B_2^1c_6-c_6B_2^2+c_7B_1^2-B_1^1c_7-F^2c_8\\ \vdots\\ c_{3i-1}F^i+B_{2}^{i-1}c_{3i}-c_{3i}B_2^i+c_{3i+1}B_1^i-B_1^{i-1}c_{3i+1}-F^ic_{3i+2}\\ \vdots \end{pmatrix}. \end{aligned} \end{align*}$$

Altogether, the datum of the vector bundles $C^i$ , $i=0,\dots ,3$ , and of the bundle homomorphisms $d^j$ , $j=0,1,2$ , descends to the datum of vector bundles $\mathcal C^i$ , $i=0,\dots ,3$ , and bundle homomorphisms $\delta ^j:\mathcal C^i\to \mathcal C^{i+1}$ , $j=0,1,2$ over $\mathcal Y(r,{\mathbf n})$ . Let us point out that when restricted to ${\mathcal N}(r,{\mathbf n})$ , the couple $(\mathcal C^{\bullet },\delta ^{\bullet })$ forms a complex of vector bundles over ${\mathcal N}(r,{\mathbf n})$ , which we claim to have vanishing cohomology in degrees $0$ and $3$ . Let $\mathcal C_A$ , $\mathcal C_B$ and $\mathcal C_{A,B}$ be the following auxiliary complexes of vector bundles over ${\mathcal N}(r,{\mathbf n})$ :⁴

with

$$\begin{align*}\delta^0_A(h_0)=\begin{pmatrix} [h_0,B_1^0]\\ [h_0,B_2^0]\\ h_0I\\ -Jh_0 \end{pmatrix},\qquad \delta^1_A\begin{pmatrix} b_1^0\\ b_2^0\\ i\\ j \end{pmatrix}=[b_1^0,B_2^0]+[B_1^0,b_2^0]+Ij+iJ; \end{align*}$$

with

$$\begin{align*}\delta^0_B\begin{pmatrix} h_1\\ \vdots\\ h_N \end{pmatrix}=\begin{pmatrix} [h_1,B_1^1]\\ [h_1,B_2^1]\\ \vdots\\ [h_N,B_1^N]\\ [h_N,B_2^N] \end{pmatrix},\qquad \delta^1_B\begin{pmatrix} b_1^1\\ b_2^1\\ \vdots\\ b_1^N\\ b_2^N \end{pmatrix}=\begin{pmatrix} [b_1^1,B_2^1]+[B_1^1,b_2^1]\\ \vdots\\ [b_1^N,B_2^N]+[B_1^N,b_2^N] \end{pmatrix}; \end{align*}$$

with

$$\begin{align*}\delta^0_{A,B}\begin{pmatrix} f^1\\ \vdots\\ f^N \end{pmatrix}=\begin{pmatrix} -B_1^0f^1+f^1B_1^1\\ -B_2^0f^1+f^1B_2^1\\ \vdots\\ -B_1^{N-1}f^N+f^NB_1^N\\ -B_2^{N-1}f^N+f^NB_2^N\\ -Jf^1 \end{pmatrix}, \end{align*}$$

$$\begin{align*}\delta^1_{A,B}\begin{pmatrix} c_3\\ \vdots\\ c_{2N+2}\\ c_2 \end{pmatrix}=\begin{pmatrix} -B_2^0c_3+c_3B_2^1-c_4B_1^1+B_1^0c_4+Ic_2\\ -B_2^1c_5+c_5B_2^2-c_6B_1^2+B_1^1c_6\\ \vdots\\ -B_2^{N-1}c_{2N+1}+c_{2N+1}B_2^N-c_{2N+2}B_1^N+B_1^{N-1}c_{2N+2} \end{pmatrix}. \end{align*}$$

It is then readily verified that there exists a distinguished triangle

(1.2.8)

coming from the fact that $\mathcal C[1]$ is a cone for $\rho =(\rho _0,\rho _1,\rho _2)$ , where

(1.2.9a) $$ \begin{align} \rho_0\begin{pmatrix} h_0\\ \vdots\\ h_N \end{pmatrix}&=\begin{pmatrix} -h_0F^1+F^1h_1\\ \vdots\\ -h_{N-1}F^N+F^Nh_N \end{pmatrix}, \end{align} $$

(1.2.9b) $$ \begin{align} \rho_1\begin{pmatrix} b_1^0\\ b_2^0\\ i\\ j\\ b_1^1\\ b_2^1\\ \vdots \end{pmatrix}&=\begin{pmatrix} -b_1^0F^1+F^1b_1^1\\ -b_2^0F^1+F^1b_2^1\\ \vdots\\ -b_1^{N-1}F^N+F^Nb_1^N\\ -b_2^{N-1}F^N+F^Nb_2^N\\ -jF^1 \end{pmatrix}, \end{align} $$

(1.2.9c) $$ \begin{align} \rho_2\begin{pmatrix} c_1\\ c_{2N+3}\\ \vdots\\ c_{3N+3} \end{pmatrix}&=\begin{pmatrix} -c_1F^1+F^1c_{2N+3}\\ \vdots\\ -c_{3N+2}F^N+F^Nc_{3N+3} \end{pmatrix}. \end{align} $$

From the triangle (1.2.8), one gets a long exact sequence in cohomology

and since $\mathcal C_A$ is simply the deformation complex of the standard ADHM quiver, it is well known that $H^0(\mathcal C_A)=H^2(\mathcal C_A)=0$ , [43]. Then $H^0(\mathcal C)=0$ by the injectivity of $H^0(\rho ):H^0(\mathcal C_B)\to H^0(\mathcal C_{A,B})$ . In fact, we have

$$\begin{align*}H^0(\rho)\begin{pmatrix} h_1\\ \vdots\\ h_N \end{pmatrix}=0\Rightarrow\begin{pmatrix} F^1h_1\\ \vdots\\ F^Nh_N \end{pmatrix}=0\Rightarrow\begin{pmatrix} h_1\\ \vdots\\ h_N \end{pmatrix}=0 \end{align*}$$

since $F^i$ is injective. Moreover, the map $\mathcal \delta ^1_{A,B}:\mathcal C_{A,B}^1\to \mathcal C_{A,B}^2$ is surjective: this, in turn, means that $H^2(\mathcal C_{A,B})=0$ , which implies $H^3(\mathcal C)=0$ . Indeed, let us consider $(\delta ^1_{A,B})^\vee $ . One has

$$\begin{align*}(\delta^1_{A,B})^\vee(\boldsymbol\phi)=(\delta^1_{A,B})^\vee\begin{pmatrix} \phi_1\\ \vdots\\ \phi_N \end{pmatrix}=\begin{pmatrix} B_2^1\phi_1-\phi_1B_2^0\\ -B_1^1\phi_1+\phi_1B_1^1\\ \vdots\\ B_2^N\phi_N-\phi_NB_2^{N-1}\\ -B_1^N\phi_N+\phi_NB_1^{N-1}\\ \phi_1I \end{pmatrix}, \end{align*}$$

and if $\boldsymbol \phi \in \ker ((\delta ^1_{A,B})^\vee )$ , then $\ker (\phi _1)$ would be a $(B_1^0,B_2^0)$ -invariant subset of $V_0$ containing $\operatorname {\mathrm {Im}}(I)$ , which contradicts the stability of X, by which we conclude that $\ker (\phi _1)=V_0$ . Similar statements also hold true for each other component of $\boldsymbol \phi $ , which we then conclude to be $\boldsymbol \phi =0$ .

Let now s be the section $s\in H^0(\mathbb Y(r,{\mathbf n}),C^2)$ cutting via its zero locus $Z(s)\subset \mathbb Y(r,{\mathbf n})$ the solutions to the nested ADHM equations (1.1.3) (neglecting $G^i$ ) in $\mathbb Y(r,{\mathbf n})$ – that is,

$$\begin{align*}\begin{aligned} s(X)=\left([B_1^0,B_2^0]+IJ\right)\oplus JF^1\oplus\bigoplus_{i=1}^N\bigg(\left(F^iB_1^i-B_1^{i-1}F^i\right)\oplus\left(F^iB_2^i-B_2^{i-1}F^i\right)\oplus [B_1^i,B_2^i]\bigg). \end{aligned} \end{align*}$$

As s is naturally $\mathcal G$ -equivariant, it also restricts to the stable locus and descends to a section $\sigma \in H^0(\mathcal Y(r,{\mathbf n}),\mathcal C^2)$ . It is easily checked that $d^2\circ s=0$ , so $s(\mathbb Y(r,{\mathbf n}))\subseteq \ker (C^2\xrightarrow {d^2}C^3)$ . Let now $A\subset \mathbb Y^{\mathrm {st}}(r,{\mathbf n})$ be the open subscheme of $\mathbb Y^{\mathrm {st}}(r,{\mathbf n})$ where $d^2$ is surjective, and whose image under the quotient map is $\mathcal A\subseteq \mathcal Y(r,{\mathbf n})$ . Then $\ker (C^2|_A\xrightarrow {d^2}C^3|_A)$ is a $\mathcal G$ -equivariant vector bundle $E\longrightarrow A$ , which descends to a vector bundle ${\mathcal E}\longrightarrow \mathcal A$ . Also, $s|_A\in H^0(A,E)$ is a $\mathcal G$ -equivariant section, inducing a section $\sigma \in H^0(\mathcal A,{\mathcal E})$ . The moduli space of nested instantons is then constructed as the zero scheme $Z(\sigma )\subset \mathcal A$ . Indeed, as $\delta ^2$ is surjective over ${\mathcal N}(r,{\mathbf n})$ , we have ${\mathcal N}(r,{\mathbf n})\subset \mathcal A$ . Moreover, as the section $\sigma \in H^0(\mathcal A,\mathcal C^2)$ factors through a section of the subbundle ${\mathcal E}\longrightarrow \mathcal A$ , then $Z(s)$ and ${\mathcal N}(r,{\mathbf n})$ define the same closed subscheme of $\mathcal A$ .

Thus, we are precisely in the situation of diagram (1.2.7), and ${\mathcal N}(r,{\mathbf n})\cong Z(\sigma )$ is naturally equipped with a perfect obstruction theory of rank $\operatorname {\mathrm {vd}}=\operatorname {\mathrm {rk}}\mathbb E=(2n_0-n_1)r$ , whose virtual tangent complex is

$$\begin{align*}\mathbb E^\vee=\left[T_{\mathcal A}|_{{\mathcal N}(r,{\mathbf n})}\xrightarrow{{\text d}\sigma}{\mathcal E}|_{{\mathcal N}(r,{\mathbf n})}\right]\in\mathbf D^{[-1,0]}({\mathcal N}(r,{\mathbf n})). \end{align*}$$

In particular, retracing all the steps we have discussed so far, we can see that the K-theory class of the virtual tangent space over a point $X\in {\mathcal N}(r,{\mathbf n})$ attached to the perfect obstruction theory $\mathbb E$ is

$$\begin{align*}\left.T^{\mathrm{vir}}_{{\mathcal N}(r,{\mathbf n})}\right|_X=H^1(\mathcal C^{\bullet}|_X)-H^2(\mathcal C^{\bullet}|_X)\in K^0(\operatorname{\mathrm{pt}}).\end{align*}$$

Remark 1.9. With the notation introduced in the proof of Theorem 1.8, the moduli space of nested instantons ${\mathcal N}(r,{\mathbf n})$ is also equipped with another perfect obstruction theory, induced by the vector bundle $C^2\longrightarrow \mathbb Y(r,{\mathbf n})$ , together with the section $s\in H^0(\mathbb Y(r,{\mathbf n}),C^2)$ . Indeed, after restriction to the stable locus, the couple $(C^2,s)$ descends to a vector bundle and a section $(\mathcal C^2,\sigma )$ over $\mathcal Y(r,{\mathbf n})$ , and ${\mathcal N}(r,{\mathbf n})\cong Z(\sigma )\subset \mathcal Y(r,{\mathbf n})$ . Then, ${\mathcal N}(r,{\mathbf n})$ is equipped with a second perfect obstruction theory $\tilde {\mathbb E}$ , whose tangent complex is $\tilde {\mathbb E}^\vee =[T_{\mathcal Y(r,{\mathbf n})}|_{{\mathcal N}(r,{\mathbf n})}\xrightarrow {{\text d}\sigma }\mathcal C^2|_{{\mathcal N}(r,{\mathbf n})}]$ . However, the rank of $\tilde {\mathbb E}$ is

$$\begin{align*}\operatorname{\mathrm{rk}}\tilde{\mathbb E}=(2n_0-n_1)r-\sum_{i=1}^Nn_in_{i-1}, \end{align*}$$

which might become negative (e.g., this is the case for $r=1,n_0\ge 3,n_1=n_0-1$ ). This is due to the relations (1.1.3) being overdetermined, as they are not independent. Thus, in Theorem 1.8, we constructed a ‘reduced’ perfect obstruction theory, which automatically takes into account the dependency of the equations cutting the moduli space of nested instantons by exploiting the fact that $s(\mathbb Y(r,{\mathbf n}))\subseteq \ker (C^2\xrightarrow {d^2}C^3)$ .

For future reference, we want now to exhibit some morphisms between different nested instantons moduli spaces and between them and usual moduli spaces of instantons, which are moduli spaces of framed torsion-free sheaves on ${\mathbb P}^2$ . We obviously have iterative forgetting projections $\eta _i:{\mathcal N}(r,n_0,\dots ,n_i)\to {\mathcal N}(r,n_0,\dots ,n_{i-1})$ . Moreover, we also have other morphisms to underlying moduli spaces of framed torsion-free sheaves on ${\mathbb P}^2$ , which are summarized by the commutative diagram in Figure 3.

Figure 3 Morphisms between moduli spaces of sheaves.

In order to see that these maps do indeed exist, take a stable representation $[X]$ of the nested instantons quiver. The fact that $[X]$ is stable implies that the morphisms $F^i$ are injective, so that we can construct the stable ADHM datum $(W,\tilde V_i,\tilde B_1^i,\tilde B_2^i,\tilde I^i,\tilde J^i)$ as follows. Let $\tilde V_i$ be $V_0/\operatorname {\mathrm {Im}}(F^1\cdots F^i)$ and choose a basis of $V_i$ in such a way that

(1.2.10)

whence $V_0=V_i\oplus \tilde V_i$ . Then define the projections $\pi _i=V_0\to V_i$ and $\tilde \pi _i:V_0\to \tilde V_i$ as $\pi _i(v,\tilde v)=v$ and $\tilde \pi _i(v,\tilde v)=\tilde v$ , with $v\in V_i$ , $\tilde v\in \tilde V_i$ . We can then show how $\tilde V_i$ inherits an ADHM structure by its embedding in $V_0$ . Indeed, if we define $\tilde B_1^i=B_1^0|_{\tilde V_i}$ , $\tilde B_2^i=B_2^0|_{\tilde V_i}$ , $\tilde I^i=\tilde \pi _i\circ I$ and $\tilde J^i=J|_{\tilde V_i}$ , the datum $(W,\tilde V_i,\tilde B_1^i,\tilde B_2^i,\tilde I^i,\tilde J^i)$ satisfies the ADHM Equation (1.2.11):

(1.2.11) $$ \begin{align} [\tilde B_1^i,\tilde B_2^i]+\tilde I^i\tilde J^i=[B_1^0|_{\tilde V_i},B_2^0|_{\tilde V_i}]+\tilde\pi_i\circ I\circ J|_{\tilde V_i}=\bigg(\left[B_1^0,B_2^0\right]+IJ\bigg)\bigg|_{\tilde V_i}=0. \end{align} $$

This new ADHM datum is moreover stable, as if it would exist $0\subset \tilde S_i\subset \tilde V_i$ such that $\tilde B_{1,2}^i(\tilde S_i),\tilde I^i(W)\subset \tilde {S_i}$ it would imply that also the ADHM datum $(W,V_0,B_1^0,B_2^0,I,J)$ would not be stable. In fact, in that case, we could take $0\subset V_i\oplus \tilde S_i\subset V_0$ , and it would be such that $B_1^0(V_i\oplus \tilde S_i),B_2^0(V_i\oplus \tilde S_i),I(W)\subset V_i\oplus \tilde S_i$ . In fact, if we take any $(v,\tilde s)\in V_i\oplus \tilde S_i$ , it happens that $B_1^0(v,s)=(B_1^0|_{V_i}(v),B_1^0|_{\tilde V_i}\tilde s)=(B_1^0|_{V_i},\tilde B_1^i(\tilde s))\in V_i\oplus \tilde S_i$ , $B_2^0(v,s)=(B_2^0|_{V_i}(v),B_2^0|_{\tilde V_i}\tilde s)=(B_2^0|_{V_i},\tilde B_2^i(\tilde s))\in V_i\oplus \tilde S_i$ and $I(W)=I(W)\cap V_i\oplus I(W)\cap \tilde V_i=(\pi _i\circ I)(W)\oplus (\tilde \pi _i\circ I)(W)\subset V_i\oplus \tilde S_i$ . Thus, we constructed a map $\mathfrak p^{(N)}_i:{\mathcal N}(r,n_0,\dots ,n_{N})\to \mathcal M(r,n_0-n_i)$ .

1.3 Nakajima quiver varieties

In this section, we exhibit an embedding of the moduli space of nested instantons into a Nakajima quiver variety, which is a smooth hyperkähler variety; [41, 42, 22]. In the vector space ${\mathbb X}(r,{\mathbf n})$ of representations of quiver (1.1.1) of numerical type $(r,{\mathbf n})$ , which, we recall, is defined as

(1.3.1) $$ \begin{align} \begin{aligned} {\mathbb X}(r,{\mathbf n})=\operatorname{\mathrm{End}}(V_0)^{\oplus 2}\oplus\operatorname{\mathrm{Hom}}(W,V_0)\oplus\operatorname{\mathrm{Hom}}(V_0,W)\oplus\bigoplus_{k=1}^{N}\left[\operatorname{\mathrm{End}}(V_k)^{\oplus 2}\right.\\ \left.\oplus\operatorname{\mathrm{Hom}}(V_{k-1},V_k)\oplus\operatorname{\mathrm{Hom}}(V_k,V_{k-1})\right], \end{aligned} \end{align} $$

we will introduce a family of relations:

(1.3.2) $$ \begin{align} & [B_1^0,B_2^0]+IJ+F^1G^1=0, \end{align} $$

(1.3.3) $$ \begin{align} & [B_1^i,B_2^i]-G^iF^i+F^{i+1}G^{i+1}=0,\quad i=1,\dots,N. \end{align} $$

Then a point $X\in {\mathbb X}(r,{\mathbf n})$ is called stable if it satisfies conditions $\mathbf {S1}$ and $\mathbf {S2}$ in Proposition 1.2. With these conventions, we will define $\mathbb M(r,{\mathbf n})$ to be the space of stable elements of ${\mathbb X}(r,{\mathbf n})$ satisfying the relations (1.3.2)--(1.3.3):

(1.3.4) $$ \begin{align} \mathbb M(r,{\mathbf n})=\{X\in{\mathbb X}(r,{\mathbf n}):X\ \textrm{is stable and satisfies } (1.3.2), (1.3.3)\}. \end{align} $$

Exactly in the same way as we did before, we can easily see that there is a natural action of $\mathcal G=\operatorname {\mathrm {GL}}(V_0)\times \cdots \times \operatorname {\mathrm {GL}}(V_N)$ which is free on $\mathbb M(r,{\mathbf n})$ and preserves the equations 1.3.2--1.3.3: the same is then true for the natural $\mathcal U$ -action on $\mathbb M(r,{\mathbf n})$ , with $\mathcal U=U(V_0)\times \cdots \times U(V_N)$ . Thus, a moduli space $\mathcal M(r,{\mathbf n})$ of stable $\mathcal U$ -orbits in $\mathbb M(r,{\mathbf n})$ can be defined by means of GIT theory, as was the case for ${\mathcal N}(r,{\mathbf n})$ in the previous sections. Moreover, any stable point of ${\mathbb X}$ satisfying the nested ADHM equations automatically satisfies (1.3.2) and (1.3.3). Indeed, a stable representation of quiver (1.1.1) satisfies, among other relations, the equations

$$\begin{align*}[B_1^0,B_2^0]+IJ=0,\qquad [B_1^{i>0},B_2^{i>0}]=0, \end{align*}$$

while $G^i=0$ , for $i=1,\dots ,N$ , by Corollary 1.3. Thus, any stable representation of quiver (1.1.1) with relations (1.1.3) also satisfies the relations (1.3.2)--(1.3.3) so that ${\mathcal N}(r,{\mathbf n})\hookrightarrow \mathcal M(r,{\mathbf n})$ via the natural inclusion.

Next, let us point out that on each $T\operatorname {\mathrm {Hom}}(V_i,V_k)$ , we can introduce a hermitean metric by defining

(1.3.5) $$ \begin{align} \langle X,Y\rangle=\frac{1}{2}\operatorname{\mathrm{tr}}\left(X\cdot Y^\dagger+X^\dagger\cdot Y\right),\qquad\forall X,Y\in\operatorname{\mathrm{Hom}}(V_i,V_k), \end{align} $$

which, in turn, can be linearly extended to a hermitean metric $\langle -,-\rangle :T\mathbb M(r,{\mathbf n})\times T\mathbb M(r,{\mathbf n})\to {\mathbb C}$ . Finally, we can introduce some complex structures on $T\mathbb M(r,{\mathbf n})$ : given $X\in T\mathbb M(r,{\mathbf n})$ , these are defined as the following $I,J,K\in \operatorname {\mathrm {End}}(T\mathbb M(r,{\mathbf n}))$

(1.3.6) $$ \begin{align} & I(X)=\sqrt{-1}X, \end{align} $$

(1.3.7) $$ \begin{align} & J(X)=(-b_2^{0\dagger},b_1^{0\dagger}, -j^\dagger,i^\dagger,\{-b_2^{i\dagger},b_1^{i\dagger},-g^{i\dagger},f^{i\dagger}\}), \end{align} $$

(1.3.8) $$ \begin{align} & K(X)=I\circ J(X), \end{align} $$

with $X=(b_1^0,b_2^0,i,j,\{b_1^i,b_2^i,f^i,g^i\})$ . These three complex structures make the datum of

$$ \begin{align*}(\mathbb M(r,{\mathbf n}),\langle-,-\rangle,I,J,K)\end{align*} $$

a hyperkähler manifold, as one can readily verify. It is a standard fact that once we fix a particular complex structure, say I, and its respective Kähler form, $\omega _I$ , the linear combination $\omega _{{\mathbb C}}=\omega _J+\sqrt {-1}\omega _K$ is a holomorphic symplectic form for $\mathbb M(r,{\mathbf n})$ . The thing we finally want to prove is that the hyperkähler structure on $\mathbb M(r,{\mathbf n})$ induces a hyperkähler structure on the GIT quotient $\mathcal M(r,{\mathbf n})$ , which will be moreover proven to be smooth. This is made possible by the fact that the natural $\mathcal U$ -action on $\mathbb M(r,{\mathbf n})$ preserves the hermitean metric and the complex structures we introduced. Then, letting ${\mathfrak u}$ be the Lie algebra of the group $\mathcal U$ , we need to construct a moment map

$$ \begin{align*} \mu:\mathbb M(r,{\mathbf n})\to {\mathfrak u}^{*}\otimes\mathbb R^3, \end{align*} $$

satisfying

1. 1. $\mathcal G$ -equivariance: $\mu (g\cdot X)=\operatorname {\mathrm {Ad}}_{g^-1}^{*}\mu (X)$ ;

2. 2. $\langle {\text d}\mu _i(X),\xi \rangle =\omega _i(\xi ^{*},X)$ , for any $X\in T\mathbb M(r,{\mathbf n})$ and $\xi \in {\mathfrak u}$ generating the vector field $\xi ^{*}\in T\mathbb M(r,{\mathbf n})$ .

If then $\zeta \in {\mathfrak u}^{*}\otimes \mathbb R^3$ is such that $\operatorname {\mathrm {Ad}}_g^{*}(\zeta _i)=\zeta _i$ for any $g\in \mathcal U$ , $\mu ^{-1}(\zeta )$ is $\mathcal U$ -invariant, and it makes sense to consider the quotient space $\mu ^{-1}(\zeta )/\mathcal U$ . It is known ([30]) that if $\mathcal U$ acts freely on $\mu ^{-1}(\zeta )/\mathcal U$ , the latter is a smooth hyperkähler manifold, with complex structures and metric induced by those of $\mathbb M(r,{\mathbf n})$ .

Our task of finding a moment map $\mu :\mathbb M(r,{\mathbf n})\to {\mathfrak u}^{*}\otimes \mathbb R^3$ then translates into the following. Define $(\mu _1^0,\dots ,\mu _1^N)=\mu _1:\mathbb M(r,{\mathbf n})\to {\mathfrak u}$

(1.3.9) $$ \begin{align} \left\{ \begin{aligned} &\mu_1^0(X)=\frac{\sqrt{-1}}{2}\left([B_1^0,B_1^{0\dagger}]+[B_2^0,B_2^{0\dagger}]+II^\dagger-J^\dagger J+F^1F^{1\dagger}-G^{1\dagger}G^1\right)\\ &\mu_1^1(X)=\frac{\sqrt{-1}}{2}\left([B_1^1,B_1^{1\dagger}]+[B_2^1,B_2^{1\dagger}]-F^{1\dagger}F^1+G^1G^{1\dagger}+F^2F^{2\dagger}-G^{2\dagger}G^2\right)\\ &\qquad\qquad\vdots\\ &\mu_1^N(X)=\frac{\sqrt{-1}}{2}\left([B_1^N,B_1^{N\dagger}]+[B_2^N,B_2^{N\dagger}]-F^{N\dagger}F^N+G^NG^{N\dagger}\right), \end{aligned} \right. \end{align} $$

with $X=(B_1^0,B_2^0,I,J,\{B_1^i,B_2^i,F^i,G^i\})\in \mathbb M(r,{\mathbf n})$ . In addition to $\mu _1$ , we also define a map $\mu _{{\mathbb C}}:\mathbb M(r,{\mathbf n})\to \mathfrak {g}$ , with $\mathfrak g=\mathfrak {gl}(V_0)\times \cdots \times \mathfrak {gl}(V_N)$ :

(1.3.10) $$ \begin{align} \left\{ \begin{aligned} &\mu_{{\mathbb C}}^0(X)=[B_1^0,B_2^0]+IJ+F^1G^1\\ &\mu_{{\mathbb C}}^1(X)=[B_1^1,B_2^1]-G^1F^1+F^2G^2\\ &\qquad\qquad\vdots\\ &\mu_{{\mathbb C}}^N(X)=[B_1^N,B_2^N]-G^NF^N, \end{aligned} \right. \end{align} $$

by means of which we define $\mu _{2,3}:\mathbb M(r,{\mathbf n})\to {\mathfrak u}$ as $\mu _{{\mathbb C}}(X)=(\mu _2+\sqrt {-1}\mu _3)(X)$ . Notice that in absence of $B_i^j$ and $I,J$ the complex moment map we defined would reduce to the Crawley-Boevey moment map in [12]. We then claim that $\mu =(\mu _1,\mu _2,\mu _3)$ is a moment map for the $\mathcal U$ -action on $\mathbb M(r,{\mathbf n})$ . If this is true and $\chi $ is the algebraic character we introduced in §1.2, the space

(1.3.11) $$ \begin{align} \widetilde{\mathcal M}(r,{\mathbf n})=\frac{\mu_1^{-1}(\sqrt{-1}{\text d}\chi)\cap\mu_{{\mathbb C}}^{-1}(0)\cap\mathbb M(r,{\mathbf n})}{\mathcal U}=\frac{\mu^{-1}(\sqrt{-1}{\text d}\chi,0,0)\cap\mathbb M(r,{\mathbf n})}{\mathcal U} \end{align} $$

is a smooth hyperkähler manifold which, by an analogue of Kempf-Ness theorem, is also isomorphic to $\mathcal M(r,{\mathbf n})$ . In fact, it is known, due to a result of [33, 43] and the characterization of $\chi $ -(semi)stable points we gave in the previous sections, that there exists a bijection between $\mu _1^{-1}(\sqrt {-1}{\text d}\chi )$ and the set of $\chi $ -(semi)stable points in $\mu _{{\mathbb C}}^{-1}(0)$ . Then, in order to prove that $\mu $ is actually a moment map, we will first compute the vector field $\xi ^{*}$ generated by a generic $\xi \in {\mathfrak u}$ . Let then $X=(b_1^0,b_2^0,i,j,\{b_1^i,b_2^i,f^i,g^i\})$ be a vector in $T\mathbb M(r,{\mathbf n})$ and $\Psi _X:\mathcal U\to \mathbb M(r,{\mathbf n})$ the action of $\mathcal U$ onto $X\in \mathbb M(r,{\mathbf n})$ : the fundamental vector field generated by $\xi \in {\mathfrak u}$ is

(1.3.12)

where $\gamma $ is a smooth curve $\gamma :(-\epsilon ,\epsilon )\to \mathcal U$ such that and $\dot \gamma (0)=\xi $ . Thus, we can compute

(1.3.13) $$ \begin{align} \begin{aligned} \xi^{*}|_X=\left([\xi_0,b_1^0],[\xi_0,b_2^0],\xi_0i,-j\xi_0,[\xi_1,b_1^1],[\xi_1,b_2^1],\xi_0f^1-f^1\xi_1,\xi_1g^1-g^1\xi_0,\dots\right.\\ \left.\dots,[\xi_N,b_1^N],[\xi_N,b_2^N],\xi_{N-1}f^N-f^N\xi_N,\xi_Ng^N-g^N\xi_{N-1}\right). \end{aligned} \end{align} $$

Then if $\pi _i:\mathbb M(r,{\mathbf n})\to \mathbb M(r,{\mathbf n})$ denotes the projection on the i-th component of the direct sum decomposition induced by (1.3.1) so that i runs over the index set ${\mathcal I}$ , by inspection, one can see that $\omega _1$ is exact, and in particular, $\omega _1={\text d}\lambda _1$ , with

(1.3.14) $$ \begin{align} \lambda_1=\frac{\sqrt{-1}}{2}\operatorname{\mathrm{tr}}\left(\sum_{i\in{\mathcal I}}\pi_i\wedge\pi_{i*}^\dagger\right). \end{align} $$

This implies that

(1.3.15) $$ \begin{align} \langle\mu_1(x),\xi\rangle=\imath_{\xi^{*}}\lambda_1, \end{align} $$

and it is easy to verify that $\mu _1:\mathbb M(r,{\mathbf n})\to {\mathfrak u}^{*}$ thus defined indeed matches with the definition (1.3.9). Similarly, one can realize that

(1.3.16) $$ \begin{align} &\lambda_2=\Re\left[\operatorname{\mathrm{tr}}\left(\sum_{i\in2\mathbb Z\cap{\mathcal I}}\pi_i\wedge\pi_{1+1*}\right)\right], \end{align} $$

(1.3.17) $$ \begin{align} &\lambda_3=-\sqrt{-1}\Im\left[\operatorname{\mathrm{tr}}\left(\sum_{i\in2\mathbb Z\cap{\mathcal I}}\pi_i\wedge\pi_{1+1*}\right)\right], \end{align} $$

and the moment map components satisfying $\langle \mu _i(x),\xi \rangle =\imath _{\xi ^{*}}\lambda _i$ agree with the combination $\mu _2+\sqrt {-1}\mu _3=\mu _{{\mathbb C}}$ we gave previously in Equation (1.3.10).

2 Flags of framed torsion-free sheaves on ${\mathbb P}^2$

We give in this paragraph the construction of the moduli space of flags of framed torsion-free sheaves of rank r on the complex projective plane. We also show that there exists a natural isomorphism between the moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ and the stable representations of the nested instantons quiver. In the rank $r=1$ case, our definition reduces to the nested Hilbert scheme of points on ${\mathbb C}^2$ , as is to be expected. By this reason, we first want to carry out the analysis of the simpler $r=1$ case, which also has the advantage of providing us with a new characterization of punctual nested Hilbert schemes on ${\mathbb C}^2$ , analogous to that of [8].

2.1 $\operatorname {\mathrm {Hilb}}^{\hat {{\mathbf n}}}({\mathbb C}^2)$ and ${\mathcal N}(1,{\mathbf n})$

Before delving into the analysis of the relation between nested instantons moduli spaces and flags of framed torsion-free sheaves on ${\mathbb P}^2$ , we want to show a special simpler case. In particular, we will prove the existence of an isomorphism between the nested Hilbert scheme of points in ${\mathbb C}^2$ and the nested instantons moduli space ${\mathcal N}(1,n_0,\dots ,n_N)$ . This effectively gives us the ADHM construction of a general nested punctual nested Hilbert scheme on ${\mathbb C}^2$ , which will also be the local model for more general nested Hilbert schemes of points on, say, toric surfaces S. In order to see this, we first recall the definition of a nested Hilbert scheme of points.

Definition 2.1. Let S be a complex (projective) surface and $n_1\ge n_2\ge \cdots \ge n_k$ a sequence of integers. The nested Hilbert scheme of points on S is defined as

(2.1.1) $$ \begin{align} \operatorname{\mathrm{Hilb}}^{(n_1,\dots,n_k)}(S)=S^{[n_1,\dots,n_k]}=\left\{I_1\subseteq I_2\subseteq\cdots\subseteq I_k\subseteq\mathcal O_S:\mathrm{length}\left(\mathcal O_S/I_i\right)=n_i\right\}. \end{align} $$

Alternatively, if X is a quasi-projective scheme over the complex numbers, we can equivalently define the nested Hilbert scheme $X^{[n_1,\dots ,n_k]}=\operatorname {\mathrm {Hilb}}^{(n_1,\dots ,n_k)}(X)$ as

(2.1.2) $$ \begin{align} \operatorname{\mathrm{Hilb}}^{(n_1,\dots,n_k)}(X)=\left\{(Z_1,\dots,Z_k):Z_i\in\operatorname{\mathrm{Hilb}}^{n_i}(X),\ Z_i\ \mathrm{is\ a\ subscheme\ of }\ Z_j\ \mathrm{if }\ i<j\right\}, \end{align} $$

with $Z_i$ being a zero-dimensional scheme, for every $i=1,\dots ,k$ .

Before actually exhibiting the isomorphism we are interested in, we want to prove an auxiliary result, which gives an alternative definition for the nested Hilbert schemes over the affine plane, analogously to the case of Hilbert schemes studied in [43].

Proposition 2.2. Let be an algebraically closed field, and ${\mathbf n}$ a sequence of integers $n_0\ge n_1\ge \cdots \ge n_k$ . Define $\hat {{\mathbf n}}$ to be the sequence of integers $\hat {n}_0=n_0\ge \hat {n}_1=n_0-n_k\ge \cdots \ge \hat {n}_k=n_0-n_1$ . Then there exists an isomorphism

(2.1.3)

where , , and . The action of $\mathcal G_{{\mathbf n}}$ is given by

$$ \begin{align*} \boldsymbol g\cdot(b_1^0,b_2^0,i,\dots,b_1^k,b_2^k,f_k)=(g_0b_1^0g_0^{-1},g_0b_2^0g_0^{-1},g_0i,\dots,g_kb_1^kg_k^{-1},g_kb_2^kg_k^{-1},g_{k-1}f_kg_k^{-1}). \end{align*} $$

Proof. Suppose we have a sequence of ideals $I_0\subseteq I_1\subseteq \cdots I_k\in \operatorname {\mathrm {Hilb}}^{\hat {{\mathbf n}}}(\mathbb A^2)$ . Let us first define , $b_{1,2}^0\in \operatorname {\mathrm {End}}(V_0)$ to be the multiplication by $z_{1,2}\mod I_0$ , and by $i(1)=1\mod I_0$ . Then obviously $[b_1^0,b_2^0]=0$ and condition (iii) holds since $1$ multiplied by products of $z_1$ and $z_2$ spans the whole . Then define , $i>0$ , so that $\dim \tilde V_i=n_0-n_{k-i+1}$ . By letting $V_{k-i+1}=\ker (V_0\twoheadrightarrow \tilde V_i)$ , we have $V_i\subseteq V_{i-1}$ , and $\dim V_i=n_i$ . The restrictions of $b_{1,2}^0$ to $V_i$ then yield homomorphisms $b_{1,2}^i\in \operatorname {\mathrm {End}}(V_i)$ naturally satisfying $[b_1^i,b_2^i]=0$ , while the inclusion of the ideals $I_0\subseteq I_1\subseteq \cdots \subseteq I_k$ implies the existence of an embedding $f_i:V_i\hookrightarrow V_{i-1}$ such that condition (ii) holds by construction.

Conversely, let $(b_1^0,b_2^0,i,\dots ,b_1^k,b_2^k,f_k)$ be given as in the proposition. In the first place, one can define a map to be $\phi _0(f)=f(b_1^0,b_2^0)i(1)$ . This map is surjective, so that $I_0=\ker \phi $ is an ideal for of length $n_0$ . Then, since is injective, we can embed into through $F_i=f_1\circ \cdots \circ f_{i-1}\circ f_i$ in such a way that , which is a simple consequence of condition (ii). Then we have the direct sum decomposition , the restrictions and the projection $\tilde \imath _i=\pi _i\circ i$ , with , satisfying $[\tilde b_1^i,\tilde b_2^i]=0$ and a stability condition analogous to (iii). Thus, we define by $\phi _i(f)=f(\tilde b_1^i,\tilde b_2^i)\tilde \imath (1)$ . This map is surjective, just like $\phi _0$ , so that $I_j=\ker (\phi _j)$ is an ideal for of length $n_0-n_i$ . Finally, due to the successive embeddings , we have the inclusion of the ideals $I_j\subset I_{j-1}$ .

One can readily notice that the description given by the previous proposition of the nested Hilbert scheme of points does not really coincide with the quiver we were studying throughout this section. However, we can very easily overcome this problem by using the fact that if $(b_1^0,b_2^0,i,j)$ is a stable ADHM datum with $r=1$ , then $j=0$ ; [43]. This proves the following proposition.

Proposition 2.3. With the same notations of Proposition 2.2, we have that

All the previous observations, together with corollary 1.7, immediately prove the following theorem.

Theorem 2.4. The moduli space of nested instantons ${\mathcal N}(r,{\mathbf n})$ is isomorphic to the nested Hilbert scheme of points on ${\mathbb C}^2$ when $r=1$ .

$$\begin{align*}{\mathcal N}(1,{\mathbf n})={\mathbb X}_0/\!\!/_\chi\mathcal G\cong\operatorname{\mathrm{Hilb}}^{\hat{{\mathbf n}}}({\mathbb C}^2).\end{align*}$$

Remark 2.5. The isomorphism given in Theorem 2.4 is only valid at the level of points. A version in families of the proofs of Proposition 2.2 and Proposition 2.3 would suffice to get a scheme-theoretic version of this isomorphism. We will not do that here, as the result of Theorem 2.9 already implies it as a special case.

2.2 $\mathcal F(r,\boldsymbol \gamma )$ and ${\mathcal N}(r,{\mathbf n})$

A more general result relates the moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ to the moduli space of nested instantons. In the case of the two-step quiver, this result was proved in [54]. Here, we give a generalization of their theorem in the case of the moduli space ${\mathcal N}_{r,[r^1],n,\mu }$ represented by a quiver with an arbitrary number of nodes.

Definition 2.6. Let $\ell _\infty \subset {\mathbb P}^2$ be a line and F a coherent sheaf on ${\mathbb P}^2$ . A framing $\phi $ for F is then a choice of an isomorphism $\phi :F|_{\ell _\infty }\xrightarrow {\sim }\mathcal O_{\ell _\infty }^{\oplus r}$ , with $r=\operatorname {\mathrm {rk}} F$ . An $(N+2)$ -tuple $(E_0,E_1,\dots ,E_N,\phi )$ is a framed flag of sheaves on ${\mathbb P}^2$ if $E_N$ is a torsion-free (coherent) sheaf on ${\mathbb P}^2$ framed at $\ell _\infty $ by $\phi $ , and $E_{j}$ , $j=0,\dots ,N-1$ , form a flag of subsheaves $E_0\subseteq \cdots \subseteq E_N$ of $E_N$ s.t. the quotients $E_i/E_j$ , $i>j$ , are supported away from $\ell _\infty $ .

By the framing condition, we get that $c_1(E_N)=0$ , while the quotient condition on the subsheaves of $E_N$ naturally implies that the quotients $E_i/E_j$ are $0$ -dimensional sheaves and $c_1(E_j)=0$ , for all $j=0,\dots ,N$ . Then a framed flag of sheaves on ${\mathbb P}^2$ is characterized by the set of integers $(r,\boldsymbol \gamma )$ , where $r=\operatorname {\mathrm {rk}} E_0=\cdots =\operatorname {\mathrm {rk}} E_N$ , $c_2(E_N)=\gamma _N$ , $h^0(E_N/E_j)=\gamma _j+\cdots +\gamma _{N-1}$ so that $c_2(E_j)=\gamma _j+\cdots +\gamma _N$ .

We now define the moduli functor

(2.2.1) $$ \begin{align} \mathsf F_{(r,\boldsymbol\gamma)}:{\mathsf{Sch}}_{{\mathbb C}}^{\mathrm{op}}\to{\mathsf{Sets}} \end{align} $$

by assigning to a ${\mathbb C}$ -scheme S the set

$$ \begin{align*}\mathsf F_{(r,\boldsymbol\gamma)}(S)=\{\mathrm{isomorphism\ classes\ of\ }(2N+2)-\mathrm{ tuples}\ (F_S,\varphi_S,Q^0_S,g^0_S,\dots,Q^{N-1}_S,g^{N-1}_S\}\end{align*} $$

with

• ○ $F_S$ a coherent sheaf over ${\mathbb P}^2\times S$ flat over S and such that $F_S|_{{\mathbb P}^2\times \{s\}}$ is a torsion-free sheaf for any closed point $s\in S$ , $\operatorname {\mathrm {rk}} F_S=r$ , $c_1(F_S)=0$ and $c_2(F_S)=\gamma _N$ ;

• ○ $\varphi _S:F_S|_{\ell _\infty \times S}\xrightarrow {\sim }\mathcal O^{\oplus r}_{\ell _\infty \times S}$ is an isomorphism of $\mathcal O_{\ell _\infty \times S}$ -modules;

• ○ $Q^i_S$ is a coherent sheaf on ${\mathbb P}^2\times S$ , flat over S and supported away from $\ell _\infty \times S$ , such that $h^0(Q_S^i|_{{\mathbb P}^2\times \{s\}})=\gamma _i+\cdots +\gamma _{N-1}$ , for any closed point $s\in S$ ;

• ○ $g^0_S:F_S\to Q^0_S$ and $g^i_S:Q^{i-1}_S\to Q^i_S$ , $i=1,\dots ,N-1$ are surjective morphisms of $\mathcal O_{{\mathbb P}^2\times S}$ -modules.

Two tuples $(F_S,\varphi _S,Q_S^1,g_S^1,\dots ,Q_S^N,g_S^N)$ and $(F_S^{\prime },\varphi _S^{\prime },Q_S^{1,\prime },g_S^{1,\prime },\dots ,Q_S^{{N-1},\prime },g_S^{N-1,\prime })$ are said to be isomorphic if there exist isomorphisms of $\mathcal O_{{\mathbb P}^2\times S}$ -modules $\Theta _S:F_S\to F_S^{\prime }$ and $\Gamma _S^i:Q_S^i\to Q_S^{i\prime }$ such that the following diagrams commute:

If this functor is representable, the variety representing it will be called the moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ .

What we want to show next is that the moduli space of flags of torsion-free sheaves on ${\mathbb P}^2$ is a fine moduli space, and that it is indeed isomorphic (as a scheme) to the moduli space of nested instantons we defined previously. First of all, we will focus our attention on proving the following statement.

Proposition 2.7. The moduli functor $\mathsf F_{(r,\boldsymbol \gamma )}$ is represented by a (quasi-projective) variety $\mathcal F(r,\boldsymbol \gamma )$ isomorphic to a relative quot-scheme.

Proof. Our proof strongly relies on the use of (relative) nested ${\mathsf {Quot}}$ functors, so let us recall their construction and basic properties. First of all, let N be a fixed positive integer and take the universal framed sheaf $(\mathcal F,\psi )$ on ${\mathbb P}^2\times \mathcal M(r,\gamma _N)$ , with $\psi :\mathcal F|_{\ell _\infty \times \mathcal M(r,\gamma _N)}\xrightarrow {\sim }\mathcal O^{\oplus r}_{\ell _\infty \times \mathcal M(r,\gamma _N)}$ an isomorphism of $\mathcal O_{\ell _\infty \times \mathcal M(r,\gamma _N)}$ -modules. Let $\boldsymbol {\gamma }$ an N-tuple of integers $(\gamma _0,\gamma _1,\dots ,\gamma _{N-1})$ . We define the nested ${\mathsf {Quot}}$ functor

(2.2.2) $$ \begin{align} {\mathsf{Quot}}_{(\mathcal F,\boldsymbol\gamma)}:{\mathsf{Sch}}^{\mathrm{op}}_{\mathcal M(r,\gamma_N)}\to{\mathsf{Sets}} \end{align} $$

by

(2.2.3) $$ \begin{align} {\mathsf{Quot}}_{(\mathcal F,\boldsymbol\gamma)}(S)=\{\mathrm{isomorphism\ classes\ of\ }(Q_S^0,q_S^0,\dots,Q_S^{N-1},q_S^{N-1})\}, \end{align} $$

where

• ○ Each $Q^i_S$ is a coherent sheaf on ${\mathbb P}^2\times S$ , flat over S, supported away from $\ell _\infty \times S$ and such that $h^0(Q^i_S|_{\ell _\infty \times \{s\}})=\gamma _i+\cdots +\gamma _{N-1}$ , for any closed point $s\in S$ ;

• ○ $q_S^i:Q_S^{i-1}\to Q_S^i$ is a surjective morphism of $\mathcal O_{{\mathbb P}^2\times S}$ -modules for $i=1,\dots ,N-1$ ;

• ○ $q^0_S:\mathcal F_S\to Q^0_S$ is a surjective morphism of $\mathcal O_{{\mathbb P}^2\times S}$ -modules, where $\mathcal F_S$ is the pullback of $\mathcal F$ to ${\mathbb P}^2\times S$ via

  

By an induction on N along the lines of [31, § 2.A.1] one can prove the representability of the nested ${\mathsf {Quot}}$ functor. Note that when $N=1$ , the nested ${\mathsf {Quot}}$ functor reduces to an ordinary ${\mathsf {Quot}}$ functor, which is representable by Grothendieck’s theory. In this case, representability of $\mathsf F_{(r,\gamma _0,\gamma _1)}$ has been established in [54, Prop. 1]. In general, there is a natural forgetting map $\mathsf F_{(r,\boldsymbol \gamma )}\to {\mathsf {Quot}}_{(\mathcal F,\boldsymbol \gamma )}$ , which acts as $(F_S,\varphi _S,Q_S^0,g_S^0,\dots ,Q_S^{N-1},g_S^{N-1})\mapsto (Q_S^0,g_S^0,\dots ,Q_S^{N-1},g_S^{N-1})$ . This map also has an inverse given by setting $F_S$ to be the framed sheaf $F_S=\ker (g^0_S)$ with framing $\varphi _S$ at $\ell _\infty \times S$ induced by the framing $\psi $ of $\mathcal F$ at $\ell _\infty \times \mathcal M(r,\gamma _N)$ . Thus, the functors $\mathsf F_{(r,\boldsymbol \gamma )}$ and ${\mathsf {Quot}}_{(\mathcal F,\boldsymbol \gamma )}$ are naturally isomorphic, and representability of the latter implies representability of the former. The moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ , representing $\mathsf F_{(r,\boldsymbol \gamma )}$ , is then the nested quot scheme $\operatorname {\mathrm {Quot}}^{\boldsymbol \gamma }(\mathcal F)$ relative to $\mathcal M(r,\gamma _N)$ .

Remark 2.8. The previous description of the moduli space of framed flags of sheaves on ${\mathbb P}^2$ suggests we could also take a slightly different perspective on $\mathcal F(r,\boldsymbol \gamma )$ and study its closed subscheme of the moduli of sequences of quotients

$$\begin{align*}Z_0\hookrightarrow\cdots\hookrightarrow Z_N\hookrightarrow F\twoheadrightarrow Q_0\twoheadrightarrow\cdots\twoheadrightarrow Q_N, \end{align*}$$

where F is the trivial vector bundle $F\cong \mathcal O_{{\mathbb P}^2}^{\oplus r}$ . In this sense, $\mathcal F(r,\boldsymbol \gamma )$ seems to be analogous to the Filt-scheme studied by Mochizuki in [38] in the case of curves.

Now that we proved that the definition of moduli space of framed flags of sheaves on ${\mathbb P}^2$ is indeed a good one, we are ready to tackle the problem of showing that there exists an isomorphism between this moduli space and the space of stable representation of the nested instantons quiver we studied in the previous sections. First of all, let us point out that our definition of flags of framed torsion-free sheaves reduces in the rank $1$ case to the nested Hilbert scheme of points on ${\mathbb C}^2$ , and the isomorphism we are interested in was shown to exist in Theorem 2.4 of §2.1. This is, in fact, compatible with the statement of Theorem 2.9.

Theorem 2.9. The moduli space of stable representations of the nested ADHM quiver is a fine moduli space isomorphic to the moduli space of flags of framed torsion-free sheaves on ${\mathbb P}^2$ : $\mathcal F(r,\boldsymbol \gamma )\cong {\mathcal N}(r,{\mathbf n})$ , as schemes, where $n_i=\gamma _0+\cdots +\gamma _{N-i}$ .

Before diving into the proof of Theorem 2.9, we need to recall a few facts about the moduli space of framed torsion-free sheaves on ${\mathbb P}^2$ . Our main reference is [43, § 2]. Recall then that with any (ordinary) ADHM datum $(W,V,B_1,B_2,I,J)$ is associated a complex of locally free sheaves on ${\mathbb P}^2$ , which we call the ADHM complex of X, of the form

(2.2.4)

where $[x:y:z]$ are homogeneous coordinates on ${\mathbb P}^2$ , $\ell _\infty =\{z=0\}$ is a line at infinity, and

As in [43, Lemma 2.1], $\alpha $ is always injective if X satisfies the ADHM equation, while $\beta $ is surjective if and only if X is stable. The cohomology sheaf is a rank r torsion-free sheaf on ${\mathbb P}^2$ , with $c_2(E)=n$ , framed by the induced isomorphism $E|_{\ell _\infty }\xrightarrow {\sim } W\otimes \mathcal O_{\ell _\infty }$ . Conversely, given any framed torsion-free sheaf E on ${\mathbb P}^2$ , there is a stable ADHM datum X such that E is the cohomology of $E_X^{\bullet }$ . This realizes an isomorphism between the moduli space of framed torsion-free sheaves on ${\mathbb P}^2$ with rank r and second Chern class n, and the moduli space $\mathcal M(r,n)$ of stable representations of numerical type $(r,n)$ of the ADHM quiver; cf. [43, § 2]. Moreover, if we let $\operatorname {\mathsf {Kom}}({\mathbb P}^2)$ be the category of complexes of sheaves on ${\mathbb P}^2$ and $\operatorname {\mathsf {Rep}}_{\text {ADHM}}$ be the category of representations of the ADHM quiver, we have that the functor

$$\begin{align*}\mathsf K:\operatorname{\mathsf{Rep}}_{\text{ADHM}}\to \operatorname{\mathsf{Kom}}({\mathbb P}^2) \end{align*}$$

defined by $\mathsf K(X)=E_X^{\bullet }$ , with the obvious definition for morphisms, is exact and fully faithful; cf. [54, Prop. 20].

Proof of Theorem 2.9.

We first want to show how, starting from an element of ${\mathcal N}(r,n_0,\dots ,n_N)$ , one can construct a flag of framed torsion-free sheaves on ${\mathbb P}^2$ . As we showed previously, to each $(V_i,B_1^i,B_2^i,F^i)$ in the datum of $X\in {\mathcal N}(r,n_0,\dots ,n_N)$ , we can associate a stable ADHM datum $(W,\widetilde V_i,\widetilde B_1^i,\widetilde B_2^i,\tilde I^i,\tilde J^i)$ , fitting in the diagram (2.2.5)

(2.2.5)

where we suppressed all of the endomorphisms $B_{1,2}^i$ , $\widetilde B_{1,2}^j$ . We will then call $\mathbf Z_i$ , $\mathbf S$ and ${\mathbf Q}_i$ the representations of the ADHM data $(\{0\},V_i,B_1^i,B_2^i)$ , $(W,V_0,B_1^0,B_2^0,I,J)$ and $(W,\widetilde V_i,\widetilde B_1^i,\widetilde B_2^i,\tilde I^i,\tilde J^i)$ , respectively. The the diagram (2.2.5) can be restated in the following form in $\operatorname {\mathsf {Rep}}_{\text {ADHM}}$ :

(2.2.6)

Moreover, if $E^{\bullet }_{\mathbf Z_i}$ , $E^{\bullet }_{\mathbf S}$ and $E^{\bullet }_{{\mathbf Q}_i}$ denotes the ADHM complex corresponding to $\mathbf Z_i$ , $\mathbf S$ and ${\mathbf Q}_i$ , the diagram (2.2.6) induces the following

(2.2.7)

Then, since $\mathbf S$ and ${\mathbf Q}_i$ are stable, by [43, Lemma 2.6], one has that ${\mathcal H}^p(E^{\bullet }_{\mathbf S})={\mathcal H}^p(E^{\bullet }_{{\mathbf Q}_i})=0$ , for $p=-1,1$ . Moreover, one can also show that ${\mathcal H}^0(E^{\bullet }_{\mathbf Z_i})=0$ . Indeed, we have

If we let $[x:y:0]=p\in \ell _\infty \subset {\mathbb P}^2$ , we have

and $\ker \beta _{i,p}=0$ , for all $p\in \ell _\infty $ . Thus, ${\mathcal H}^0(E^{\bullet }_{\mathbf Z_i})$ is a zero-dimensional sheaf supported outside of $\ell _\infty $ . In particular, $H^0({\mathcal H}^0(E^{\bullet }_{\mathbf Z_i}))=H^0({\mathcal H}^0(E^{\bullet }_{\mathbf Z_i})(-1))$ , and the right-hand side vanishes. Indeed, consider the following short exact sequences of sheaves:

(2.2.8a)

(2.2.8b)

From Equation (2.2.8a), we get $H^0({\mathcal H}^0(E^{\bullet }_{\mathbf Z_i})(-1))\cong H^0(\ker \beta _i(-1))$ , while from Equation (2.2.8b), we get $H^0(\ker \beta _i(-1))=0$ . Thus, $H^0({\mathcal H}^0(E^{\bullet }_{\mathbf Z_i}))=0$ , and for each line in (2.2.7), the long exact sequence for the cohomology associated to it reduces to

$$\begin{align*}0\longrightarrow {\mathcal H}^0(E^{\bullet}_{\mathbf S})\longrightarrow {\mathcal H}^0(E^{\bullet}_{{\mathbf Q}_i})\longrightarrow {\mathcal H}^1(E^{\bullet}_{\mathbf Z_i})\longrightarrow 0, \end{align*}$$

and by the ADHM construction, $({\mathcal H}^0(E^{\bullet }_{{\mathbf Q}_i}),\varphi )$ is a rank r framed torsion-free sheaf on ${\mathbb P}^2$ , with framing $\varphi :{\mathcal H}^0(E^{\bullet }_{{\mathbf Q}_i})|_{\ell _\infty }\xrightarrow {\cong }W\otimes \mathcal O_{\ell _\infty }$ . Moreover, ${\mathcal H}^0(E^{\bullet }_{\mathbf S})$ is a subsheaf of ${\mathcal H}^0(E^{\bullet }_{{\mathbf Q}_i})$ , and ${\mathcal H}^1(E^{\bullet }_{\mathbf Z_i})$ is a quotient sheaf

$$ \begin{align*}{\mathcal H}^1(E^{\bullet}_{\mathbf Z_i})\cong {\mathcal H}^0(E^{\bullet}_{{\mathbf Q}_i})/{\mathcal H}^0(E^{\bullet}_{\mathbf S}), \end{align*} $$

which is $0$ -dimensional and supported away from $\ell _\infty \subset {\mathbb P}^2$ . Finally, one can immediately see from (2.2.7) that ${\mathcal H}^0(E^{\bullet }_{{\mathbf Q}_i})$ is a subsheaf of ${\mathcal H}^0(E^{\bullet }_{{{\mathbf Q}}_{i+1}})$ . One can moreover check that the numerical invariants classifying flags of sheaves do agree with the statement of the theorem.

Conversely, let $(E_0,\dots ,E_N,\varphi )$ be a flag of framed torsion-free sheaves on ${\mathbb P}^2$ such that $\operatorname {\mathrm {rk}} E_j=r$ , $c_2(E_N)=\gamma _N$ , $h^0(E_N/E_j)=\gamma _j+\cdots +\gamma _{N-1}$ . By definition, each $(E_j,\varphi )$ defines a stable ADHM datum ${\mathbf Q}_j=(\widetilde W_j,\widetilde V_j,\widetilde B_1^j,\widetilde B_2^j,\tilde I^j,\tilde J^j)$ (with the convention of calling $\mathbf S={\mathbf Q}_0$ ), since it can be identified with a framed torsion-free sheaf on ${\mathbb P}^2$ , with $\operatorname {\mathrm {rk}} E_j=r$ , $c_2(E_j)=\gamma _0+\cdots +\gamma _j$ . Moreover, we have the inclusion $E_0\hookrightarrow E_j$ , which induces an epimorphism $\Psi _j:\mathbf S\to {\mathbf Q}_j$ . In fact, we can construct vector spaces $V_0$ , $\widetilde V_j$ , W and $\widetilde W_j$ as in [43], so that

$$\begin{align*}V_0\cong H^0(E_0(-1)),\quad \widetilde V_j\cong H^0(E_j(-1)),\quad W\cong H^0\left(E_0|_{\ell_\infty}\right),\quad \widetilde W_j\cong H^0\left(E_j|_{\ell_\infty}\right), \end{align*}$$

and by the fact that the quotient sheaf $E_j/E_0$ is $0$ -dimensional and supported away from $\ell _\infty $ , we can construct an isomorphism

$$\begin{align*}\Psi_{j,2}:H^0\left(E_0|_{\ell_\infty}\right)\xrightarrow{\cong}H^0\left(E_j|_{\ell_\infty}\right). \end{align*}$$

Finally, we have the exact sequence

$$ \begin{align*} 0\longrightarrow E_0\longrightarrow E_j\longrightarrow E_j/E_0\longrightarrow 0, \end{align*} $$

which induces the following exact sequence of cohomology, thanks to the fact that $H^0(E_j(-1))=0$ , being that $E_j$ is a framed torsion-free $\mu $ -semistable sheaf with $c_1(E_j)=0$ (due to the standard ADHM construction), and $H^1(E_j/E_0(-1))=0$ , since the quotient sheaf $E_j/E_0$ is $0$ -dimensional,

$$\begin{align*}0\longrightarrow H^0(E_j/E_0(-1))\longrightarrow H^1(E_0(-1))\xrightarrow{\Psi_{j,1}} H^1(E_j(-1))\longrightarrow 0. \end{align*}$$

The morphism $\Psi _j=(\Psi _{j,1},\Psi _{j,2})$ is then an epimorphism, since both $\Psi _{j,1}$ and $\Psi _{j,2}$ are surjective. Taking into account the flag structure of the datum $(E_0,\dots ,E_N,\varphi )$ , the sequences

give us $(N+1)$ stable ADHM data fitting in the following diagram:

Finally, we need to show that there exists a scheme-theoretic isomorphism between the moduli space of flags of sheaves $\mathcal F(r,\boldsymbol \gamma )$ and the nested ADHM moduli space ${\mathcal N}(r,{\mathbf n})$ . Generalizing the proof of [54, Thm. 18], one may first notice that the complex of sheaves on ${\mathbb P}^2$ in Equation (2.2.4) can be regarded as a family of complexes parametrized by $\mathcal M(r,n_0)$ whose cohomology, by [43, § 2] and the relative version of Beilinson’s Theorem (cf. for instance, [7, § 3.4]), yields a family of framed torsion-free sheaves on ${\mathbb P}^2$ parametrized by $\mathcal M(r,n_0)$ . Similarly, diagram (2.2.5) enables us to think of (2.2.7) as sequences of complexes parameterized by ${\mathcal N}(r,{\mathbf n})$ (i.e., complexes of sheaves on ${\mathbb P}^2\times {\mathcal N}(r,{\mathbf n})$ ). Thus, passing to cohomology, we get a family of framed torsion-free sheaves also parameterized by ${\mathcal N}(r,{\mathbf n})$ , and the isomorphism we were just describing may be regarded as an element of $\mathsf F_{(r,\boldsymbol \gamma )}({\mathcal N}(r,{\mathbf n}))$ , to which we may associate a unique bijective morphism of schemes ${\mathcal N}(r,{\mathbf n})\to \mathcal F(r,\boldsymbol \gamma )$ by the representability of $\mathsf F_{(r,\boldsymbol \gamma )}$ . Conversely, suppose we are given coherent sheaves $(F_S,Q^0_S,\dots ,Q^{N-1}_S)$ on ${\mathbb P}^2\times S$ , flat over S, defining a family of flags of framed torsion-free sheaves $(F_S,\varphi _S,Q^0_S,g^0_S,\dots ,Q^{N-1}_S,g^{N-1}_S)$ . We can associate to this family of sheaves a family of representations of the nested ADHM quiver parameterized by S (i.e., a morphism $S\to {\mathcal N}(r,{\mathbf n})$ ). Then, corresponding to the universal family, we get a morphism $\mathcal F(r,\boldsymbol \gamma )\to {\mathcal N}(r,{\mathbf n})$ .

3 Virtual invariants

In this section, we study fixed points under the action of a torus ${\mathbb T}$ on the moduli space of framed stable representations of fixed numerical type of the nested instantons quiver. By doing this, we can apply virtual equivariant localization and compute certain relevant virtual invariants. More precisely, since our moduli space ${\mathcal N}(r,{\mathbf n})$ is quasi-projective, hence non-compact, we define invariants in (equivariant) localized K-theory, using the localization theorem [52, Thm. 2.1] to push forward along the structure morphism. Indeed, as we will see in §3.1, we have a natural action of a torus ${\mathbb T}=({\mathbb C}^{\ast })^2\times ({\mathbb C}^{\ast })^r$ on ${\mathcal N}(r,{\mathbf n})$ , and the perfect obstruction theory lifts to the equivariant setting. Moreover, the ${\mathbb T}$ -fixed locus in ${\mathcal N}(r,{\mathbf n})$ is compact, so we will define the general K-theoretic invariants via the composition

$$\begin{align*}\chi\left({\mathcal N}(r,{\mathbf n}),-\right):K_0^{{\mathbb T}}\left({\mathcal N}(r,{\mathbf n})\right)\to K_0^{{\mathbb T}}\left({\mathcal N}(r,{\mathbf n})\right)_{\mathrm{loc}}\cong K_0^{{\mathbb T}}\left({\mathcal N}(r,{\mathbf n})^{{\mathbb T}}\right)_{\mathrm{loc}}\to K_0^{{\mathbb T}}(\operatorname{\mathrm{pt}})_{\mathrm{loc}}, \end{align*}$$

where the first map is a suitable localization in $K_0^{{\mathbb T}}({\mathcal N}(r,{\mathbf n}))$ , the isomorphism follows from Thomason’s abstract localization [52], and the last map is proper pushforward on the ${\mathbb T}$ -fixed locus. On the physics side, this is equivalent to the computation of partition functions of some suitable quiver GLSM theory by means of the SUSY localization technique.

3.1 Equivariant torus action and localization

Given an algebraic torus ${\mathbb T}=({\mathbb C}^{\ast })^n$ , any finite-dimensional ${\mathbb T}$ -representation V splits in a direct sum of its weights, which are one-dimensional ${\mathbb T}$ -representations. Each of the weights appearing in the decomposition of the ${\mathbb T}$ -representation V corresponds to a character $\mu \in \widehat {{\mathbb T}}=\operatorname {\mathrm {Hom}}({\mathbb T},{\mathbb C}^{\ast })\cong \mathbb Z^n$ , and thus to a monomial $t^\mu =t_1^{\mu _1}\cdots t_n^{\mu _n}$ in the coordinates $(t_1,\dots ,t_n)$ of ${\mathbb T}$ . We have then a map $\operatorname {\mathrm {tr}}:K_0^{{\mathbb T}}(\operatorname {\mathrm {pt}})\to \mathbb Z[t^\mu :\mu \in \widehat {{\mathbb T}}]$ on the representation ring $K_0^{{\mathbb T}}(\operatorname {\mathrm {pt}})\cong R({\mathbb T})$ , sending a ${\mathbb T}$ -representation V to its decomposition into weight spaces $\operatorname {\mathrm {tr}}_V=\sum _{\mu }t^\mu $ . Since this map is an isomorphism, in what follows, we will often identify (virtual) ${\mathbb T}$ -modules with their characters.

Consider the tori ${\mathbb T}_1=({\mathbb C}^{\ast })^2$ and ${\mathbb T}_2=({\mathbb C}^{\ast })^r$ . Let $T_1,T_2$ and $R_1,\dots ,R_r$ be the generators of the representation rings $R({\mathbb T}_1)\cong K_{{\mathbb T}_1}^0(\operatorname {\mathrm {pt}})$ and $R({\mathbb T}_2)\cong K_{{\mathbb T}_2}^0(\operatorname {\mathrm {pt}})$ , respectively. On ${\mathbb X}(r,{\mathbf n})$ , there is an action of the algebraic torus ${\mathbb T}={\mathbb T}_1\times {\mathbb T}_2$ . Indeed, given $X\in {\mathbb X}(r,{\mathbf n})$ , a torus element $\mathbf t=(T_1,T_2,R)$ , $R\in ({\mathbb C}^{\ast })^r\subset \operatorname {\mathrm {GL}}_r$ , acts on X as

$$\begin{align*}\mathbf t\cdot X=(T_1B_1^0,T_2B_2^0,IR^{-1},RT_1T_2J,\dots,T_1B_1^N,T_2B_2^N,F^N). \end{align*}$$

The ${\mathbb T}$ -action we just described commutes with the $\mathcal G$ -action of §1.2, so it descends to an action on the GIT quotient ${\mathcal N}(r,{\mathbf n})$ . Similarly, the vector bundle ${\mathcal E}$ in Theorem 1.8 is naturally equivariant; thus, the perfect obstruction theory $\mathbb E\xrightarrow {\phi }{\mathcal N}(r,{\mathbf n})$ lifts to the derived category $\mathbf D^{[-1,0]}_{{\mathbb T}}({\mathcal N}(r,{\mathbf n}))$ of ${\mathbb T}$ -equivariant coherent sheaves on ${\mathcal N}(r,{\mathbf n})$ . Indeed, one can introduce a natural ${\mathbb T}$ -equivariant structure on the vector bundles $C^i$ and the bundle homomorphism $d^i:C^i\to C^{i+1}$ in such a way that the section $s\in H^0(\mathbb Y(r,{\mathbf n}),C^2)$ is ${\mathbb T}$ -equivariant. These then restrict to the stable locus and descend to ${\mathbb T}$ -equivariant vector bundles $\mathcal C^i$ and bundle maps $\delta ^i:\mathcal C^i\to \mathcal C^{i+1}$ over $\mathcal Y(r,{\mathbf n})$ , together with a ${\mathbb T}$ -equivariant section $\sigma \in H^0(\mathcal Y(r,{\mathbf n}),\mathcal C^2)$ .

In the following, we will denote by $Q=T_1\cdot {\mathbb C}\oplus T_2\cdot {\mathbb C}\in K_0^{{\mathbb T}_1}(\operatorname {\mathrm {pt}})$ the ${\mathbb T}_1$ -representation corresponding to ${\mathbb C}^2$ , and by $\Lambda ^2Q=T_1T_2\cdot {\mathbb C}\in K_0^{{\mathbb T}_1}(\operatorname {\mathrm {pt}})$ its top exterior power.

We begin the analysis of the fixed locus under the ${\mathbb T}$ -action on the moduli space of nested instantons with a brief recall of the results obtained in [54] and show how they enable us to fully characterize the ${\mathbb T}$ -fixed locus of the two-step nested instantons quiver. The main result we want to recall is the following theorem.

Theorem 3.1 (von Flach-Jardim, [54]).

The moduli space ${\mathcal N}(r,n_0,n_1)\cong \mathcal F(r,n_0-n_1,n_1)$ of stable representations of the nested ADHM quiver is a quasi-projective variety equipped with a perfect obstruction theory. Its ${\mathbb T}$ -equivariant deformation-obstruction complex is the following:

with

$$ \begin{align*} \left\{ \begin{aligned} &d_0(h_0,h_1)=\left([h_0,B_1^0],[h_0,B_2^0],h_0I,-Jh_0,[h_1,B_1^1],[h_1,B_2^1],h_0F-Fh_1\right), \\ &d_1(b_1^0,b_2^0,i,j,b_1^1,b_2^1,f)=\left([b_1^0,B_2^1]+[B_1^0,b_2^0]+iJ+Ij,B_1^0f+b_1^0F-Fb_1^1-fB_1^1,\right. \\ &\qquad\qquad\qquad\qquad\qquad\quad \left.B_2^0f+b_2^0F-Fb_2^1-fB_2^1,jF+Jf,[b_1^1,B_2^1]+[B_1^1,b_2^1]\right),\\ &d_2(c_1,c_2,c_3,c_4,c_5)=c_1F+B_2^0c_2-c_2B_2^1+c_3B_1^0-B_1^1c_3-Ic_4-Fc_5. \end{aligned}\right. \end{align*} $$

Thus, the infinitesimal deformation space and the obstruction space at any X will be isomorphic to $H^1[\mathcal C(X)]$ and $H^2[\mathcal C(X)]$ , respectively. ${\mathcal N}(r,n_1,n_2)$ is smooth iff $n_1=1$ ([10]).

Moreover, it turns out that there exists a surjective morphism $\mathfrak q:(B_1^0,B_2^0,I,J,B_1^1,B_2^1,F)\mapsto (B_1^{\prime },B_2^{\prime },I',J')$ mapping the nested ADHM data of type $(r,n_0,n_1)$ to the ADHM data of numerical type $(r,n_0-n_1)$ ; [54]. Thus, we have two different maps sending the moduli space of stable representations of the nested ADHM quiver to the moduli space of stable representations of ADHM data. Moreover, from Theorem 2.9, one deduces that ${\mathcal N}(r,n_0,n_1)$ can be identified as the incidence variety

$$\begin{align*}{\mathcal N}(r,n_0,n_1)\hookrightarrow\mathcal M(r,n_0)\times\mathcal M(r,n_0-n_1), \end{align*}$$

by means of which one can characterize ${\mathbb T}$ -fixed points of ${\mathcal N}(r,n_0,n_1)$ in terms of the fixed points of $\mathcal M(r,n_0)$ and $\mathcal M(r,n_0-n_1)$ . For the sake of simplicity, let us first focus on the $r=1$ case. We can first take the decomposition $V_0=V\oplus V_1$ , then decompose the vector spaces $V_0$ , V with respect to the action of ${\mathbb T}$ : if $\lambda _0:{\mathbb T}\to \operatorname {\mathrm {End}}(V_0)$ and $\lambda :{\mathbb T}\to \operatorname {\mathrm {End}}(V)$ are morphisms for the toric action on $V_0$ , V, we have

$$\begin{align*}\left\{ \begin{aligned} V & = \bigoplus_{k,l} V(k,l) = \bigoplus_{k,l}\left\{v\in V|\lambda(t)v=t_1^kt_2^lv\right\}\\ V_0 & = \bigoplus_{k,l} V_0(k,l) = \bigoplus_{k,l}\left\{v_0\in V_0|\lambda_0(t)v_0=t_1^kt_2^lv_0\right\}. \end{aligned} \right. \end{align*}$$

Thus, if $X=(W,V,B_1^{\prime },B_2^{\prime },I',J')$ , $X_0=(W,V_0,B_1^0,B_2^1,I,J)$ are ${\mathbb T}$ -fixed points, the very well-known results about the classification of fixed points for ADHM data leads us to the following commutative diagram:

(3.1.1)

Proposition 3.2. Let $X\in {\mathbb X}_0$ be a ${\mathbb T}$ -fixed point. The following statements hold:

1. 1. If $k>0$ or $l>0$ , then $V_{0}(k,l)=0$ , $V(k,l)=0$ ;

2. 2. $\dim V_{0}(k,l)\le 1$ , $\forall k,l$ and $\dim V(k,l)\le 1$ , $\forall k,l$ ;

3. 3. If $k,l\le 0$ , then $\dim V_{0}(k,l)\ge \dim V_{0}(k-1,l)$ , $\dim V_{0}(k,l)\ge \dim V_{0}(k,l-1)$ , $\dim V(k,l)\ge \dim V(k-1,l$ , $\dim V(k,l)\ge \dim V(k,l-1)$ and $\dim V_0(k,l)\ge \dim V(k,l)$ .

The previous propositions give us an easy way of visualizing fixed points of the ${\mathbb T}$ -action on the nested ADHM data. If we suitably normalize each nonzero map to $1$ by the action of $\prod _{k,l}\operatorname {\mathrm {GL}}(V_0(k,l))\times \prod _{k',l'}\operatorname {\mathrm {GL}}(V(k',l'))$ each critical point point can be put into one-to-one correspondence with nested Young diagrams $Y_\mu \subseteq Y_\nu $ . Thus, the fixed points of the original nested ADHM data are classified by couples $(\nu ,\nu \setminus \mu )$ , where $\mu \subset \nu $ and $\nu \setminus \mu $ is the skew Young diagram constructed by taking the complement of $\mu $ in $\nu $ .

If we now take a fixed point $Z=(\nu ,\mu )$ and define $\nu _i=\sum _k\dim V_0(k,1-i)$ , $\nu ^{\prime }_j=\sum _l\dim V_0(1-j,l)$ and similarly $\mu _i=\sum _k\dim V(k,1-i)$ , $\mu ^{\prime }_j=\sum _l\dim V(1-j,l)$ , we can regard $V_0$ and V as ${\mathbb T}$ -modules and write them as

$$\begin{align*}\left\{ \begin{aligned} &V_0=\bigoplus_{k,l}V_0(k,l)=\sum_{i=1}^{M_1}\sum_{j=1}^{\nu^{\prime}_i}T_1^{-i+1}T_2^{-j+1}= \sum_{j=1}^{N_1}\sum_{j=1}^{\nu_j}T_1^{-i+1}T_2^{-j+1}\\ &V=\bigoplus_{k,l}V(k,l)=\sum_{i=1}^{M_2}\sum_{j=1}^{\mu^{\prime}_i}T_1^{-i+1}T_2^{-j+1}= \sum_{j=1}^{N_2}\sum_{j=1}^{\mu_j}T_1^{-i+1}T_2^{-j+1}\\ \end{aligned} \right. \end{align*}$$

with $M_1=\nu _1$ , $M_2=\mu _1$ , $N_1=\nu ^{\prime }_1$ , $N_2=\mu ^{\prime }_1$ . If we now take $V_0=V\oplus V_1$ , we have

$$\begin{align*}V_1=\sum_{(i,j)\in\nu\setminus\mu}T_1^{-i+1}T_2^{-j+1}= \sum_{i=1}^{M_1}\sum_{j=1}^{\nu^{\prime}_i-\mu^{\prime}_i}T_1^{-i+1}T_2^{-\mu_i^{\prime}-j+1}. \end{align*}$$

The virtual tangent space $T^{\textrm {vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z\in K_0^{{\mathbb T}}(\operatorname {\mathrm {pt}})$ to ${\mathcal N}(1,n_0,n_1)$ at the ${\mathbb T}$ -fixed point Z can be regarded as a ${\mathbb T}$ -module, so that

$$\begin{align*}\begin{aligned} T_Z^{\textrm{vir}}{\mathcal N}(1,n_0,n_1)&=\operatorname{\mathrm{End}}(V_0)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{End}}(V_1)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{Hom}}(W,V_0)\\ &\quad+\operatorname{\mathrm{Hom}}(V_0,W)\otimes\Lambda^2Q-\operatorname{\mathrm{Hom}}(V_1,W)\otimes\Lambda^2Q+\operatorname{\mathrm{Hom}}(V_1,V_0)(1+\Lambda^2Q-Q)\\ &=(V_1\otimes V_0^{\ast}+V_1\otimes V_1^{\ast}-V_1^{\ast}\otimes V_0)\otimes(Q-1-\Lambda^2Q)+V_0+V_0^{\ast}\otimes\Lambda^2Q\\ &\quad-V_1^{\ast}\otimes\Lambda^2Q. \end{aligned} \end{align*}$$

In the first place, we might recognize the term $V_0^{\ast }\otimes V_0\otimes (Q-\Lambda ^2Q-1)+V_0+V_0^{\ast }\otimes \Lambda ^2Q$ in the sum as being the tangent space at the moduli space of stable representation of the ADHM quiver $T_{\mathcal M(1,n_0)}|_{\tilde Z}$ , with $\tilde Z=(\nu )$ . Thus, we have

(3.1.2) $$ \begin{align} \begin{aligned} T^{\textrm{vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z&=T_{\mathcal M(1,n_0)}|_{\tilde Z}+(V_1\otimes V_1^{\ast}-V_1^{\ast}\otimes V_0)\otimes(Q-1-\Lambda^2Q)-V_1^{\ast}\otimes\Lambda^2Q. \end{aligned} \end{align} $$

Then

$$\begin{align*}\begin{aligned} V_1^{\ast}\otimes(Q-1-\Lambda^2Q)&=(T_1-1)(1-T_2)\sum_{i=1}^{M_1}\sum_{j=1}^{\nu_i^{\prime}-\mu_i^{\prime}}T_1^{i-1}T_2^{\mu_i^{\prime}+j-1}\\ &=(T_1-1)\sum_{i=1}^{M_1}T_1^{i-1}T_2^{\mu_i^{\prime}-1}(1-T_2)\sum_{j=1}^{\nu^{\prime}_i-\mu^{\prime}_i}T_2^{j}\\ &=(T_1-1)\sum_{i=1}^{M_1}T_1^{i-1}T_2^{\mu_i^{\prime}-1}(1-T_2)\left(\frac{1-T_2^{\nu_i^{\prime}-\mu^{\prime}_i+1}}{1-T_2}-1\right)\\ &=(T_1-1)\sum_{i=1}^{M_1}T_1^{i-1}T_2^{\mu_i^{\prime}-1}(1-T_2)\left(\frac{T_2-T_2^{\nu_i^{\prime}-\mu^{\prime}_i+1}}{1-T_2}\right)\\ &=(T_1-1)\sum_{i=1}^{M_1}T_1^{i-1}(T_2^{\mu_i^{\prime}}-T_2^{\nu_i^{\prime}}), \end{aligned} \end{align*}$$

so that

$$\begin{align*}\begin{aligned} V_1^{\ast}\otimes V_1\otimes(Q-1-\Lambda^2Q)&=(T_1-1)\sum_{j=1}^{N_1}\sum_{j'=1}^{\nu_{j}-\mu_{j}}T_1^{-\mu_j-j'+1}T_2^{-j+1}\sum_{i=1}^{M_1}T_1^{i-1}(T_2^{\mu_i^{\prime}}-T_2^{\nu_i^{\prime}})\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1} T_1^{i-\mu_j}(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1})(T_1-1)\sum_{j'=1}^{\nu_j-\mu_j}T_1^{-j'}\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}T_1^{i-\mu_j}(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1})(T_1-1)\left(\frac{1-T_1^{-\nu_j+\mu_j-1}}{1-T_1^{-1}}-1\right)\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}(T_1^{i-\mu_j}-T_1^{i-\nu_j})(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1}), \end{aligned} \end{align*}$$

while we have

$$\begin{align*}\begin{aligned} V_1^{\ast}\otimes V_0\otimes(Q-1-\Lambda^2Q)&=(T_1-1)\sum_{j=1}^{N_1}\sum_{j'=1}^{\nu_{j}}T_1^{-j'+1}T_2^{-j+1}\sum_{i=1}^{M_1} T_1^{i-1}(T_2^{\mu_i^{\prime}}-T_2^{\nu_i^{\prime}})\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}T_1^i(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1})(T_1-1)\sum_{j'=1}^{\nu_j}T_1^{-j'}\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}T_1^i(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1})(T_1-1)\left(\frac{1-T_1^{-\nu_j-1}}{1-T_1^{-1}}-1\right)\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}(T_1^i-T_1^{i-\nu_j})(T_2^{-j+\mu^{\prime}_i+1}-T_2^{-j+\nu^{\prime}_i+1}), \end{aligned} \end{align*}$$

and

$$\begin{align*}\begin{aligned} V_1^{\ast}\otimes\Lambda^2Q&=T_1T_2\sum_{i=1}^{M_1}\sum_{j=1}^{\nu_i^{\prime}-\mu_i^{\prime}}T_1^{i-1}T_2^{\mu_i^{\prime}+j-1}\\ &=\sum_{i=1}^{M_1}\sum_{j=1}^{\nu_i^{\prime}-\mu_i^{\prime}}T_1^iT_2^{\mu_i^{\prime}+j}. \end{aligned} \end{align*}$$

Assembling everything together, we finally get that

(3.1.3) $$ \begin{align} \begin{aligned} T_{{\mathcal N}(1,n_0,n_1)}^{\textrm{vir}}|_Z=&T_{\mathcal M(1,n_0)}|_{\tilde Z}+\sum_{i=1}^{M_1}\sum_{j=1}^{N_1}(T_1^{i-\mu_j}-T_1^{i})(T_2^{-j+\mu_i^{\prime}+1}-T_2^{-j+\nu_i^{\prime}+1})\\ &-\sum_{i=1}^{M_1}\sum_{j=1}^{\nu_i^{\prime}-\mu_i^{\prime}}T_1^iT_2^{j+\mu_i^{\prime}}. \end{aligned} \end{align} $$

As an immediate generalization of (3.1.3), we can easily see that the ( ${\mathbb T}$ -equivariant) K-theory class of the virtual tangent space to ${\mathcal N}(r,n_0,n_1)$ at a fixed point Z is

$$\begin{align*}\begin{aligned} T_{{\mathcal N}(r,n_0,n_1)}^{\textrm{vir}}|_Z &=T_{\mathcal M(r,n_0)}|_{\tilde Z}+\sum_{a,b=1}^r\sum_{i=1}^{M_1^{(a)}}\sum_{j=1}^{N_1^{(b)}}R_bR_a^{-1}\left(T_1^{i-\mu_j^{(b)}}-T_1^{i}\right)\\ & \quad \times\left(T_2^{-j+\mu_i^{(a)'}+1}-T_2^{-j+\nu_i^{(a)'}+1}\right)-\sum_{i=1}^{M_1^{(a)}}\sum_{j=1}^{\nu_i^{(a)'}-\mu_i^{(a)'}}T_1^iT_2^{j+\mu_i^{(a)'}}. \end{aligned} \end{align*}$$

Remark 3.3. It turns out that the character representation of the virtual tangent $T^{\mathrm {vir}}_{{\mathcal N}}|_Z$ can be computed by exploiting deformation theory techniques. These techniques may also be employed to compute the virtual fundamental class and ( ${\mathbb T}_1$ -character of) the virtual tangent bundle at fixed points of nested Hilbert schemes on surfaces, as it is done in [19].

If, in particular, one takes $({\mathbb C}^2)^{[N_0\ge N_1]}$ to be the nested Hilbert scheme of points on ${\mathbb C}^2=\operatorname {\mathrm {Spec}}({\mathbb C}[x_0,x_1])$ , by lifting the natural torus action on ${\mathbb C}^2$ to $\left({\mathbb C}^2\right)^{[N_0\ge N_1]}$ , it is proved in [19] that the ${\mathbb T}_1$ -fixed locus is isolated and given by the inclusion of monomial ideals $I_0\subseteq I_1$ , which is equivalent to the assignment of couples of nested partitions $\mu \subseteq \nu $ . Then the virtual tangent space at a fixed point is given by

$$ \begin{align*} T_{I_0\subseteq I_1}^{\mathrm{vir}}=-\chi(I_0,I_0)-\chi(I_1,I_1)+\chi(I_0,I_1)+\chi(R,R), \end{align*} $$

with $\chi (-,-)=\sum _{i=0}^2(-1)^i\operatorname {\mathrm {Ext}}_R^i(-,-)$ . Then the ${\mathbb T}_1$ -representation of $T^{\mathrm { vir}}_{I_0\subseteq I_1}$ can be explicitly written in terms of Laurent polynomials in the torus characters $t_1,t_2$ of ${\mathbb T}_1$ . Then in terms of the characters $\mathsf Z_0,\mathsf Z_1$ of the ${\mathbb T}_1$ -fixed $0$ -dimensional subschemes $Z_1\subseteq Z_0\subset {\mathbb C}^2$ corresponding to $I_0\subseteq I_1$ , one has (cf. [19, Eq. (29)])

$$ \begin{align*} \operatorname{\mathrm{tr}} T^{\mathrm{vir}}_{I_0\subseteq I_1}=\mathsf Z_0+\frac{\overline{\mathsf Z}_1}{t_1t_2}+\left(\overline{\mathsf Z}_0\mathsf Z_1-\overline{\mathsf Z}_0\mathsf Z_0-\overline{\mathsf Z}_1\mathsf Z_1\right)\frac{(1-t_1)(1-t_2)}{t_1t_2}. \end{align*} $$

If we now make the necessary identifications $t_i=T_i^{-1}$ , $\mathsf Z_0=V_0$ and $\mathsf Z_1=V$ , we can see that Equation (29) of [19] exactly agrees with our prescription for the character representation (3.1.2) of the virtual tangent space $T^{\mathrm {vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z$ , with $n_0=N_0$ and $n_1=N_0-N_1$ .⁵

We now move on studying the fixed locus of the more general nested instantons moduli space ${\mathcal N}(r,n_0,\dots ,n_N)$ . However, similarly to the previous case, we first want to show that the moduli space of stable representations of the nested ADHM quiver is equivalently described by the datum of $(N+1)$ moduli spaces of framed torsion-free sheaves on ${\mathbb P}^2$ – namely, $\mathcal M(r,n_0),\mathcal M(r,n_0-n_1),\dots ,\mathcal M(r,n_0-n_N)$ . To do this, we want to know if it is possible to recover the structure of the nested ADHM quiver given a set of stable ADHM data. First of all, we can notice that as $F^i$ is injective $\forall i$ , we have the sum decomposition $V_0=V_i\oplus \tilde V_i$ , where suitable choices of bases of $V_i$ are made so that Equation (1.2.10) holds. We also have, with analogous choices being made, $V_i=V_{i+1}\oplus \hat V_{i+1}$ , with $\hat {V}_{i+1}=V_i/\operatorname {\mathrm {Im}} F_i$ , so that $V_0=V_i\oplus \hat V_i\oplus \tilde V_{i-1}$ , thus $\tilde V_i=\hat V_i\oplus \tilde V_{i-1}$ .

Let us first focus on the vector spaces $V_0$ and $V_1$ . It can be shown as in [7, 54] that once we fix a stable ADHM datum $(W,\tilde V_1,\tilde B_1^1,\tilde B_2^1,\tilde I^1,\tilde J^1)$ and the endomorphisms $B_1^1,B_2^1\in \operatorname {\mathrm {End}} V_1$ , it is always possible to reconstruct the stable ADHM datum $(W,V_0,B_1^0,B_2^0,I,J)$ as

(3.1.4) $$ \begin{align} B_1^0=\begin{pmatrix} B_1^1 & B_1^{'1}\\ 0 & \tilde B_1^1 \end{pmatrix},\qquad B_2^0=\begin{pmatrix} B_2^1 & B_2^{'1}\\ 0 & \tilde B_2^1 \end{pmatrix},\qquad I=\begin{pmatrix} I^{'1}\\ \tilde I^1 \end{pmatrix},\qquad J=\begin{pmatrix} 0 & \tilde J^1 \end{pmatrix} \end{align} $$

together with the morphism such that $[B_1^1,B_2^1]=0$ , $B_1^0F^1-F^1B_1^1=B_2^0F^1-F^1B_2^1=0$ and $JF^1=0$ . The same can obviously be done for any of the stable ADHM data $(W,\tilde V_i,\tilde B_1^i,\tilde B_2^i,\tilde I^i,\tilde J^i)$ we constructed previously, and we would have

(3.1.5) $$ \begin{align} B_1^0=\begin{pmatrix} B_1^i & B_1^{'i}\\ 0 & \tilde B_1^i \end{pmatrix},\qquad B_2^0=\begin{pmatrix} B_2^i & B_2^{'i}\\ 0 & \tilde B_2^i \end{pmatrix},\qquad I=\begin{pmatrix} I^{'i}\\ \tilde I^i \end{pmatrix},\qquad J=\begin{pmatrix} 0 & \tilde J^i \end{pmatrix} \end{align} $$

together with the morphism such that $[B_1^i,B_2^i]=0$ , $B_1^0f^i-f^iA_i=B_2^0f^i-f^iB_2^i=0$ and $Jf^i=0$ . If we now fix

(3.1.6)

which is clearly injective, then obviously $f^i=F^1F^2\cdots F^i$ , where $F^j$ now stands for the linear extension to $V_0$ , and $B_1^0f^i-f^iB_1^i=0$ (resp. $B_2^0f^i-f^iB_2^i=0$ ) is equivalent to $B_1^0F^1F^2\cdots F^{i-1}F^i-F^1F^2\cdots F^iB_1^i=B_1^{i-1}F^i-F^iB_1^i=0$ (resp. $B_2^{i-1}F^i-F^iB_2^i=0$ ), and $Jf^i=JF^1F^2\cdots F^i=0$ . This construction makes it possible to us to classify the ${\mathbb T}$ -fixed locus of ${\mathcal N}(r,n_0,\dots ,n_N)$ in terms of the ${\mathbb T}$ -fixed loci of $\mathcal M(r,n_0)$ and $\{\mathcal M(r,n_0-n_i)\}_{i>0}$ . In particular, the ${\mathbb T}$ -fixed locus of $\mathcal M(r,k)$ is into $1-1$ correspondence with coloured partitions $\boldsymbol \mu =(\mu ^1,\dots ,\mu ^r)\in \mathcal P^r$ such that $|\boldsymbol \mu |=|\mu ^1|+\cdots +|\mu ^r|=k$ . This fact and the inclusion relations between the vector spaces $V_i$ prove the following.

Proposition 3.4. The ${\mathbb T}$ -fixed locus ${\mathcal N}(r,n_0,\dots ,n_N)^{{\mathbb T}}$ is in bijection with $(N+1)$ -tuples of nested coloured partitions $\boldsymbol \mu _1\subseteq \cdots \subseteq \boldsymbol \mu _N\subseteq \boldsymbol \mu _0$ , with $|\boldsymbol \mu _0|=n_0$ and $|\boldsymbol \mu _{i>0}|=n_0-n_i$ .

As we pointed out in Theorem 1.8, we can read the ( ${\mathbb T}$ -equivariant) K-theory class of the virtual tangent space to ${\mathcal N}(r,{\mathbf n})$ at a fixed point $Z\in {\mathcal N}(r,{\mathbf n})^{{\mathbb T}}$ off the following equivariant complex:

giving us (3.1.7).

(3.1.7) $$ \begin{align} \begin{aligned} T^{\textrm{vir}}_{{\mathcal N}(1,{\mathbf n})}|_{Z}&=\operatorname{\mathrm{End}}(V_0)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{Hom}}(W,V_0)+\operatorname{\mathrm{Hom}}(V_0,W)\otimes\Lambda^2Q\\ &\quad+\operatorname{\mathrm{End}}(V_1)\otimes(Q-1-\Lambda^2Q)-\operatorname{\mathrm{Hom}}(V_1,W)\otimes\Lambda^2Q\\ &\quad+\operatorname{\mathrm{Hom}}(V_1,V_0)\otimes(1+\Lambda^2Q-Q)\\ &\quad+\operatorname{\mathrm{End}}(V_2)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{Hom}}(V_2,V_1)\otimes(1+\Lambda^2Q-Q)+\\ &\quad\cdots\\ &\quad+\operatorname{\mathrm{End}}(V_N)\otimes(Q-1-\Lambda^2Q)+\operatorname{\mathrm{Hom}}(V_N,V_{N-1})\otimes(1+\Lambda^2Q-Q) \end{aligned} \end{align} $$

By decomposing the vector spaces $V_i$ in terms of the ${\mathbb T}$ -characters, we can also rewrite the representation of (3.1.7) in $R({\mathbb T})$ as

$$\begin{align*}\begin{aligned} T^{\textrm{vir}}_{{\mathcal N}(r,{\mathbf n})}|_Z & = T_{\mathcal M(r,n_0)}|_{\tilde Z}+\sum_{a,b=1}^r\sum_{i=1}^{M_0^{(a)}}\sum_{j=1}^{N_0^{(b)}}R_bR_a^{-1}\left(T_1^{i-\mu_{1,j}^{(b)}}-T_1^{i}\right)\left(T_2^{-j+\mu_{1,i}^{(a)'}+1}-T_2^{-j+\mu_{0,i}^{(a)'}+1}\right)\\ &\quad -\sum_{i=1}^{M_0^{(a)}}\sum_{j=1}^{\mu_{0,i}^{(a)'}-\mu_{1,i}^{(a)'}}T_1^iT_2^{j+\mu_{1,i}^{(a)'}}\\ &\quad +\sum_{k=2}^{N}\left[\sum_{a,b=1}^r\sum_{i=1}^{M_0^{(a)}}\sum_{j=1}^{N_0^{(b)}}R_bR_a^{-1}\left(T_1^{i-\mu_{k,j}^{(b)}}-T_1^{i-\mu_{k-1,j}^{(b)}}\right)\left(T_2^{-j+\mu_{k,i}^{(a)'}+1}-T_2^{-j+\mu_{0,i}^{(a)'}+1}\right)\right], \end{aligned} \end{align*}$$

where the fixed point Z is to be identified with a choice of a sequence of coloured nested partitions $\boldsymbol \mu _1\subseteq \boldsymbol \mu _{2}\subseteq \cdots \subseteq \boldsymbol \mu _{N}\subseteq \boldsymbol \mu _0$ as in Proposition 3.4, $\tilde Z\in \mathcal M(r,n_0)^{{\mathbb T}}$ is the ${\mathbb T}$ -fixed point corresponding to the coloured partition $\boldsymbol \mu _0$ .

Lemma 3.5. The virtual tangent space $T^{\mathrm {vir}}_{{\mathcal N}(r,{\mathbf n})}|_Z$ at the fixed point $Z\in {\mathcal N}(r,{\mathbf n})^{{\mathbb T}}$ contains no constant terms (i.e., it is entirely ${\mathbb T}$ -movable).

Proof. Assuming the generators $R_a$ , $a=1,\dots ,r$ of the the representation ring $R({\mathbb T}_2)$ to be sufficiently generic, we only need to show that $T^{\mathrm {vir}}_{{\mathcal N}(1,{\mathbf n})}|_Z$ is ${\mathbb T}$ -movable. It is moreover sufficient to prove the claim in the case of flags of length $2$ , and the proof immediately generalizes. If $n_0=n_1$ , there is nothing to show, so let $n_0>n_1$ and $Z\in {\mathcal N}(1,n_0,n_1)^{{\mathbb T}}$ a fixed point associated with the nested partitions $\mu _0\supset \mu _1$ . The K-theory class of the virtual tangent space at Z is

$$ \begin{align*} T^{\mathrm{vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z&=V_0\otimes V_0^{\ast}\otimes\left(Q-1-\Lambda^2Q\right)+V_0+V_0^{\ast}\otimes\Lambda^2Q\\ &\quad+\left(V_1\otimes V_1^{\ast}-V_0\otimes V_1^{\ast}\right)\otimes\left(Q-1-\Lambda^2Q\right)-V_1^{\ast}\otimes\Lambda^2Q\\ &=V_0\otimes V_0^{\ast}\otimes\left(Q-1-\Lambda^2Q\right)+V_0+V_0^{\ast}\otimes\Lambda^2Q\\ &\quad-V\otimes V_1^{\ast}\otimes\left(Q-1-\Lambda^2Q\right)-V_1^{\ast}\otimes\Lambda^2Q, \end{align*} $$

where

$$\begin{align*}V_0=\sum_{(i,j)\in Y_{\mu_0}}T_1^{-i+1}T_2^{-j+1},\quad V_1=\sum_{(i,j)\in Y_{\mu_0\setminus\mu_1}}T_1^{-i+1}T_2^{-j+1},\quad V=\sum_{(i,j)\in Y_{\mu_1}}T_1^{-i+1}T_2^{-j+1}, \end{align*}$$

and $Y_{\mu _i}$ denotes the Young diagram associated with $\mu _i$ . By construction, if $\mu _0\supset \mu _1$ , one has $i'>i$ or $j'>j$ , or both, for all $(i,j)\in Y_{\mu _1}$ , $(i',j')\in Y_{\mu _0\setminus \mu _1}$ . We also have that

$$\begin{align*}T_{\mathcal M(1,n_0)}|_{Z_0}=V_0\otimes V_0^{\ast}\otimes\left(Q-1-\Lambda^2Q\right)+V_0+V_0^{\ast}\otimes\Lambda^2Q \end{align*}$$

contains no constant term, being the tangent space to $\mathcal M(1,n_0)$ at the fixed point $Z_0$ associated with the partition $\mu _0$ . Similarly, $V_1^{\ast }\otimes \Lambda ^2Q$ is manifestly ${\mathbb T}$ -movable. Consider then the remaining term $V\otimes V_1^{\ast }\otimes \left(Q-1-\Lambda ^2Q\right)$ in $T^{\mathrm {vir}}_{{\mathcal N}(1,n_0,n_1)}|_Z$ . The contribution corresponding to $V\otimes V_1^{\ast }$ consists of a sum of monomials of the form $T_1^{-i+i'}T_2^{-j+j'}$ , where $(i,j)\in Y_{\mu _1}$ and $(i',j')\in Y_{\mu _0\setminus \mu _1}$ . The only possibility for a constant term to arise is if $i=i'$ , $j=j'$ , for some $(i,j)\in Y_{\mu _1}$ and $(i',j')\in Y_{\mu _0\setminus \mu _1}$ . This is, however, not possible if $\mu _0\supset \mu _1$ . In a completely analogous way, one can show that no constant term can arise from either $V\otimes V_1^{\ast }\otimes Q$ or $V\otimes V_1^{\ast }\otimes \Lambda ^2Q$ .

3.2 Virtual equivariant holomorphic Euler characteristic

The first virtual invariant we are going to study is the holomorphic virtual equivariant Euler characteristic of the moduli space of nested instantons. The fact that we can decompose the virtual tangent bundle as a direct sum of equivariant line bundles under the torus action we previously described greatly simplifies the computations.

In particular, given a scheme X with a $1$ -perfect obstruction theory $\mathbb E$ , one can define a virtual structure sheaf $\mathcal O_X^{\mathrm {vir}}$ . Moreover, one can choose an explicit resolution of $\mathbb E$ as $[E^{-1}\to E^0]$ a complex of vector bundles. If $[E_0\to E_1]$ denotes the dual complex, then one can also define the virtual tangent bundle $T_X^{\mathrm {vir}}\in K^0(X)$ as the class $T_X^{\mathrm {vir}}=[E_0]-[E_1]$ . With these definitions, the virtual Todd genus of X is defined as $\operatorname {\mathrm {td}}^{\mathrm { vir}}(X)=\operatorname {\mathrm {td}}(T_X^{\mathrm {vir}})$ , and if X is proper, given any $V\in K^0(X)$ , one defines the virtual holomorphic Euler characteristic as

$$\begin{align*}\chi^{\mathrm{vir}}(X,V)=\chi(X,V\otimes O^{\mathrm{vir}}_X), \end{align*}$$

and as a consequence of the virtual Riemann-Roch theorem [17], if X is proper and $V\in K^0(X)$ , the virtual holomorphic Euler characteristic admits an equivalent definition as

(3.2.1) $$ \begin{align} \chi^{\mathrm{vir}}(X,V)=\int_{[X]^{\mathrm{vir}}}\operatorname{\mathrm{ch}}(V)\cdot\operatorname{\mathrm{td}}(T_X^{\mathrm{vir}}), \end{align} $$

where $[X]^{\mathrm {vir}}$ is the virtual fundamental class of X, $[X]^{\mathrm {vir}}\in A_{\operatorname {\mathrm {vd}}}(X)$ and $\operatorname {\mathrm {vd}}$ denotes the virtual dimension of X, $\operatorname {\mathrm {vd}}=\operatorname {\mathrm {rk}} E_0-\operatorname {\mathrm {rk}} E_1$ . Clearly, if we are interested in $\chi ^{\mathrm { vir}}(X)$ , then the previous formula reduces to

(3.2.2) $$ \begin{align} \chi^{\mathrm{vir}}(X)=\int_{[X]^{\mathrm{vir}}}\operatorname{\mathrm{td}}(T_X^{\mathrm{vir}}) \end{align} $$

whenever X is proper.

Equations (3.2.1) and (3.2.2) can be made even more explicit. In fact, if we take $n=\operatorname {\mathrm {rk}} E_0$ , $m=\operatorname {\mathrm {rk}} E_1$ , so that $\operatorname {\mathrm {vd}}=n-m$ , and define $x_1,\dots ,x_n$ and $u_1,\dots ,u_m$ to be respectively the Chern roots of $E_0$ and $E_1$ , then (3.2.2) becomes

$$\begin{align*}\chi^{\mathrm{vir}}(X)=\int_{[X]^{\mathrm{vir}}}\prod_{i=1}^n\frac{x_i}{1-{\text e}^{-x_1}}\prod_{j=1}^m\frac{1-{\text e}^{-u_j}}{u_j}, \end{align*}$$

while for (3.2.1), we have

$$\begin{align*}\chi^{\mathrm{vir}}(X,V)=\int_{[X]^{\mathrm{vir}}}\left(\sum_{k=1}^r{\text e}^{v_k}\right)\prod_{i=1}^n\frac{x_i}{1-{\text e}^{-x_1}}\prod_{j=1}^m\frac{1-{\text e}^{-u_j}}{u_j} \end{align*}$$

since we can consider $V\in K^0(X)$ to be a vector bundle on X with Chern roots $v_1,\dots ,v_r$ .

Now, if we have a proper scheme X equipped with an action of a torus $({\mathbb C}^{*})^N$ and an equivariant $1$ -perfect obstruction theory, we can apply virtual equivariant localization in order to compute virtual invariants of X. We will now briefly recall how virtual localization works. First of all, for any equivariant vector bundle B over a proper scheme Z with a $1$ -perfect obstruction theory, which is moreover equipped with a trivial action of $({\mathbb C}^{*})^N$ , we have the decomposition

$$\begin{align*}B=\bigoplus_{{\mathbf k}\in\mathbb Z^N}B^{\mathbf k}, \end{align*}$$

where $B^{\mathbf k}$ denotes the $({\mathbb C}^{*})^N$ -eigenbundles on which the torus acts by $t_1^{k_1}\cdots t_N^{k_N}$ . If we now give a set of variables $\varepsilon _1,\dots ,\varepsilon _N$ , we identify B with $B=\sum _{{\mathbf k}}B^{\mathbf k}{\text e}^{k_1\varepsilon _1}\cdots {\text e}^{k_N\varepsilon _N}\in K^0(Z)[\![\varepsilon _1,\dots ,\varepsilon _N]\!]$ . One then defines $B^{\mathrm {fix}}=B^{{\mathbf 0}}$ and $B^{\mathrm { mov}}=\oplus _{{\mathbf k}\neq {\mathbf 0}}B^{\mathbf k}$ . Then the Chern character $\operatorname {\mathrm {ch}}:K^0(Z)\to A^{*}(Z)$ can be extended by $\mathbb Q(\!(\varepsilon _1,\dots ,\varepsilon _N)\!)$ -linearity to

$$ \begin{align*}\operatorname{\mathrm{ch}}:K^0(Z)(\!(\varepsilon_1,\dots,\varepsilon_N)\!)\to A^{*}(Z)(\!(\varepsilon_1,\dots,\varepsilon_N)\!).\end{align*} $$

Since the Grothendieck group of equivariant vector bundles $K^0_{({\mathbb C}^{*})^N}(Z)$ is a subring of $K^0(Z)[\![\varepsilon _1,\dots ,\varepsilon _N]\!]$ , the restriction of the extension of $\operatorname {\mathrm {ch}}$ to $K^0_{({\mathbb C}^{*})^N}(Z)$ is naturally identified with the equivariant Chern character. Finally, if one denotes by $p_*^{\mathrm {vir}}$ the $\mathbb Q(\!(\varepsilon _1,\dots ,\varepsilon _N)\!)$ -linear extension of $\chi ^{\mathrm {vir}}(Z,-):K^0(Z)\to \mathbb Z$ , and $p_*$ is the equivariant pushforward to a point, one can prove as in [17] that

$$\begin{align*}p_*^{\mathrm{vir}}(V)=p_*\left(\operatorname{\mathrm{ch}}(V)\operatorname{\mathrm{td}}(T_Z^{\mathrm{vir}})\cap[Z]^{\mathrm{vir}}\right),\qquad V\in K^0(Z)(\!(\varepsilon_1,\dots,\varepsilon_N)\!). \end{align*}$$

Then, following [23], if we have a global equivariant embedding of a scheme X into a nonsingular scheme Y with $({\mathbb C}^{*})^N$ action, we can identify the maximal $({\mathbb C}^{*})^N$ -fixed closed subscheme $X^f$ of X with the scheme-theoretic intersection $X^f=X\cap Y^f$ , where $Y^f$ is the nonsingular set-theoretic fixed point locus. By decomposing $Y^f$ into irreducible components $Y^f=\bigcup _iY_i$ , one can also define $X_i=X\cap Y_i$ , which carry a perfect obstruction theory with virtual fundamental class $[X_i]^{\mathrm {vir}}$ . In this way, if $\tilde V\in K^0_{({\mathbb C}^{*})^N}(X)$ is an equivariant lift of the vector bundle V, $\tilde V_i$ is its restriction to $X_i$ and $p_i:X_i\to \mathrm { pt}$ is the projection, one has that

(3.2.3) $$ \begin{align} \chi^{\mathrm{vir}}(X,\tilde V;\varepsilon_1,\dots,\varepsilon_N)=\sum_i p_{i*}^{\mathrm{vir}}\left(\tilde V_i/\Lambda_{-1}(N_i^{\mathrm{vir}})^\vee\right)=\sum_i p_{i*}^{\mathrm{vir}}\left(\tilde V_i/\Lambda_{-1}(T_X^{\mathrm{vir}}|_{X_i}^{\mathrm{mov}})^\vee\right) \end{align} $$

belongs to $\mathbb Q[\![\varepsilon _1,\dots ,\varepsilon _N]\!]$ and the virtual holomorphic Euler characteristic is $\chi ^{\mathrm {vir}}(X,V)=\chi ^{\mathrm {vir}}(X,\tilde V;\boldsymbol 0)$ .

As pointed out at the beginning of §3, we will define invariants in localization, as the ${\mathbb T}$ -fixed locus of ${\mathcal N}(r,{\mathbf n})$ is proper, while ${\mathcal N}(r,{\mathbf n})$ is only quasi-projective, so the pushforward in the right-hand side of Equation 3.2.3 is well defined. Computations are now made very easy by the fact that we represented the virtual tangent space to the ${\mathbb T}$ -fixed points to the moduli space of nested instantons in the representation ring $R({\mathbb T})$ . In this way, $T_{X_i}^{\mathrm {vir}}$ is decomposed as a direct sum of line bundles which are moreover eigenbundles of the torus action. Then we can use the properties

$$\begin{align*}\operatorname{\mathrm{ch}}(E\oplus F)=\operatorname{\mathrm{ch}} E+\operatorname{\mathrm{ch}} F,\quad \Lambda_{t}(E\oplus F)=\Lambda_{t}(E)\cdot\Lambda_{t}(F),\quad S_t(E\oplus F)=S_t(E)\cdot S_t(F) \end{align*}$$

and Equation (3.2.3) to compute the equivariant holomorphic Euler characteristic of the moduli space of nested instantons in terms of the fundamental characters $\mathfrak q_{1,2}$ of the torus ${\mathbb T}_1$ . These will be related to the equivariant parameters by $\mathfrak q_i={\text e}^{\beta \varepsilon _i}$ , with $\beta $ being a parameter having a very clear meaning in the physical framework modelling the moduli space of nested instantons as a low energy effective theory. In this framework, it is very easy to explicitly compute the virtual equivariant holomorphic Euler characteristic of the moduli space of nested instantons, as we already described the ${\mathbb T}$ -fixed locus of ${\mathcal N}(r,n_0,\dots ,n_N)$ as being zero-dimensional and reduced.⁶ As we saw in §3.1, the fixed points of ${\mathcal N}(r,n_0,\dots ,n_N)$ are completely described by r-tuples of nested coloured partitions $\boldsymbol \mu _1\subseteq \cdots \subseteq \boldsymbol \mu _N\subseteq \boldsymbol \mu _0$ , with $\boldsymbol \mu _j\in \mathcal P^r$ , in such a way that $|\boldsymbol \mu _0|=\sum _j|\boldsymbol \mu _0^j|=n_0$ and $|\boldsymbol \mu _0\setminus \boldsymbol \mu _{i>0}|=n_{i>0}$ . In the simplest case of $r=1$ , we get

(3.2.4) $$ \begin{align} \begin{aligned} \chi^{\mathrm{vir}}({\mathcal N}(1,{\mathbf n}),\tilde V;\mathfrak q_1,\mathfrak q_2)=&\sum_{\substack{\mu_1\subseteq\cdots\subseteq\mu_0\\ |\mu_0\setminus\mu_j|=n_j}}\frac{\displaystyle{T_{\mu_0,\mu_1}(\mathfrak q_1,\mathfrak q_2)W_{\mu_0,\dots,\mu_N}(\mathfrak q_1,\mathfrak q_2)}}{\displaystyle{N_{\mu_0}(\mathfrak q_1,\mathfrak q_2)}}\left.\left[\tilde V\right]\right|_{\mu_0,\dots,\mu_N}, \end{aligned} \end{align} $$

where we defined

(3.2.5) $$ \begin{align} N_{\mu_0}(\mathfrak q_1,\mathfrak q_2)&=\prod_{s\in Y_{\mu_0}}\left(1-\mathfrak q_1^{-l(s)-1}\mathfrak q_2^{a(s)}\right)\left(1-\mathfrak q_1^{l(s)}\mathfrak q_2^{-a(s)-1}\right), \end{align} $$

(3.2.6) $$ \begin{align} T_{\mu_0,\mu_1}(\mathfrak q_1,\mathfrak q_2)&=\prod_{i=1}^{M_0}\prod_{j=1}^{\mu_{0,i}^{\prime}-\mu_{1,i}^{\prime}}\left(1-\mathfrak q_1^{-i}\mathfrak q_2^{-j-\mu_{1,i}^{\prime}}\right), \end{align} $$

(3.2.7) $$ \begin{align} W_{\mu_0,\dots,\mu_N}(\mathfrak q_1,\mathfrak q_2)&=\prod_{k=1}^{N}\prod_{i=1}^{M_0}\prod_{j=1}^{N_0}\frac{\displaystyle{\left(1-\mathfrak q_1^{\mu_{k,j}-i}\mathfrak q_2^{j-\mu_{0,i}^{\prime}-1}\right)\left(1-\mathfrak q_1^{\mu_{k-1,j}-i}\mathfrak q_2^{j-\mu_{k,i}^{\prime}-1}\right)}}{\displaystyle{\left(1-\mathfrak q_1^{\mu_{k,j}-i}\mathfrak q_2^{j-\mu_{k,i}^{\prime}-1}\right)\left(1-\mathfrak q_1^{\mu_{k-1,j}-i}\mathfrak q_2^{j-\mu_{0,i}^{\prime}-1}\right)}}, \end{align} $$

with $a(s)$ and $l(s)$ the arm length and the leg length of the box s in the Young diagram $Y_{\mu }$ associated to $\mu $ , respectively. A very interesting and surprising fact can be observed if we rearrange the expression of the holomorphic virtual Euler characteristic of ${\mathcal N}(1,{\mathbf n})$ . In fact, if we perform the summation over the smaller partitions $\mu _1\subseteq \cdots \subseteq \mu _N$ and redefine $q=\mathfrak q_1^{-1}$ , $t=\mathfrak q_2^{-1}$ , we get

(3.2.8) $$ \begin{align} \chi^{\mathrm{vir}}({\mathcal N}(1,{\mathbf n});\mathfrak q_1,\mathfrak q_2)=\sum_{\mu_0}\frac{P_{\mu_0}(q,t)}{N_{\mu_0}(q,t)}, \end{align} $$

and the unexpected fact is that we think $P_{\mu _0}(q,t)$ to be a polynomial in $q,t$ except for a factor $(1-qt)^{-N}$ .

Conjecture 3.6. $P_{\mu _0}(q,t)$ in Equation (3.2.8) is a function of the form

(3.2.9) $$ \begin{align} P_{\mu_0}(q,t)=\frac{Q_{\mu_0}(q,t)}{(1-qt)^N}, \end{align} $$

with $Q_{\mu _0}(q,t)\in \mathbb Z[q,t]$ a polynomial in the $(q,t)$ -variables.

Remark 3.7. The rational function $P_{\mu _0}(q,t)$ and the polynomials $Q_{\mu _0}(q,t)$ in Equation (3.2.8) and Equation (3.2.9) also depend on the discrete nesting profile ${\mathbf n}$ . The dependence on ${\mathbf n}$ is suppressed in the notation to avoid cluttering.

Sometimes the polynomials in (3.2.9) can be given an interpretation in terms of some known symmetric polynomials. Consider the ring $\Lambda (\mathbf x)$ of symmetric functions in the infinite set of variables $\{x_1,x_2,\dots \}$ . It is convenient to sometimes denote by the same symbol X both the formal sum $X=x_1+x_2+\cdots $ and the alphabet $\mathbf x=\{x_1,x_2,\dots \}$ . If $\lambda $ is an integer partition, we can define the monomial functions

$$\begin{align*}m_\lambda=\sum_{\alpha}x_1^{\alpha_1}x_2^{\alpha_2}\cdots, \end{align*}$$

where the sum runs over all the permutations of $\lambda $ . The ring of symmetric functions $\Lambda (\mathbf x)$ will then be the free $\mathbb Z$ -module generated by the monomial functions $m_\lambda $ , for all partitions $\lambda $ . Two other sets of symmetric functions in $\Lambda (\mathbf x)$ are the complete homogeneous symmetric functions $h_\lambda $ and the power functions $p_\lambda $ , defined as

$$ \begin{align*} &h_\lambda=h_{\lambda_1}h_{\lambda_2}\cdots,\qquad h_k=\sum_{\lambda\vdash k}m_\lambda\\ &p_\lambda=p_{\lambda_1}p_{\lambda_2}\cdots,\qquad p_k=\sum_{j>0}x_j^k. \end{align*} $$

The symmetric functions $h_\lambda $ and $p_\lambda $ form other $\mathbb Q$ -bases of $\Lambda (\mathbf x)$ indexed by integer partitions, and we can moreover introduce a symmetric positive-definite bilinear form, the Hall pairing $\langle -,-\rangle $ , such that the two $\mathbb Q$ -bases $m_\lambda $ and $h_\lambda $ are dual to each other (i.e., $\langle h_\lambda ,m_\mu \rangle =\delta _{\lambda \mu }$ , for any choice of partitions $\lambda ,\mu $ ). Consider then the function

$$\begin{align*}\Omega[X]=\exp\left(\sum_{k\ge 1}\frac{p_k[X]}{k}\right), \end{align*}$$

where the plethystic notation $P[X]=P(x_1,x_2,\dots )$ is used, for any $P\in \Lambda (\mathbf x)$ and $X=x_1+x_2+\cdots $ , as before. On the ring $\Lambda (\mathbf x)\otimes _{\mathbb Z}\mathbb Q(q,t)$ , we can introduce the operator

$$\begin{align*}\Delta f=\operatorname{\mathrm{Coeff}}_{z^0}\left(f[X+(1-q)(1-t)/z]\Omega[-zX]\right), \qquad f\in\Lambda(\mathbf x)\otimes_{\mathbb Z}\mathbb Q(q,t), \end{align*}$$

and a set of eigenfunctions for $\Delta $ is given by the modified (or transformed) Macdonald polynomials $\widetilde H_\lambda (\mathbf x;q,t)$ . They form a basis for $\Lambda (\mathbf x)\otimes _{\mathbb Z}\mathbb Q(q,t)$ and are defined as

(3.2.10) $$ \begin{align} \widetilde H_\lambda[X;q,t]=\sum_{\mu\vdash|\lambda|}\widetilde K_{\mu\lambda}(q,t)s_\mu[X], \end{align} $$

where $s_\lambda $ are the Schur symmetric functions and $\tilde K_{\mu ,\lambda }(q,t)\in \mathbb N[q,t]$ are the modified $q,t$ -Kostka functions, introduced by Macdonald in [36].

Let us then define the generating function

$$\begin{align*}Z_{MD}(q,t;x_0,\dots,x_N)=\sum_{n_0\ge\dots\ge n_N}\chi^{\mathrm{vir}}({\mathcal N}(1,\tilde{{\mathbf n}});q,t)\prod_{i=0}^N x_i^{m_i}, \end{align*}$$

where $m_i=n_i-n_{i+1}$ and the integers $\tilde n_i$ form a sequence obtained by permuting $n_i$ in such a way that the sequence defined $\tilde n_i-\tilde n_{i+1}$ is ordered. By construction, $Z_{MD}(q,t;x_0,\dots ,x_N)\in \mathbb Q(q,t)\otimes _{\mathbb Z}\Lambda (\mathbf x)$ . As a consequence of conjecture 3.6, we have

$$\begin{align*}\begin{aligned} Z_{MD}(q,t;x_0,\dots,x_N)&=\sum_{n_0\ge\cdots\ge n_N}\sum_{\mu_0\in\mathcal P(n_0)}\frac{Q_{\mu_0}^{(\tilde{{\mathbf n}})}(q,t)}{(1-qt)^NN_{\mu_0}(q,t)}\prod_{i=0}^Nx_i^{m_i}, \end{aligned} \end{align*}$$

where, as in Remark 3.7, we emphasized the dependence of $Q_{\mu _0}(q,t)$ on the discrete profile $\tilde {{\mathbf n}}$ .

Conjecture 3.8. When $|\mu _0|=|\mu _N|+1=|\mu _{N-1}|+2=\cdots =|\mu _1|+N$ , we have

(3.2.11) $$ \begin{align} Q_{\mu_0}(q,t)=\left\langle h_{\mu_0}(\mathbf x),\widetilde{H}_{\mu_0}(\mathbf x;q,t)\right\rangle. \end{align} $$

The Schur functions $s_\mu $ can be expressed in terms of the monomial functions as

$$\begin{align*}s_\mu[X]=\sum_{\nu\vdash|\mu|}K_{\mu\nu}m_\nu[X], \end{align*}$$

where the Kostka coefficients $K_{\mu \nu }$ count the number of semi-standard Young tableaux of shape $\mu $ and weight $\nu $ , so that $K_{\mu \mu }=1$ . Thus, using (3.2.10), we can rewrite (3.2.11) as

$$ \begin{align*} Q_{\mu_0}(q,t)&=\left\langle h_{\mu_0}(\mathbf x),\sum_{\lambda,\nu\in\mathcal P(n_0)}\widetilde{K}_{\lambda,\mu_0}(q,t)K_{\mu_0,\nu}m_\nu(\mathbf x)\right\rangle\\ &=\sum_{\substack{\lambda\in\mathcal P(n_0)\\ m_\lambda(\mathbf x)\neq 0}}\widetilde{K}_{\lambda,\mu_0}(q,t). \end{align*} $$

In particular, within the assumptions of Conjecture 3.8, we have $Q_{\mu _0}(q,t)\in \mathbb N[q,t]$ . We checked the previous conjectures up to $n_0=10$ .

If instead $r>1$ , we get a more complicated result, even though its structure is the same as we had previously:

(3.2.12) $$ \begin{align} \begin{aligned} \chi^{\mathrm{vir}}({\mathcal N}(r,{\mathbf n}),\tilde V;\mathfrak q_1,\mathfrak q_2,\{\mathfrak t_i\})=&\sum_{\substack{\boldsymbol\mu_1\subseteq\cdots\subseteq\boldsymbol\mu_0\\ |\boldsymbol\mu_0\setminus\boldsymbol\mu_j|=n_j}}\frac{\displaystyle{T_{\boldsymbol\mu_0,\boldsymbol\mu_1}^{(r)}(\mathfrak q_1,\mathfrak q_2)W_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}^{(r)}(\mathfrak q_1,\mathfrak q_2)}}{\displaystyle{N_{\boldsymbol\mu_0}^{(r)}(\mathfrak q_1,\mathfrak q_2)}}\left.\left[\tilde V\right]\right|_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}, \end{aligned} \end{align} $$

with

$$ \begin{align*} N_{\boldsymbol\mu_0}^{(r)}(\mathfrak q_1,\mathfrak q_2)&=\prod_{a,b=1}^r\prod_{s\in Y_{\mu_0^{(a)}}}\left(1-\mathfrak t_{ab}\mathfrak q_1^{-l_a(s)-1}\mathfrak q_2^{a_b(s)}\right)\left(1-\mathfrak q_1^{l_a(s)}\mathfrak q_2^{-a_b(s)-1}\right),\\ T_{\boldsymbol\mu_0,\boldsymbol\mu_1}^{(r)}(\mathfrak q_1,\mathfrak q_2)&=\prod_{a,b}^r\prod_{i=1}^{M_0^{(a)}}\prod_{j=1}^{\mu_{0,i}^{(a)\prime}-\mu_{1,i}^{(a)\prime}}\left(1-\mathfrak t_{ab}\mathfrak q_1^{-i}\mathfrak q_2^{-j-\mu_{1,i}^{(a)\prime}}\right),\\ W_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}^{(r)}(\mathfrak q_1,\mathfrak q_2)&=\prod_{k=1}^{N}\prod_{a,b}^r\prod_{i=1}^{M_0^{(a)}}\prod_{j=1}^{N_0^{(b)}}\frac{\displaystyle{\left(1-\mathfrak t_{ab}\mathfrak q_1^{\mu_{k,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{0,i}^{(a)\prime}-1}\right)\left(1-\mathfrak t_{ab}\mathfrak q_1^{\mu_{k-1,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{k,i}^{(a)\prime}-1}\right)}}{\displaystyle{\left(1-\mathfrak t_{ab}\mathfrak q_1^{\mu_{k,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{k,i}^{(a)\prime}-1}\right)\left(1-\mathfrak t_{ab}\mathfrak q_1^{\mu_{k-1,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{0,i}^{(a)\prime}-1}\right)}}, \end{align*} $$

where now $\mathfrak t_{ab}=\mathfrak t_a\mathfrak t_b^{-1}$ , and $\{\mathfrak t_i\}$ are the fundamental characters of ${\mathbb T}_2$ in ${\mathbb T}$ , and $a_b(s)$ denotes the arm length of the box s with respect to the Young diagram $Y_{\mu ^{(b)}}$ associated to the partition $\mu ^{(b)}$ of $\boldsymbol \mu $ (with an analogous definition for the leg length).

3.3 Virtual equivariant $\chi _{-y}$ -genus

The first refinement of the equivariant holomorphic Euler characteristic we are going to study is the virtual equivariant $\chi _{-y}$ -genus, as defined in [17]. In order to exhibit the definition of virtual $\chi _{-y}$ -genus, let us first recall that if E is a rank r vector bundle on r, one can define the antisymmetric product $\Lambda _tE$ and the symmetric one $S_tE$ as

$$ \begin{align*}\Lambda_tE=\sum_{i=0}^r[\Lambda^iE]t^i\in K^0(X)[t],\qquad S_tE=\sum_{i\ge 0}[S^iE]t^i\in K^0(X)[\![t]\!], \end{align*} $$

so that $1/\Lambda _tE=S_{-t}E$ in $K^0(X)[\![t]\!]$ . We can then define the virtual cotangent bundle $\Omega _X^{\mathrm {vir}}=(T_X^{\mathrm {vir}})^\vee $ and the bundle of virtual n-forms $\Omega _X^{n,\mathrm { vir}}=\Lambda ^n\Omega _X^{\mathrm {vir}}$ . If then X is a proper scheme equipped with a perfect obstruction theory of virtual dimension d, the virtual $\chi _{-y}$ -genus of X is defined by

(3.3.1) $$ \begin{align} \chi_{-y}^{\mathrm{vir}}(X)=\chi^{\mathrm{vir}}(X,\Lambda_{-y}\Omega_X^{\mathrm{vir}})=\sum_{i\ge 0}(-y)^i\chi^{\mathrm{ vir}}(X,\Omega^{i,\mathrm{vir}}_X), \end{align} $$

while if $V\in K^0(X)$ , the virtual $\chi _{-y}$ -genus of X with values in V is

$$\begin{align*}\chi_{-y}^{\mathrm{vir}}(X,V)=\chi^{\mathrm{vir}}(X,V\otimes\Lambda_{-y}\Omega_X^{\mathrm{vir}})=\sum_{i\ge 0}(-y)^i\chi^{\mathrm{vir}}(X,V\otimes\Omega^{i,\mathrm{vir}}_X). \end{align*}$$

Though in principle one would expect $\chi _{-y}^{\mathrm {vir}}(X,V)$ to be an element of $\mathbb Z[\![y]\!]$ , it is, in fact, true that $\chi _{-y}^{\mathrm {vir}}(X,V)\in \mathbb Z[y]$ , [17].

By the form (3.2.2) and (3.2.1) of the holomorphic Euler characteristic, it is easy to see that

$$ \begin{align*} \chi_{-y}^{\mathrm{vir}}(X) &=\int_{[X]^{\mathrm{vir}}}\operatorname{\mathrm{ch}}(\Lambda_{-y}T_X^{\mathrm{vir}})\cdot\operatorname{\mathrm{td}}(T_X^{\mathrm{ vir}})=\int_{[X]^{\mathrm{vir}}}\mathcal X_{-y}(X),\\ \chi_{-y}^{\mathrm{vir}}(X,V) &=\int_{[X]^{\mathrm{vir}}}\operatorname{\mathrm{ch}}(\Lambda_{-y}T_X^{\mathrm{vir}})\cdot\operatorname{\mathrm{ch}}(V)\cdot\operatorname{\mathrm{td}}(T_X^{\mathrm{vir}})=\int_{[X]^{\mathrm{vir}}}\mathcal X_{-y}(X)\cdot\operatorname{\mathrm{ch}}(V), \end{align*} $$

which, in terms of the Chern roots of $E_0$ , $E_1$ and V become

$$ \begin{align*} \chi_{-y}^{\mathrm{vir}}(X) &=\int_{[X]^{\mathrm{vir}}}\prod_{i=1}^nx_i\frac{1-y{\text e}^{-x_i}}{1-{\text e}^{-x_i}}\prod_{j=1}^m\frac{1}{u_j}\frac{1-{\text e}^{-u_j}}{1-y{\text e}^{-u_j}},\\ \chi_{-y}^{\mathrm{vir}}(X,V) &=\int_{[X]^{\mathrm{vir}}}\left(\sum_{k=1}^r{\text e}^{v_k}\right)\prod_{i=1}^nx_i\frac{1-y{\text e}^{-x_i}}{1-{\text e}^{-x_i}}\prod_{j=1}^m\frac{1}{u_j}\frac{1-{\text e}^{-u_j}}{1-y{\text e}^{-u_j}}. \end{align*} $$

Finally, one can define the virtual Euler number $e^{\mathrm {vir}}(X)$ and the virtual signature $\sigma ^{\mathrm { vir}}(X)$ of X as $e^{\mathrm {vir}}(X)=\chi _{-1}^{\mathrm {vir}}(X)$ and $\sigma ^{\mathrm {vir}}(X)=\chi _1^{\mathrm {vir}}(X)$ . Whenever $y=0$ , one recovers the holomorphic virtual Euler characteristic instead.

By extending the definition of $\chi _{-y}$ -genus to the equivariant case in the obvious way and by equivariant virtual localization, one gets

(3.3.2) $$ \begin{align} \chi_{-y}^{\mathrm{vir}}(X,\tilde V;\varepsilon_1,\dots,\varepsilon_N)=\sum_ip_{i*}^{\mathrm{vir}}\left(\tilde V_i\otimes\Lambda_{-y}(\Omega_X^{\mathrm{vir}}|_{X_i})/\Lambda_{-1}(N_i^{\mathrm{vir}})^\vee\right), \end{align} $$

whence $\chi _{-y}^{\mathrm {vir}}(X,V)=\chi _{-y}^{\mathrm {vir}}(X,\tilde V;0,\dots ,0)$ .

A simple computation in equivariant localization gives us the following result:

(3.3.3) $$ \begin{align} \begin{aligned} \chi^{\mathrm{vir}}_{-y}({\mathcal N}(1,{\mathbf n}),\tilde V;\mathfrak q_1,\mathfrak q_2)=&\sum_{\substack{\mu_1\subseteq\cdots\subseteq\mu_0\\ |\mu_0\setminus\mu_j|=n_j}}\frac{\displaystyle{T_{\mu_0,\mu_1}^{-y}(\mathfrak q_1,\mathfrak q_2)W_{\mu_0,\dots,\mu_N}^{-y}(\mathfrak q_1,\mathfrak q_2)}}{\displaystyle{N_{\mu_0}^{-y}(\mathfrak q_1,\mathfrak q_2)}}\left.\left[\tilde V\right]\right|_{\mu_0,\dots,\mu_N}, \end{aligned} \end{align} $$

with

(3.3.4) $$ \begin{align} N_{\mu_0}^{-y}(\mathfrak q_1,\mathfrak q_2)&=\prod_{s\in Y_{\mu_0}}\frac{\displaystyle{\left(1-\mathfrak q_1^{-l(s)-1}\mathfrak q_2^{a(s)}\right)\left(1-\mathfrak q_1^{l(s)}\mathfrak q_2^{-a(s)-1}\right)}}{\displaystyle{\left(1-y\mathfrak q_1^{-l(s)-1}\mathfrak q_2^{a(s)}\right)\left(1-y\mathfrak q_1^{l(s)}\mathfrak q_2^{-a(s)-1}\right)}}, \end{align} $$

(3.3.5) $$ \begin{align} T_{\mu_0,\mu_1}^{-y}(\mathfrak q_1,\mathfrak q_2)&=\prod_{i=1}^{M_0}\prod_{j=1}^{\mu_{0,i}^{\prime}-\mu_{1,i}^{\prime}}\frac{\displaystyle{\left(1-\mathfrak q_1^{-i}\mathfrak q_2^{-j-\mu_{1,i}^{\prime}}\right)}}{\displaystyle{\left(1-y\mathfrak q_1^{-i}\mathfrak q_2^{-j-\mu_{1,i}^{\prime}}\right)}}, \end{align} $$

(3.3.6) $$ \begin{align} W_{\mu_0,\dots,\mu_N}^{-y}(\mathfrak q_1,\mathfrak q_2)&=\prod_{k=1}^{N}\prod_{i=1}^{M_0}\prod_{j=1}^{N_0}\frac{\displaystyle{\left(1-\mathfrak q_1^{\mu_{k,j}-i}\mathfrak q_2^{j-\mu_{0,i}^{\prime}-1}\right)\left(1-\mathfrak q_1^{\mu_{k-1,j}-i}\mathfrak q_2^{j-\mu_{k,i}^{\prime}-1}\right)}}{\displaystyle{\left(1-y\mathfrak q_1^{\mu_{k,j}-i}\mathfrak q_2^{j-\mu_{0,i}^{\prime}-1}\right)\left(1-y\mathfrak q_1^{\mu_{k-1,j}-i}\mathfrak q_2^{j-\mu_{k,i}^{\prime}-1}\right)}}\cdot \nonumber\\ &\quad \cdot\frac{\displaystyle{\left(1-y\mathfrak q_1^{\mu_{k,j}-i}\mathfrak q_2^{k-\mu_{k,i}^{\prime}-1}\right)\left(1-y\mathfrak q_1^{\mu_{k-1,j}-i}\mathfrak q_2^{j-\mu_{0,i}^{\prime}-1}\right)}}{\displaystyle{\left(1-\mathfrak q_1^{\mu_{k,j}-i}\mathfrak q_2^{k-\mu_{k,i}^{\prime}-1}\right)\left(1-\mathfrak q_1^{\mu_{k-1,j}-i}\mathfrak q_2^{j-\mu_{0,i}^{\prime}-1}\right)}}. \end{align} $$

The limit $y\to 0$ manifestly reverts to the case of the equivariant holomorphic Euler characteristic of the moduli space of nested instantons.

A similar result holds also for the general case $r>1$ :

(3.3.7) $$ \begin{align} \begin{aligned} \chi^{\mathrm{vir}}_{-y}({\mathcal N}(r,{\mathbf n}),\tilde V;\mathfrak q_1,\mathfrak q_2,\{\mathfrak t_i\})=&\sum_{\substack{\boldsymbol\mu_1\subseteq\cdots\subseteq\boldsymbol\mu_0\\ |\boldsymbol\mu_0\setminus\boldsymbol\mu_j|=n_j}}\frac{\displaystyle{T_{\boldsymbol\mu_0,\boldsymbol\mu_1}^{(r),y}(\mathfrak q_1,\mathfrak q_2)W_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}^{(r),y}(\mathfrak q_1,\mathfrak q_2)}}{\displaystyle{N_{\boldsymbol\mu_0}^{(r),y}(\mathfrak q_1,\mathfrak q_2)}}\left.\left[\tilde V\right]\right|_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}, \end{aligned} \end{align} $$

with

$$ \begin{align*} N_{\boldsymbol\mu_0}^{(r),y}(\mathfrak q_1,\mathfrak q_2)&=\prod_{a,b=1}^r\prod_{s\in Y_{\mu_0^{(a)}}}\frac{\displaystyle{\left(1-\mathfrak t_{ab}\mathfrak q_1^{-l_a(s)-1}\mathfrak q_2^{a_b(s)}\right)\left(1-\mathfrak t_{ab}\mathfrak q_1^{l_a(s)}\mathfrak q_2^{-a_b(s)-1}\right)}}{\displaystyle{\left(1-y\mathfrak t_{ab}\mathfrak q_1^{-l_a(s)-1}\mathfrak q_2^{a_b(s)}\right)\left(1-y\mathfrak t_{ab}\mathfrak q_1^{l_a(s)}\mathfrak q_2^{-a_b(s)-1}\right)}},\\ T_{\boldsymbol\mu_0,\boldsymbol\mu_1}^{(r),y}(\mathfrak q_1,\mathfrak q_2)&=\prod_{a,b}^r\prod_{i=1}^{M_0^{(a)}}\prod_{j=1}^{\mu_{0,i}^{(a)\prime}-\mu_{1,i}^{(a)\prime}}\frac{\displaystyle{\left(1-\mathfrak t_{ab}\mathfrak q_1^{-i}\mathfrak q_2^{-j-\mu_{1,i}^{(a)\prime}}\right)}}{\displaystyle{\left(1-y\mathfrak t_{ab}\mathfrak q_1^{-i}\mathfrak q_2^{-j-\mu_{1,i}^{(a)\prime}}\right)}},\\ \nonumber W_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}^{(r),y}(\mathfrak q_1,\mathfrak q_2)&=\prod_{k=1}^{N}\prod_{a,b}^r\prod_{i=1}^{M_0^{(a)}}\prod_{j=1}^{N_0^{(b)}}\frac{\displaystyle{\left(1-\mathfrak t_{ab}\mathfrak q_1^{\mu_{k,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{0,i}^{(a)\prime}-1}\right)\left(1-\mathfrak t_{ab}\mathfrak q_1^{\mu_{k-1,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{k,i}^{(a)\prime}-1}\right)}}{\displaystyle{\left(1-y\mathfrak t_{ab}\mathfrak q_1^{\mu_{k,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{0,i}^{(a)\prime}-1}\right)\left(1-y\mathfrak t_{ab}\mathfrak q_1^{\mu_{k-1,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{k,i}^{(a)\prime}-1}\right)}}\cdot\\ &\quad \cdot\frac{\displaystyle{\left(1-y\mathfrak t_{ab}\mathfrak q_1^{\mu_{k,j}^{(b)}-i}\mathfrak q_2^{k-\mu_{k,i}^{(a)\prime}-1}\right)\left(1-y\mathfrak t_{ab}\mathfrak q_1^{\mu_{k-1,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{0,i}^{(a)\prime}-1}\right)}}{\displaystyle{\left(1-\mathfrak t_{ab}\mathfrak q_1^{\mu_{k,j}^{(b)}-i}\mathfrak q_2^{k-\mu_{k,i}^{(a)\prime}-1}\right)\left(1-\mathfrak t_{ab}\mathfrak q_1^{\mu_{k-1,j}^{(b)}-i}\mathfrak q_2^{j-\mu_{0,i}^{(a)\prime}-1}\right)}}, \end{align*} $$

with the same notations of the previous section.

Virtual Euler number

Recall that the virtual Euler characteristic $e^{\mathrm {vir}}(X)$ and the virtual signature $\sigma ^{\mathrm { vir}}(X)$ of a scheme X endowed with a perfect obstruction theory are defined as $\chi _{-1}^{\mathrm {vir}}(X)$ and $\chi _{1}^{\mathrm {vir}}(X)$ , respectively (cf. [17, § 5]). Here, we are interested in virtual Euler characteristics, for which we use an analogous definition in the equivariant context. In general, these are highly nontrivial to compute explicitly, even when the ${\mathbb T}$ -fixed locus is isolated reduced. An interesting feature in this case is that the computation of virtual Euler characteristics of nested Hilbert schemes of points seems to reduce to just the enumeration of ${\mathbb T}$ -fixed points.

Conjecture 3.9. Let $Z(q_0,q_1,\dots )$ be the generating function of virtual Euler characteristics of nested Hilbert schemes, that is

There is an identity

$$\begin{align*}Z(q_0,q_1,\dots)=\sum_{j=0}^\infty\sum_{n_0,\dots,n_j}\#\left\{\mu_1\subseteq\cdots\subseteq\mu_j\subseteq\mu_0\right\}q_0^{n_0}\cdots q_j^{n_j}. \end{align*}$$

Unfortunately, we are not able to provide a complete proof of the previous conjecture as of yet. We checked its validity numerically up to $n_0=10$ . However, assuming the validity of Conjecture 3.9, we are able to express the generating function of virtual Euler characteristic in closed form for specific nesting profiles ${\mathbf n}$ .

Proposition 3.10. Assuming Conjecture 3.9 holds, we have the following identities:

(3.3.8a) $$ \begin{align} & \sum_{n\ge 0}e^{\mathrm{vir}}\left({\mathcal N}(1,n)\right)q^n=\prod_{k=1}^\infty\left(\frac{1}{1-q^k}\right) \end{align} $$

(3.3.8b) $$ \begin{align} & \sum_{n\ge 1}e^{\mathrm{vir}}\left({\mathcal N}(1,n,n-1)\right)q^n=-1+\sum_{n\ge 0}e^{\mathrm{vir}}\left({\mathcal N}(1,n)\right)q^n \end{align} $$

(3.3.8c) $$ \begin{align} & \sum_{n\ge 1}e^{\mathrm{vir}}\left({\mathcal N}(1,n,1)\right)q^n=\frac{q}{1-q}\prod_{k=1}^\infty\left(\frac{1}{1-q^k}\right). \end{align} $$

Proof. Equation (3.3.8a) follows from the isomorphism ${\mathcal N}(1,n)\cong \mathcal M(1,n)$ . Fixed points are in bijection with integer partitions, whose partition function is precisely given by Equation (3.3.8a). Similarly, fixed points in ${\mathcal N}(1,n,n-1)$ are in bijection with collections of nested partitions $\mu _1\subset \mu _0$ , such that $|\mu _0|=n$ , $|\mu _1|=1$ . Given $\mu _0$ , there is just one possible choice for $\mu _1\subset \mu _0$ . Thus, assuming Conjecture 3.9 holds, $e^{\mathrm {vir}}({\mathcal N}(1,n,n-1))$ counts partitions of size at least one, which immediately implies Equation 3.3.8b. Finally, the fixed locus in ${\mathcal N}(1,n,1)$ is in bijection with nested partitions $\mu _1\subset \mu _0$ , such that $|\mu _0|=n$ , $|\mu _1|=n-1$ . Given $\mu _0$ with $|\mu _0|=n$ , the possible choices for $\mu _1\subset \mu _0$ are determined by the boxes $s\in Y_{\mu _0}$ in the Young diagram $Y_{\mu _0}$ associated to $\mu _0$ such that $a_{\mu _0}(s)+l_{\mu _0}(s)=0$ . If we let $P(n)$ be the number of all such boxes in all integer partitions of n, one has (cf., for example, [49, Ex. 1.80])

$$\begin{align*}\sum_{n\ge 1}P(n)q^n=\sum_{n\ge 1}\sum_{k=0}^{n-1}p(k)q^n, \end{align*}$$

where $p(k)$ denotes the number of integer partitions of k. Then

$$\begin{align*}\operatorname{\mathrm{Coeff}}_{q^n}\left(\frac{q}{1-q}\prod_{k=1}^\infty\frac{1}{1-q^k}\right)=\sum_{k=0}^{n-1}\operatorname{\mathrm{Coeff}}_{q^k}\left(\prod_{k=1}^\infty\frac{1}{1-q^k}\right)=\sum_{k=0}^{n-1}p(k), \end{align*}$$

whence we get Equation (3.3.8c).

Proposition 3.11. Assuming Conjecture 3.9 holds, by setting $|{\mathbf n}|=n_0+\cdots +n_N$ there is an identity of generating functions

$$\begin{align*}Z(q)=\sum_{n\ge 0}\sum_{|{\mathbf n}|=n}e^{\mathrm{vir}}\left({\mathcal N}(1,{\mathbf n})\right)q^n=M(q), \end{align*}$$

where $M(q)$ is the MacMahon function

Proof. Assuming Conjecture 3.9 holds, $Z(q)$ counts the number of all unrestricted nested partitions. A fixed point $\mu _1\subseteq \cdots \subseteq \mu _N\subseteq \mu _0$ in ${\mathcal N}(1,{\mathbf n})$ determines a plane partition $\pi $ of size $|\pi |=(N+1)n_0-n_1-\cdots -n_N$ as follows:

$$\begin{align*}\pi_{ij}=\#\left\{\text{partitions }\mu\text{ in }\mu_0\supseteq\mu_N\supseteq\cdots\supseteq\mu_1\text{ s.t. }(\mu)_i\ge j\right\}. \end{align*}$$

Conversely, any plane partition $\pi $ determines a nested partition $\mu _0\supseteq \cdots \supseteq \mu _{\pi _{11}}$ as

$$\begin{align*}(\mu_{k})_i=\#\left\{\pi_{ij}\text{ in }\pi\text{ s.t. }\pi_{ij}\ge k\right\}. \end{align*}$$

We conclude that $Z(q)$ is nothing but the generating function of plane partitions (i.e., the MacMahon function).

Proposition 3.12. Considering the generating function of (non-virtual) Euler characteristics, there is an identity

$$\begin{align*}\sum_{n\ge 0}\sum_{|{\mathbf n}|=n}e({\mathcal N}(1,{\mathbf n}))q^n=M(q). \end{align*}$$

Proof. The Euler characteristic of a scheme coincides with the one of its ${\mathbb T}$ -fixed locus. The result then follows immediately from the fact that the ${\mathcal N}(1,{\mathbf n})^{{\mathbb T}}$ is reduced and zero-dimensional, so the Euler characteristic is just counting the number of nested partitions.

3.4 Virtual equivariant elliptic genus

A further refinement of the virtual $\chi _{-y}$ -genus is finally given by the virtual elliptic genus. In this case, if F is any vector bundle over X, we define

$$\begin{align*}{\mathcal E}(F)=\bigotimes_{n\ge 1}\left(\Lambda_{-yq^n}F^\vee\otimes\Lambda_{-y^{-1}q^n}F\otimes S_{q^n}(F\oplus F^\vee)\right)\in 1+q\cdot K^0(X)[y,y^{-1}][\![q]\!], \end{align*}$$

so that the virtual elliptic genus $\operatorname {\mathrm {Ell}}^{\mathrm {vir}}(X;y,q)$ of X is defined by

$$\begin{align*}\operatorname{\mathrm{Ell}}^{\mathrm{vir}}(X;y,q)=y^{-d/2}\chi_{-y}^{\mathrm{vir}}(X,{\mathcal E}(T_X^{\mathrm{vir}}))\in\mathbb Q(\!(y^{1/2})\!)[\![q]\!], \end{align*}$$

and also

$$\begin{align*}\operatorname{\mathrm{Ell}}^{\mathrm{vir}}(X,V;y,q)=y^{-d/2}\chi_{-y}^{\mathrm{vir}}(X,{\mathcal E}(T_X^{\mathrm{vir}})\otimes V). \end{align*}$$

By using virtual Riemann-Roch again, one can see that $\operatorname {\mathrm {Ell}}^{\mathrm {vir}}(X;y,q)$ admits an integral form

$$ \begin{align*} \operatorname{\mathrm{Ell}}^{\mathrm{vir}}(X;y,q) &=\int_{[X]^{\mathrm{vir}}}{{{\mathcal E}}\ell\ell}(T_X^{\mathrm{vir}};y,q),\\ \operatorname{\mathrm{Ell}}^{\mathrm{vir}}(X,V;y,q) &=\int_{[X]^{\mathrm{vir}}}{{{\mathcal E}}\ell\ell}(T_X^{\mathrm{vir}};y,q)\cdot\operatorname{\mathrm{ch}}(V), \end{align*} $$

with

$$\begin{align*}{{{\mathcal E}}\ell\ell}(F;y,q)=y^{-\operatorname{\mathrm{rk}} F/2}\operatorname{\mathrm{ch}}(\Lambda_{-y}F^\vee)\cdot\operatorname{\mathrm{ch}}({\mathcal E}(F))\cdot\operatorname{\mathrm{td}}(F)\in A^{*}(X)[y^{-1/2},y^{1/2}][\![q]\!]. \end{align*}$$

It is also interesting to study how the virtual elliptic genus is described in terms of the usual Chern roots $x_i$ , $u_j$ , $v_k$ , as its formula involves the Jacobi theta function $\theta (z,\tau )$ defined as

$$\begin{align*}\theta(z,\tau)=q^{1/8}\frac{y^{1/2}-y^{-1/2}}{{\text i}}\prod_{l=1}^\infty(1-q^l)(1-q^ly)(1-q^ly^{-1}), \end{align*}$$

where $q={\text e}^{2\pi {\text i}\tau }$ and $y={\text e}^{2\pi {\text i} z}$ . In fact, if F is any vector bundle over X with Chern roots $\{f_i\}$ , one can prove [6] that

$$\begin{align*}{{{\mathcal E}}\ell\ell}(F;z,\tau)=\prod_{i=1}^{\operatorname{\mathrm{rk}} F}f_i\frac{\theta(f_i/2\pi{\text i}-z,\tau)}{\theta(f_i/2\pi{\text i},\tau)}, \end{align*}$$

so that

$$ \begin{align*} \operatorname{\mathrm{Ell}}^{\mathrm{vir}}(X;y,q) &=\int_{[X]^{\mathrm{vir}}}\prod_{i=1}^{n}x_i\frac{\theta(x_i/2\pi{\text i}-z,\tau)}{\theta(x_i/2\pi{\text i},\tau)}\prod_{j=1}^{m}\frac{1}{u_j}\frac{\theta(u_j/2\pi{\text i},\tau)}{\theta(u_j/2\pi{\text i}-z,\tau)},\\ \operatorname{\mathrm{Ell}}^{\mathrm{vir}}(X,V;y,q) &=\int_{[X]^{\mathrm{vir}}}\left(\sum_{k=1}^r{\text e}^{v_k}\right)\prod_{i=1}^{n}x_i\frac{\theta(x_i/2\pi{\text i}-z,\tau)}{\theta(x_i/2\pi{\text i},\tau)}\prod_{j=1}^{m}\frac{1}{u_j}\frac{\theta(u_j/2\pi{\text i},\tau)}{\theta(u_j/2\pi{\text i}-z,\tau)}. \end{align*} $$

Finally, by taking the same steps as in the previous paragraphs, we can equivariantly extend the definition of the virtual elliptic genus, and by virtual localization, we find that

$$\begin{align*}\operatorname{\mathrm{Ell}}^{\mathrm{vir}}(X,\tilde V,z,\tau;\varepsilon_1,\dots,\varepsilon_N)=y^{-\operatorname{\mathrm{vd}}/2}\sum_ip_{i*}^{\mathrm{vir}}\left(\tilde V_i\otimes{\mathcal E}(T_X^{\mathrm{vir}}\otimes\Lambda_{-y}(\Omega_X^{\mathrm{ vir}}|_{X_i})/\Lambda_{-1}(N_i^{\mathrm{vir}})^\vee\right) \end{align*}$$

and $\operatorname {\mathrm {Ell}}^{\mathrm {vir}}(X,V)=\operatorname {\mathrm {Ell}}^{\mathrm {vir}}(X,\tilde V;0,\dots ,0)$ . In particular, we get in rank 1

(3.4.1) $$ \begin{align} \begin{aligned} \operatorname{\mathrm{Ell}}^{\mathrm{vir}}({\mathcal N}(1,{\mathbf n}),\tilde V;\varepsilon,\varepsilon_2)=&\sum_{\substack{\mu_1\subseteq\cdots\subseteq\mu_0\\ |\mu_0\setminus\mu_j|=n_j}}\frac{\displaystyle{\mathcal T_{\mu_0,\mu_1}^{z,\tau}(\varepsilon_1,\varepsilon_2)\mathcal W_{\mu_0,\dots,\mu_N}^{z,\tau}(\varepsilon_1,\varepsilon_2)}}{\displaystyle{{\mathcal N}_{\mu_0}^{z,\tau}(\varepsilon_1,\varepsilon_2)}})\left.\left[\tilde V\right]\right|_{\mu_0,\dots,\mu_N}, \end{aligned} \end{align} $$

with

$$ \begin{align*} {\mathcal N}_{\mu_0}^{z,\tau}(\varepsilon_1,\varepsilon_2)&=\prod_{s\in Y_{\mu_0}}\left[\frac{\displaystyle{\theta(\epsilon_1(l(s)+1)-\epsilon_2a(s),\tau)}}{\displaystyle{\theta(\epsilon_1(l(s)+1)-\epsilon_2a(s)-z,\tau)}}\cdot\right.\\ &\qquad\cdot\left.\frac{\displaystyle{\theta(-\epsilon_1l(s)+\epsilon_2(a(s)+1),\tau)}}{\displaystyle{\theta(-\epsilon_1l(s)+\epsilon_2(l(s)+1)-z,\tau)}}\right],\\ \mathcal T_{\mu_0,\mu_1}^{z,\tau}(\varepsilon_1,\varepsilon_2)&=\prod_{i=1}^{M_0}\prod_{j=1}^{\mu_{0,i}^{\prime}-\mu_{1,i}^{\prime}}\frac{\displaystyle{\theta(\epsilon_1i+\epsilon_2(j+\mu_{1,i}^{\prime})-z,\tau)}}{\displaystyle{\theta(\epsilon_1i+\epsilon_2(j+\mu_{1,i}^{\prime}),\tau)}},\\ \nonumber \mathcal W_{\mu_0,\dots,\mu_N}^{z,\tau}(\varepsilon_1,\varepsilon_2)&=\prod_{k=1}^{N}\prod_{i=1}^{M_0}\prod_{j=1}^{N_0}\left[\frac{\displaystyle{\theta(\epsilon_1(i-\mu_{k,j})+\epsilon_2(1+\mu_{0,i}^{\prime}-j),\tau)}}{\displaystyle{\theta(\epsilon_1(i-\mu_{k,j})+\epsilon_2(1+\mu_{0,i}^{\prime}-j)-z,\tau)}}\cdot\right.\\ \nonumber &\qquad\qquad\qquad\cdot\frac{\displaystyle{\theta(\epsilon_1(i-\mu_{k-1,j})+\epsilon_2(1+\mu_{k,i}^{\prime}-j),\tau)}}{\displaystyle{\theta(\epsilon_1(i-\mu_{k-1,j})+\epsilon_2(1+\mu_{k,i}^{\prime}-j)-z,\tau)}}\\ \nonumber &\qquad\qquad\qquad\cdot\frac{\displaystyle{\theta(\epsilon_1(i-\mu_{k,j})+\epsilon_2(1+\mu_{k,i}^{\prime}-j)-z,\tau)}}{\displaystyle{\theta(\epsilon_1(i-\mu_{k,j})+\epsilon_2(1+\mu_{k,i}^{\prime}-j),\tau)}}\\ &\qquad\qquad\qquad\left.\cdot\frac{\displaystyle{\theta(\epsilon_1(i-\mu_{k-1,j})+\epsilon_2(1+\mu_{0,i}^{\prime}-j)-z,\tau)}}{\displaystyle{\theta(\epsilon_1(i-\mu_{k-1,j})+\epsilon_2(1+\mu_{0,i}^{\prime}-j),\tau)}}\right], \end{align*} $$

with $\epsilon _i=\varepsilon _i/2\pi {\text i}$ . One can easily see that the virtual elliptic genus we just computed is indeed a Jacobi form and that its limit $\tau \to {\text i}\infty $ reproduces the $\chi _{-y}$ -genus. Moreover, by taking the limit $y\to 0$ in the $\chi _{-y}$ -genus, one can recover the virtual equivariant holomorphic Euler characteristic.

Finally, if we study the virtual equivariant elliptic genus in the more general case of rank $r\ge 1$ , we get

(3.4.2) $$ \begin{align} \begin{aligned} \operatorname{\mathrm{Ell}}^{\mathrm{vir}}({\mathcal N}(r,{\mathbf n}),\tilde V;\varepsilon,\varepsilon_2,\{a_i\})=&\sum_{\substack{\boldsymbol\mu_1\subseteq\cdots\subseteq\boldsymbol\mu_0\\ |\boldsymbol\mu_0\setminus\boldsymbol\mu_j|=n_j}}\frac{\displaystyle{\mathcal T_{\boldsymbol\mu_0,\boldsymbol\mu_1}^{z,\tau}(\varepsilon_1,\varepsilon_2)\mathcal W_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}^{z,\tau}(\varepsilon_1,\varepsilon_2)}}{\displaystyle{{\mathcal N}_{\boldsymbol\mu_0}^{z,\tau}(\varepsilon_1,\varepsilon_2)}})\left.\left[\tilde V\right]\right|_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}, \end{aligned} \end{align} $$

with

$$ \begin{align*} {\mathcal N}_{\boldsymbol\mu_0}^{z,\tau}(\varepsilon_1,\varepsilon_2)&=\prod_{a,b=1}^r\prod_{s\in Y_{\mu_0}}\left[\frac{\displaystyle{\theta(a_{ab}+\epsilon_1(l(s)+1)-\epsilon_2a(s),\tau)}}{\displaystyle{\theta(a_{ab}+\epsilon_1(l(s)+1)-\epsilon_2a(s)-z,\tau)}}\cdot\right.\\ \nonumber &\qquad\qquad\qquad\cdot\left.\frac{\displaystyle{\theta(a_{ab}+-\epsilon_1l(s)+\epsilon_2(a(s)+1),\tau)}}{\displaystyle{\theta(a_{ab}+-\epsilon_1l(s)+\epsilon_2(l(s)+1)-z,\tau)}}\right],\\ \mathcal T_{\boldsymbol\mu_0,\boldsymbol\mu_1}^{z,\tau}(\varepsilon_1,\varepsilon_2)&=\prod_{a,b=1}^r\prod_{i=1}^{M_0^{(a)}}\prod_{j=1}^{\mu_{0,i}^{(a)\prime}-\mu_{1,i}^{(a)\prime}}\frac{\displaystyle{\theta(a_{ab}+\epsilon_1i+\epsilon_2(j+\mu_{1,i}^{(a)\prime}),\tau)}}{\displaystyle{\theta(a_{ab}+\epsilon_1i+\epsilon_2(j+\mu_{1,i}^{(a)\prime})-z,\tau)}},\\ \nonumber \mathcal W_{\boldsymbol\mu_0,\dots,\boldsymbol\mu_N}^{z,\tau}(\varepsilon_1,\varepsilon_2)&=\prod_{k=1}^{N}\prod_{a,b=1}^r\prod_{i=1}^{M_0^{(a)}}\prod_{j=1}^{N_0^{(b)}}\left[\frac{\displaystyle{\theta(a_{ab}+\epsilon_1(i-\mu_{k,j}^{(b)})+\epsilon_2(1+\mu_{0,i}^{(a)\prime}-j),\tau)}}{\displaystyle{\theta(a_{ab}+\epsilon_1(i-\mu_{k,j}^{(b)})+\epsilon_2(1+\mu_{0,i}^{(a)\prime}-j)-z,\tau)}}\cdot\right.\\ \nonumber &\qquad\qquad\qquad\qquad\cdot \frac{\displaystyle{\theta(a_{ab}+\epsilon_1(i-\mu_{k-1,j}^{(b)})+\epsilon_2(1+\mu_{k,i}-j)^{(a)\prime},\tau)}}{\displaystyle{\theta(a_{ab}+\epsilon_1(i-\mu_{k-1,j}^{(b)})+\epsilon_2(1+\mu_{k,i}^{(a)\prime}-j)-z,\tau)}}\cdot\\ \nonumber &\qquad\qquad\qquad\qquad\cdot\frac{\displaystyle{\theta(a_{ab}+\epsilon_1(i-\mu_{k,j}^{(b)})+\epsilon_2(1+\mu_{k,i}^{(a)\prime}-j)-z,\tau)}}{\displaystyle{\theta(a_{ab}+\epsilon_1(i-\mu_{k,j}^{(b)})+\epsilon_2(1+\mu_{k,i}^{(a)\prime}-j),\tau)}}\cdot\\ &\qquad\qquad\qquad\qquad\left.\cdot\frac{\displaystyle{\theta(a_{ab}+\epsilon_1(i-\mu_{k-1,j}^{(b)})+\epsilon_2(1+\mu_{0,i}^{(a)\prime}-j)-z,\tau)}}{\displaystyle{\theta(a_{ab}+\epsilon_1(i-\mu_{k-1,j}^{(b)})+\epsilon_2(1+\mu_{0,i}^{(a)\prime}-j),\tau)}}\right]. \end{align*} $$

Notice that by knowing the equivariant virtual elliptic genus one is able to recover both the virtual equivariant holomorphic Euler characteristic and $\chi _{-y}$ - genus. In fact, the limit $\tau \to {\text i}\infty $ of (3.4.2) recovers exactly the $\chi _{-y}$ -genus found in (3.3.7), and a successive limit $y\to 0$ gives us back the virtual equivariant holomorphic Euler characteristic (3.2.12).

4 Toric surfaces

In this section, we will generalize the results we got in the previous ones to the case of nested Hilbert schemes on toric surfaces, and in particular, we will be interested in ${\mathbb P}^2$ and ${\mathbb P}^1\times {\mathbb P}^1$ . This is because one might expect any complex genus of $\operatorname {\mathrm {Hilb}}^{({\mathbf n})}(S)$ to depend only on the cobordism class of S, as it was the case for $\operatorname {\mathrm {Hilb}}^n(S)$ ([16]), and the complex cobordism ring $\Omega =\Omega ^U\otimes \mathbb Q$ with rational coefficients was showed by Milnor to be a polynomial algebra freely generated by the cobordism classes $[{\mathbb P}^n]$ , $n>0$ . Then in the case of complex projective surfaces, any case can be reduced to ${\mathbb P}^2$ and ${\mathbb P}^1\times {\mathbb P}^1$ by the fact that $[S]=a[{\mathbb P}^2]+b[{\mathbb P}^1\times {\mathbb P}^1]$ . The advantage given by having an ADHM-like construction for the nested punctual Hilbert scheme on the affine plane is that it provides us with the local model of the more general case of smooth projective surfaces. In particular, whenever S is toric, one can construct it starting from its toric fan by appropriately glueing the affine patches (e.g., Figure 4a for ${\mathbb P}^2$ and 4b for ${\mathbb P}^1\times {\mathbb P}^1$ ), and computation of topological invariants can still be easily carried out by means of equivariant (virtual) localization.

Figure 4 Toric fans for ${\mathbb P}^2$ and ${\mathbb P}^1\times {\mathbb P}^1.$

In general, given the toric fan describing the patches which glued together make up a toric surface S, each patch $U_i$ will be $U_i\cong {\mathbb C}^2$ , with a natural action of ${\mathbb T}_1=({\mathbb C}^{*})^2$ . Moreover, if $S={\mathbb P}^2$ or $S={\mathbb P}^1\times {\mathbb P}^1$ and $Z\in \operatorname {\mathrm {Hilb}}^{({\mathbf n})}(S)$ is a fixed point of the ${\mathbb T}_1$ -action, its support must be contained in $\{P_0,\dots ,P_{\chi (S)-1}\}$ (as a consequence of [9]) with $P_i$ corresponding to the vertices of the polytope associated to the fan, so that one can write in general that $Z=Z_0\cup \cdots \cup Z_{\chi (S)-1}$ , with $Z_i$ being supported in $P_i$ . This also induces a decomposition of the representation in $R({\mathbb T}_1)$ of the virtual tangent space at the fixed points:

(4.0.1) $$ \begin{align} T^{\mathrm{vir}}_Z\left(\operatorname{\mathrm{Hilb}}^{({\mathbf n})}(S)\right)=\bigoplus_{\ell=0}^{\chi(S)-1}T^{\mathrm{vir}}_{Z_\ell}\left(\operatorname{\mathrm{Hilb}}^{({\mathbf n}_\ell)}(U_\ell)\right). \end{align} $$

Let then $\chi _{-y}^{{\mathbb T}_1,\mathrm {vir}}(P_{\ell },{\mathbf n}_\ell )$ be the ${\mathbb T}_1$ -equivariant virtual $\chi _{-y}$ -genus

(4.0.2)

corresponding to the affine patch $U_\ell $ of S. We will be able to compute the non-equivariant $\chi _{-y}$ -genera $\chi _{-y}^{\mathrm {vir}}(\operatorname {\mathrm {Hilb}}^{({\mathbf n})}({\mathbb P}^2))$ and $\chi _{-y}^{\mathrm {vir}}(\operatorname {\mathrm {Hilb}}^{({\mathbf n})}({\mathbb P}^1\times {\mathbb P}^1))$ in terms of $\chi _{-y}^{{\mathbb T}_1,{\mathrm {vir}}}(P_\ell ,{\mathbf n}_\ell )$ , thanks to Lemma 4.1.

Lemma 4.1. Let S be either ${\mathbb P}^2$ or ${\mathbb P}^1\times {\mathbb P}^1$ . The ${\mathbb T}_1-$ equivariant virtual $\chi _{-y}$ -genus $\chi _{-y}^{{\mathbb T}_1,{\mathrm {vir}}}(\operatorname {\mathrm {Hilb}}^{({\mathbf n})}(S))\in \mathbb Z[\![\mathfrak q_1,\mathfrak q_2]\!][y]$ is independent of the equivariant parameters $\mathfrak q_1,\mathfrak q_2$ , and

$$\begin{align*}\chi_{-y}^{{\mathbb T}_1,{\mathrm{vir}}}(\operatorname{\mathrm{Hilb}}^{({\mathbf n})}(S))=\chi_{-y}^{\mathrm{vir}}(\operatorname{\mathrm{Hilb}}^{({\mathbf n})}(S))\in K_0(\operatorname{\mathrm{pt}})[y]\cong\mathbb Z[y]. \end{align*}$$

Proof. Whenever S is a projective surface, the nested Hilbert scheme $\operatorname {\mathrm {Hilb}}^{({\mathbf n})}(S)$ is projective. Then we have that $\chi _{-y}^{{\mathbb T}_1,{\mathrm {vir}}}(\operatorname {\mathrm {Hilb}}^{({\mathbf n})}(S))$ is well defined as an element of $K_0^{{\mathbb T}_1}(\operatorname {\mathrm {pt}})[y]$ and, in particular, it has no poles of the form $y^a\mathfrak q_1^b\mathfrak q_2^c=1$ , for $a,b,c\in \mathbb Z$ . However, if S is either ${\mathbb P}^2$ or ${\mathbb P}^1\times {\mathbb P}^1$ , we can also compute $\chi _{-y}^{{\mathbb T}_1,{\mathrm {vir}}}(\operatorname {\mathrm {Hilb}}^{({\mathbf n})}(S))$ by applying the K-theoretic virtual localization theorem; cf. [45, Thm. 3.3], and using (4.0.1), we get

(4.0.3) $$ \begin{align} \chi_{-y}^{{\mathbb T}_1,{\mathrm{vir}}}(\operatorname{\mathrm{Hilb}}^{({\mathbf n})}(S))=\sum_{{\mathbf n}_0+\cdots+{\mathbf n}_{\chi(S)-1}={\mathbf n}}\prod_{\ell=0}^{\chi(S)-1}\left(\chi_{-y}^{{\mathbb T}_1,{\mathrm{vir}}}(P_\ell,{\mathbf n}_\ell)\right), \end{align} $$

where $\chi _{-y}^{{\mathbb T}_1,{\mathrm {vir}}}(P_\ell ,{\mathbf n}_\ell )$ is defined as in (4.0.2). Then, each one of the terms appearing in the right-hand side of (4.0.3) is a homogeneous rational expression of total degree 0 with respect to the variables $\mathfrak q_1,\mathfrak q_2$ , whose only poles can arise from terms of the form $(1-y^a\mathfrak q_1^b\mathfrak q_2^c)^{-1}$ ; cf. Equations (3.3.3)--(3.3.6). Since, by the previous arguments, there is no such pole, $\chi _{-y}^{{\mathbb T}_1,{\mathrm {vir}}}(\operatorname {\mathrm {Hilb}}^{({\mathbf n})}(S))$ is a constant in $\mathfrak q_1,\mathfrak q_2$ .

4.1 Case 1: $\boldsymbol{S}=\boldsymbol{{\mathbb P}}^{\boldsymbol{2}}$

We will be interested in the generating function

(4.1.1) $$ \begin{align} \sum_{{\mathbf n}\ge\boldsymbol 0}\chi_{-y}^{{\mathbb T}_1,{\mathrm{vir}}}\left(\operatorname{\mathrm{Hilb}}^{(\hat{{\mathbf n}})}({\mathbb P}^2)\right){\mathbf q}^{{\mathbf n}}=\prod_{\ell=0}^2\left(\sum_{{\mathbf n}_\ell\ge\boldsymbol 0}\chi_{-y}^{{\mathbb T}_1,{\mathrm{vir}}}(P_\ell,{\mathbf n}_\ell){\mathbf q}^{{\mathbf n}_\ell}\right), \end{align} $$

with $\hat {{\mathbf n}}$ defined as in §2.1, and since the left-hand side does not depend on $\mathfrak q_{1,2}$ , we can perform the computation of the non-equivariant virtual $\chi _{-y}$ -genus by first computing the right-hand side of Equation (4.0.3) in any limit of the equivariant parameters $\mathfrak q_1,\mathfrak q_2$ . The iterated limit $\mathfrak q_1\to +\infty $ , $\mathfrak q_2\to +\infty $ appears to be a particularly good choice, as it is well defined for each term contributing to the rhs of Equation 4.0.3, and the computation can be performed term-by-term. In each one of the three affine patches, the weights of the torus action will be

$$\begin{align*}\begin{aligned} &\mathfrak q_{1,(0)}=\mathfrak q_1\\ &\mathfrak q_{1,(1)}=1/\mathfrak q_1\\ &\mathfrak q_{1,(2)}=1/\mathfrak q_2 \end{aligned}\qquad\qquad\qquad \begin{aligned} &\mathfrak q_{2,(0)}=\mathfrak q_2\\ &\mathfrak q_{2,(1)}=\mathfrak q_2/\mathfrak q_1\\ &\mathfrak q_{2,(2)}=\mathfrak q_1/\mathfrak q_2. \end{aligned} \end{align*}$$

We will study separately the three patches $\ell =0,1,2$ . First of all, we notice that since the $\chi _{-y}$ -genus is multiplicative, the first contribution coming from $N_{\mu _0}^y(\mathfrak q_1,\mathfrak q_2)$ coincides with the same contribution arising in the context of standard Hilbert schemes. It was shown in [35] that

$$ \begin{align*} &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\frac{1}{N_{\mu_0,\mu_1}^{-y}(\mathfrak q_{1,(0)},\mathfrak q_{2,(0)})}=y^{|\mu_0|-M_0},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\frac{1}{N_{\mu_0,\mu_1}^{-y}(\mathfrak q_{1,(1)},\mathfrak q_{2,(1)})}=y^{|\mu_0|},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\frac{1}{N_{\mu_0,\mu_1}^{-y}(\mathfrak q_{1,(2)},\mathfrak q_{2,(2)})}=y^{|\mu_0|+s(\mu_0)},\quad s(\mu_0)=\#\{s\in Y_{\mu_0^{\prime}}:a(s)\le l(s)\le a(s)+1\}, \end{align*} $$

so that we just need to evaluate the other contributions. Starting from $T^{-y}_{\mu _0,\mu _1}$ , we get

$$ \begin{align*} &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\left[\prod_{i=1}^{M_0}\prod_{j=1}^{\mu_{0,i}^{\prime}-\mu_{1,i}^{\prime}}\frac{\displaystyle{\left(1-\mathfrak q_{1,(0)}^{-i}\mathfrak q_{2,(0)}^{-j-\mu_{1,i}^{\prime}}\right)}}{\displaystyle{\left(1-y\mathfrak q_{1,(0)}^{-i}\mathfrak q_{2,(0)}^{-j-\mu_{1,i}^{\prime}}\right)}}\right]=1,\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\left[\prod_{i=1}^{M_0}\prod_{j=1}^{\mu_{0,i}^{\prime}-\mu_{1,i}^{\prime}}\frac{\displaystyle{\left(1-\mathfrak q_{1,(1)}^{-i}\mathfrak q_{2,(1)}^{-j-\mu_{1,i}^{\prime}}\right)}}{\displaystyle{\left(1-y\mathfrak q_{1,(1)}^{-i}\mathfrak q_{2,(1)}^{-j-\mu_{1,i}^{\prime}}\right)}}\right]=y^{-1},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\left[\prod_{i=1}^{M_0}\prod_{j=1}^{\mu_{0,i}^{\prime}-\mu_{1,i}^{\prime}}\frac{\displaystyle{\left(1-\mathfrak q_{1,(2)}^{-i}\mathfrak q_{2,(2)}^{-j-\mu_{1,i}^{\prime}}\right)}}{\displaystyle{\left(1-y\mathfrak q_{1,(2)}^{-i}\mathfrak q_{2,(2)}^{-j-\mu_{1,i}^{\prime}}\right)}}\right]=1, \end{align*} $$

whence

$$ \begin{align*} &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}T_{\mu_0,\mu_1}^{-y}(\mathfrak q_{1,(0)},\mathfrak q_{2,(0)})=1,\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}T_{\mu_0,\mu_1}^{-y}(\mathfrak q_{1,(1)},\mathfrak q_{2,(1)})=y^{-|\mu_0\setminus\mu_1|},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}T_{\mu_0,\mu_1}^{-y}(\mathfrak q_{1,(2)},\mathfrak q_{2,(2)})=1. \end{align*} $$

Finally, we need to take care of the limit involving $W^{-y}_{\mu _0,\dots ,\mu _N}(\mathfrak q_1,\mathfrak q_2)$ , and in order to tackle, let us first point out that we can rewrite $W^{-y}_{\mu _0,\dots ,\mu _N}$ in the following simpler form

$$\begin{align*}\begin{aligned} W_{\mu_0,\dots,\mu_N}^{-y}(\mathfrak q_1,\mathfrak q_2)=\prod_{k=1}^{N}\prod_{s\in Y_{\mu_0^{\mathrm{rec}}}}\frac{\displaystyle{\left(1-\mathfrak q_1^{l_k(s)}\mathfrak q_2^{-a_0(s)-1}\right)\left(1-\mathfrak q_1^{l_{k-1}(s)}\mathfrak q_2^{-a_k(s)-1}\right)}}{\displaystyle{\left(1-y\mathfrak q_1^{l_k(s)}\mathfrak q_2^{-a_0(s)-1}\right)\left(1-y\mathfrak q_1^{l_{k-1}(s)}\mathfrak q_2^{-a_k(s)-1}\right)}}\cdot\\ \cdot\frac{\displaystyle{\left(1-y\mathfrak q_1^{l_k(s)}\mathfrak q_2^{-a_k(s)-1}\right)\left(1-y\mathfrak q_1^{l_{k-1}(s)}\mathfrak q_2^{-a_0(s)-1}\right)}}{\displaystyle{\left(1-\mathfrak q_1^{l_k(s)}\mathfrak q_2^{-a_k(s)-1}\right)\left(1-\mathfrak q_1^{l_{k-1}(s)}\mathfrak q_2^{-a_0(s)-1}\right)}}, \end{aligned} \end{align*}$$

where $\mu _0^{\mathrm {rec}}$ is the smallest rectangular partition containing $\mu _0$ and $a_k(s)$ (resp. $l_k(s)$ ) denotes the arm length (resp. leg length) of the box s with respect to $Y_{\mu _k}$ . Then, by recalling that the partitions labelling the T-fixed points are included one into the other as $\mu _1\subseteq \cdots \subseteq \mu _N\subseteq \mu _0\subseteq \mu _0^{\mathrm {rec}}$ , it is easy to realize that, in the case $\ell =0$ , one gets

$$\begin{align*}\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\frac{\displaystyle{1-\mathfrak q_1^{l_k(s)}\mathfrak q_2^{-a_0(s)-1}}}{\displaystyle{1-y\mathfrak q_1^{l_k(s)}\mathfrak q_2^{-a_0(s)-1}}}=\begin{cases} 1 &\mathrm{for}\ l_k(s)\le 0\\ y^{-1} &\mathrm{for}\ l_k(s)>0 \end{cases}, \end{align*}$$

and similarly in every other case,

$$ \begin{align*} &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\frac{\displaystyle{1-\mathfrak q_1^{l_{k-1}(s)}\mathfrak q_2^{-a_k(s)-1}}}{\displaystyle{1-y\mathfrak q_1^{l_{k-1}(s)}\mathfrak q_2^{-a_k(s)-1}}}=\begin{cases} 1 &\mathrm{for}\ l_{k-1}(s)\le 0\\ y^{-1} &\mathrm{for}\ l_{k-1}(s)>0 \end{cases},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\frac{\displaystyle{1-y\mathfrak q_1^{l_k(s)}\mathfrak q_2^{-a_k(s)-1}}}{\displaystyle{1-\mathfrak q_1^{l_k(s)}\mathfrak q_2^{-a_k(s)-1}}}=\begin{cases} 1 &\mathrm{for}\ l_{k}(s)\le 0\\ y &\mathrm{for}\ l_k(s)>0 \end{cases},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}\frac{\displaystyle{1-y\mathfrak q_1^{l_{k-1}(s)}\mathfrak q_2^{-a_0(s)-1}}}{\displaystyle{1-\mathfrak q_1^{l_{k-1}(s)}\mathfrak q_2^{-a_0(s)-1}}}=\begin{cases} 1 &\mathrm{for}\ l_{k-1}(s)\le 0\\ y &\mathrm{for}\ l_{k-1}(s)>0 \end{cases}, \end{align*} $$

so that finally

$$\begin{align*}\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}W^{-y}_{\mu_0,\dots,\mu_N}(\mathfrak q_{1,(0)},\mathfrak q_{2,(0)})=1. \end{align*}$$

It is easy to see that the same holds true also for $\ell =2$ :

$$\begin{align*}\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}W^{-y}_{\mu_0,\dots,\mu_N}(\mathfrak q_{1,(2)},\mathfrak q_{2,(2)})=1, \end{align*}$$

while the case $\ell =1$ is more difficult, even though the analysis of the different cases can be carried out exactly in the same way. We then introduce the following notation:

$$\begin{align*}s(\mu_{i_1},\mu_{i_2})=\#\left\{s\in Y_{\mu_0^{\mathrm{rec}}}:l_{i_1}(s)>a_{i_2}(s)+1\vee l_{i_1}(s)=a_{i_2}(s)+1,a_{i_2}(s)<-1\right\}, \end{align*}$$

and we get

$$\begin{align*}\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}W^{-y}_{\mu_0,\dots,\mu_N}(\mathfrak q_{1,(1)},\mathfrak q_{2,(1)})=\prod_{k=1}^Ny^{s(\mu_k,\mu_k)+s(\mu_{k-1},\mu_0)-s(\mu_k,\mu_0)-s(\mu_{k-1},\mu_k)}. \end{align*}$$

Finally, by putting everything together, we have an explicit expression for (4.1.1):

(4.1.2) $$ \begin{align} \begin{aligned} \sum_{{\mathbf n}}\chi_{-y}^{\mathrm{vir}}\left(\operatorname{\mathrm{Hilb}}^{(\hat{{\mathbf n}})}({\mathbb P}^2)\right){\mathbf q}^{{\mathbf n}}=\left(\sum_{{\mathbf n}}{\mathbf q}^{{\mathbf n}}\sum_{\{\mu_i\}}y^{|\mu_0|+M_0}\right)\left(\sum_{{\mathbf n}}{\mathbf q}^{{\mathbf n}}\sum_{\{\mu_i\}}y^{|\mu_0|-|\mu_0\setminus\mu_1|}\right.\\ \left.\prod_{k=1}^Ny^{s(\mu_k,\mu_k)+s(\mu_{k-1},\mu_0)}y^{-s(\mu_k,\mu_0)-s(\mu_{k-1},\mu_k)}\right)\left(\sum_{{\mathbf n}}{\mathbf q}^{{\mathbf n}}\sum_{\{\mu_i\}}y^{|\mu_0|-s(\mu_0)}\right) \end{aligned} \end{align} $$

4.2 Case 2: $S={\mathbb P}^1\times {\mathbb P}^1$

Similarly to the previous case, we are interested in studying the following generating function:

$$\begin{align*}\sum_{{\mathbf n}\ge\boldsymbol 0}\chi_{-y}^{{\mathbb T}_1,{\mathrm{vir}}}\left(\operatorname{\mathrm{Hilb}}^{(\hat{{\mathbf n}})}({\mathbb P}^1\times{\mathbb P}^1)\right){\mathbf q}^{{\mathbf n}}=\prod_{\ell=0}^3\left(\sum_{{\mathbf n}_\ell\ge\boldsymbol 0}\chi_{-y}^{{\mathbb T}_1,{\mathrm{vir}}}(P_\ell,{\mathbf n}_\ell){\mathbf q}^{{\mathbf n}_\ell}\right), \end{align*}$$

and we can perform the computation by taking the successive limits $\mathfrak q_1\to +\infty $ , $\mathfrak q_2\to +\infty $ , as in §4.1. The four patches are now indexed by $\ell =(00),(01),(10),(11)$ , and the characters $\mathfrak q_{i,(\ell )}$ can be identified to be in this case

$$\begin{align*}\begin{aligned} &\mathfrak q_{1,(00)}=\mathfrak q_1\\ &\mathfrak q_{1,(01)}=\mathfrak q_1\\ &\mathfrak q_{1,(10)}=1/\mathfrak q_1\\ &\mathfrak q_{1,(11)}=1/\mathfrak q_1\\ \end{aligned}\qquad\qquad\qquad \begin{aligned} &\mathfrak q_{2,(00)}=\mathfrak q_2\\ &\mathfrak q_{2,(01)}=1/\mathfrak q_2\\ &\mathfrak q_{2,(10)}=\mathfrak q_2\\ &\mathfrak q_{2,(11)}=1/\mathfrak q_2.\\ \end{aligned} \end{align*}$$

An analysis similar to the one carried out in the previous section enables then us to conclude the following:

$$\begin{align*}\begin{aligned} &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}1/N^{-y}_{\mu_0}(\mathfrak q_{1,(00)},\mathfrak q_{2,(00)})=y^{|\mu_0|-M_0},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}1/N^{-y}_{\mu_0}(\mathfrak q_{1,(00)},\mathfrak q_{2,(01)})=y^{|\mu_0|},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}1/N^{-y}_{\mu_0}(\mathfrak q_{1,(00)},\mathfrak q_{2,(10)})=y^{|\mu_0|},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}1/N^{-y}_{\mu_0}(\mathfrak q_{1,(00)},\mathfrak q_{2,(11)})=y^{|\mu_0|+M_0},\\ \end{aligned} \end{align*}$$

$$\begin{align*}\begin{aligned} &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}T^{-y}_{\mu_0,\mu_1}(\mathfrak q_{1,(00)},\mathfrak q_{2,(00)})=1,\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}T^{-y}_{\mu_0,\mu_1}(\mathfrak q_{1,(00)},\mathfrak q_{2,(01)})=1,\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}T^{-y}_{\mu_0,\mu_1}(\mathfrak q_{1,(00)},\mathfrak q_{2,(10)})=y^{-|\mu_0\setminus\mu_1|},\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}T^{-y}_{\mu_0,\mu_1}(\mathfrak q_{1,(00)},\mathfrak q_{2,(11)})=y^{-|\mu_0\setminus\mu_1|},\\ \end{aligned} \end{align*}$$

and

$$\begin{align*}\begin{aligned} &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}W^{-y}_{\mu_0,\mu_1,\dots,\mu_N}(\mathfrak q_{1,(00)},\mathfrak q_{2,(00)})=1,\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}W^{-y}_{\mu_0,\mu_1,\dots,\mu_N}(\mathfrak q_{1,(00)},\mathfrak q_{2,(01)})=1,\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}W^{-y}_{\mu_0,\mu_1,\dots,\mu_N}(\mathfrak q_{1,(00)},\mathfrak q_{2,(10)})=1,\\ &\lim_{\mathfrak q_2\to+\infty}\lim_{\mathfrak q_1\to+\infty}W^{-y}_{\mu_0,\mu_1,\dots,\mu_N}(\mathfrak q_{1,(00)},\mathfrak q_{2,(11)})=1,\\ \end{aligned} \end{align*}$$

so that, by putting everything together, we have

(4.2.1) $$ \begin{align} \begin{aligned} \sum_{{\mathbf n}}\chi_{-y}^{\mathrm{vir}}\left(\operatorname{\mathrm{Hilb}}^{(\hat{{\mathbf n}})}({\mathbb P}^1\times{\mathbb P}^1)\right){\mathbf q}^{{\mathbf n}}=\left(\sum_{{\mathbf n}}{\mathbf q}^{{\mathbf n}}\sum_{\{\mu_i\}}y^{|\mu_0|-M_0}\right)\left(\sum_{{\mathbf n}}{\mathbf q}^{{\mathbf n}}\sum_{\{\mu_i\}}y^{|\mu_0|}\right)\\ \left(\sum_{{\mathbf n}}{\mathbf q}^{{\mathbf n}}\sum_{\{\mu_i\}}y^{|\mu_0|-|\mu_0\setminus\mu_1|}\right)\left(\sum_{{\mathbf n}}{\mathbf q}^{{\mathbf n}}\sum_{\{\mu_i\}}y^{|\mu_0|-|\mu_0\setminus\mu_1|+M_0}\right). \end{aligned} \end{align} $$