Published online by Cambridge University Press: 20 August 2001
Let $G$ be an almost simple algebraic group defined over ${\Bbb F}_p$ for some prime $p$. Denote by $G_1$ the first Frobenius kernel in $G$ and let $T$ be a maximal torus. In this paper we study certain Jantzen type filtrations on various modules in the representation theory of $G_1T$. We have such filtrations on the baby Verma modules $Z(\lambda)$, where $\lambda$ is a character of $T$. They are obtained via a certain deformation of the natural homomorphism from $Z(\lambda)$ into its contravariant dual $Z(\lambda)^\tau$. Using the same deformation we construct for each projective $G_1T$-module $Q$ a filtration of the vector space $F_\lambda(Q)=\text{Hom}_{G_1T}(Z(\lambda)^\tau, Q)$. We then prove that this filtration may also be described in terms of the above-mentioned homomorphism $Z(\lambda) \rightarrow Z(\lambda)^\tau$ and this leads us to a sum formula for our filtrations. When $Q$ is indecomposable with highest weight in the bottom alcove (with respect to some special point) we are able to compute the filtrations on $F_\lambda(Q)$ explicitly for all $\lambda$. This is then the starting point of an induction which proceeds via wall crossings to higher alcoves. If our filtrations behave as expected under such wall crossings then we obtain a precise relation between the dimensions of the layers in the filtrations of $F_\lambda (Q)$ for an arbitrary indecomposable projective $Q$ and the coefficients in the corresponding Kazhdan--Lusztig polynomials. We conclude the paper by proving that the above results in the $G_1T$ theory have some analogues in the representation theory of $G$ (where, however, we have to work with representations of bounded highest weights) and the corresponding theory for quantum groups at roots of unity. These results extend previous work by the first author. 2000 Mathematics Subject Classification: 20G05, 20G10, 17B37.