On commutative homogeneous vector bundles attached to nilmanifolds

The notion of Gelfand pair (G, K) can be generalized if we consider homogeneous vector bundles over G/K instead of the homogeneous space G/K and matrix-valued functions instead of scalar-valued functions. This gives the definition of commutative homogeneous vector bundles. Being a Gelfand pair is a necessary condition of being a commutative homogeneous vector bundle. For the case in which G/K is a nilmanifold having square-integrable representations, in a previous article we determined a big family of commutative homogeneous vector bundles. In this paper, we complete that classification.


Introduction
A homogeneous space G/K is called commutative, and the pair (G, K) is called a Gelfand pair, when G is a locally compact group, K is a compact subgroup of G, and the convolution algebra L 1 0 (G) is commutative. Here L 1 0 (G) denotes the Banach algebra of L 1 functions on G satisfying f (kxk ′ ) = f (x) for x ∈ G and k, k ′ ∈ K, where the product is the usual convolution (f * h)(g) = G f (x)h(x −1 g)dx on G (where dx denotes the Haar measure on G).
By a nilmanifold we mean a differentiable manifold on which a nilpotent Lie group acts transitively. By a commutative nilmanifold we mean a commutative space G/K such that G is a Lie group and a closed nilpotent subgroup N of G acts transitively (cf. e.g. [Wo2]). In that notation, if G/K is simply connected then N acts simply transitively on G/K and G is the semidirect product K ⋉ N (cf. [12,Theorem 4.2] or [Wo1,Theorem 13.1.6]). It is shown in [BJR] that if (K ⋉ N, K) is a Gelfand pair then N must be abelian or two-step nilpotent.
Those definitions are associated to the commutative property of the algebra L 1 0 (G) of scalar-valued functions. For the vector-valued case we have analogous definitions.
Again, let G be a Lie group and K a compact subgroup of G. It is well known that all the homogeneous vector bundles over the homogeneous space G/K are described by taking finite dimensional representations (τ, W τ ) of K (cf. [Wa,Section 5.2]). Indeed, if we consider the equivalence relation over G × W τ given by (gk, w) ∼ (g, τ (k)w) for g ∈ G, w ∈ W τ and k ∈ K, the space of equivalence classes E τ has an structure of homogeneous vector bundle over G/K. Moreover, each homogeneous vector bundle over G/K is isomorphic to E τ for some representation (τ, W τ ) of K of finite dimension.
The space of compactly supported smooth sections of the homogeneous vector bundle E τ is naturally identified with the space of functions u on D(G, W τ ) such that u(xk) = τ (k) −1 u(x) for k ∈ K and x ∈ G. It follows from the Schwartz kernel theorem that every linear operator T , continuous with respect to the standard topologies, mapping D-sections of E τ into D ′ -sections of E τ and commuting with the action of G on E τ , can be represented in a unique way as a convolution operator where F is a distribution, F ∈ D ′ (G, End(W τ )), satisfying In particular, operators T = T F as in (1) with F a function in L 1 (G, End(W τ )) satisfying (2) can be composed with each other and We call L 1 τ,τ (G, End(W τ )) the convolution algebra of integrable matrix-valued functions with property (2).
Let (τ, W τ ) be an irreducible unitary representation of K. Let K denote the set of equivalence classes of irreducible unitary representations of the group K. The homogeneous vector bundle E τ is called commutative, and the triple (G, K, τ ) is also called commutative, when the algebra L 1 τ,τ (G, End(W τ )) is commutative. In particular, (G, K) is a Gelfand pair when (G, K, τ ) is a commutative triple with τ the trivial representation of K. Here L 1 τ,τ (G, End(W τ )) = L 1 0 (G). It is shown in [RS] that if G/K is connected and if there exists τ ∈ K such that (G, K, τ ) is a commutative triple then (G, K) is a Gelfand pair. Therefore, in most cases, being a Gelfand pair is a necessary condition to give rise to commutative triples. When τ is a character of K and the triple (G, K, τ ) is commutative, these cases are also known as twisted Gelfand pair with respect to the character τ . Finally, we say that (G, K) is a strong Gelfand pair if (G, K, τ ) is commutative for every τ ∈ K.
In the case where L 1 τ,τ (G, End(W τ )) is a commutative algebra, the spherical analysis consists of the computation of the continuous characters of such convolution algebra and this gives rise to a kind of simultaneous "diagonalization" of all the operators T F (cf. e.g. [DL, Ko]).
From now on we concentrate on homogeneous vector bundles associated to nilmanifolds.
The classification of Gelfand pairs (K⋉N, K) with N nilpotent was completed by E. Vinberg. In the notable article [V] are exhibited all the Gelfand pairs (K ⋉N, K) that are irreducible and maximal. On the one hand, the irreducibility means that the center z of the Lie algebra n of N must be z = [n, n] and K acts irreducibly on n/z. On the other hand, the maximality implies that the pair (K ⋉ N, K) does not have non-trivial central reductions. This means that z does not have K-invariant subspaces s such that ifs denotes the orthogonal complement of s on z andÑ is the simply connected Lie group with Lie algebrañ :=s ⊕ (n/z) ≃ n/s then (K ⋉Ñ , K) is a Gelfand pair (cf. [Wo1,Section 13.4A,p. 320]).
Firstly, J. Lauret constructs a family of Gelfand pairs (K ⋉ N, K) considering on N a Riemannian structure (cf. [L1]). Here K ⋉ N is the group of isometries of N . When N is particularly the Heisenberg group or the euclidean space R n , the commutative triples were determined in [RS]. For the corresponding (matrix) spherical analysis see that article and also [DL]. When additionally N has square integrable representations, all the commutative triples that come from these Gelfand pairs were determined in the recent article [DS].
This article has the aim of completing the classification of commutative homogeneous vector bundles associated to nilmanifolds, that is, commutative triples of the form (K ⋉ N, K, τ ) with N a nilpotent Lie group. We will analyse which commutative triples come from the Gelfand pairs in [V] that are not included in the list given in [L1].

Preliminaries
We recall that since we consider N a two-step nilpotent Lie group, its Lie algebra splits, as a vector space, as n = z ⊕ V , where V is an orthogonal complement of the center z and [V, V ] ⊂ n. The group N acts naturally on n by the adjoint action Ad. Also, N acts on n * , the real dual space of n, by the dual or contragredient representation of the adjoint representation Ad * (n)λ := λ • Ad(n −1 ) for n ∈ N and λ ∈ n * . For each λ ∈ n * , let O(λ) := {Ad * (n)λ | n ∈ N } be its coadjoint orbit. From Kirillov's theory there is a correspondence between N and the set of coadjoint orbits. Let B λ be the skew symmetric bilinear form on n given by Let m ⊂ n be a maximal isotropic subalgebra in the sense that B λ (X, Y ) = 0 for all X, Y ∈ m and let M := exp(m). Defining on M the character χ λ (exp (Y ) One can see that ρ λ ∈ N is a square integrable representation if and only if that B λ is non-degenerate on V or, equivalently, if and only if the orbits are maximal (cf. [DS]). In this situation, consider the Heisenberg algebra n λ := RX λ ⊕ V with the bracket given by Since the character χ λ is trivial on z λ , the representation ρ λ acts trivially on z λ and defines an irreducible unitary representation of the corresponding Heisenberg group N λ . Now, let P (λ) be the square root of the determinant of (B λ ) |V ×V which is called the Pfaffian. This function P depends only on λ |z and so there is a homogeneous polynomial function (which we also denote P ) on z * such that P (λ) = P (λ |z ) (cf. [Wo1,p. 333]). According to [Wo1,Theorem 14.2.10] there is a correspondence between the coadjoint orbits O(λ) with P (λ |z ) = 0 and the square integrable representations.
Apart from that, let K be a compact subgroup of automorphisms of N and let K λ be the stabilizer of X λ with respect to the action of K on n. Note that since we always assume N simply connected, we make no distinction between automorphisms of N and n. It can be see that K λ is a subgroup of the symplectic group Sp (V, (B λ Let ω be the the metaplectic representation of K λ associated to the Heisenberg group N λ . That is, (For more details see [DS]).
For the following theorem see [DS,Theorem 3] and [RS,Theorem 6.1]. An important fact in the proof is that the classes in N of square integrable representations have full Plancherel measure. Theorem 1. Let N be a connected and simply connected real two-step nilpotent Lie group which has a square integrable representation. Let K be a compact subgroup of orthogonal automorphisms of N and let (τ, W τ ) ∈ K. Then (K ⋉ N, K, τ ) is a commutative triple if and only if (K λ ⋉N λ , K λ , τ |K λ ) is a commutative triple for every square integrable representation ρ λ ∈ N , where τ |K λ denotes the restriction of τ to K λ . Also, (K λ ⋉ N λ , K λ , τ |K λ ) is a commutative triple if and only if ω ⊗ (τ |K λ ) is multiplicity free.
The previous result will mark the way of our proofs. It allows to make a reduction to Heisenberg groups simplifying our problem from two-step nilpotent Lie groups in general to Heisenberg groups. But it is extremely important the condition on N of having square integrable representations.
There are few Gelfand pairs (K ⋉ N, K) in the list of E. Vinberg such that the nilpotent group is not of the form given by J. Lauret. Specifically, they correspond to the items 3, 5, 11, 12, 20 and 26 of table 3 of [V].
In the items 3 and 26 of [V] the group N does not have square integrable representations. We will skip them from our study. In the following paragraphs we will develop the analysis of the triples derived from the remaining Gelfand pairs.
-Case A: We will develop item 5 of table 3 of [V]. Here we have n = Λ 2 (C 2n ) ⊕ iR ⊕ C 2n , where Λ 2 (C 2n ) denotes the space of antisymmetric bilinear forms on C 2n over the complex field, and K = U(2n).
-Case B: We will study item 12 of table 3 of [V]. Denoting by H the quaternions, we have n = [H 0 (H n ) ⊕ Im(H)] ⊕ H n , where H 0 (H n ) denotes the space of hermitian n×n matrices over H of trace zero and Im(H) the imaginary quaternions, and K = S 1 × Sp(n), where S 1 is the one dimensional torus and Sp(n) the symplectic group. The only difference between item 12 and 11 of table 3 of [V] is that for item 11 the group K is smaller. -Case C: We will analyze item 20 of table 3 of [V]. Here we have n = R 7 ⊕R 8 and K = Spin (7).

Analysis of commutavive and non-commutative triples
Case A. The objects that we will describe in this paragraph can be found in item 5 of table 3 in [V] as well as in item 5 of table 13.4.1 in the book [Wo1]. Consider the two-step nilpotent Lie algebra where the composition is given by where we denote by u t the row vector obtained from the column vector u and by u * the conjugate row vector. The wedge product u ∧ v can also expressed as uv t − vu t . The unitary group K = U(2n) acts on z by k·(uv t −vu t , s) := (k(uv t −vu t )k t , s) ∀k ∈ K and ∀ uv t −vu t ∈ Λ 2 (C 2n ), s ∈ iR.
We consider in particular a linear functional λ in n * such that (when we restrict it to z) has as a representative the element X λ := (u ∧ v, 0) ∈ z with u ∧ v nondegenerate. Let ρ λ denote the irreducible representation class associated to it. From the formula of the Pfaffian given in [Wo1,, P (λ) is a positive multiple of the determinant of the matrix uv t − vu t . According to our election, it is invertible. Therefore ρ λ is square integrable and we are allowed to apply Theorem 1. In this case, the subgroup K λ coincides with the symplectic group Sp(n).
Let τ ∈ K non-trivial. Then there is a non-trivial η ∈ Sp(n) appearing in the restriction of τ to K λ . Now we introduce some notation. Let δ i,j denote the Kronecker delta. Writing H i := δ i,i − δ n+i,n+i , we have that the Cartan subalgebra h of sp(n) is a complex vector space generated by {H 1 , ..., H n }. Let {L 1 , ..., L n } be its dual basis in the dual space h * , so L i , H = h i for all H ∈ h. From the theorem of the highest weight, every irreducible representation of sp(n) is in correspondence with a nonnegative integer linear combination of the fundamental weights. Hence η ∈ Sp(n) can be parametrized in terms of the weights {L i } as (η 1 , ..., η n ) where η i ∈ Z ≥0 ∀i and η 1 ≥ η 2 ≥ ... ≥ η n . (For a reference see, for example, [FH, Ka].) We will denote the representation η ∈ Sp(n) by η (η1,...,ηn) to emphasize that the representation η is in correspondence with the tuple (η 1 , ..., η n ) that we call partition.
The metaplectic representation (ω, P(C 2n )) of Sp(n) decomposes (using the notation given in the previous sections) as where each η (j) corresponds to the partition on length one (j), for some nonnegative integer j.
The following fact is proved in [DS, Corollary 2] and will be useful to deduce our main results.
Lemma 2. Let η be an irreducible representation of Sp(n) then η appears in the decomposition into irreducible factors of η ⊗ η (2) .
From the above lemma, the representation η appears in the decomposition of the factors η⊗η (0) and η⊗η (2) . Then, it appears with multiplicity in the decomposition into irreducible factors of ω ⊗ τ |K λ . Therefore, by Theorem 1, we have the following result.
Theorem 3. The triple (K ⋉ N, K, τ ), where N is the simply connected nilpotent Lie group with Lie algebra Λ 2 (C 2n ) ⊕ iR ⊕ C 2n and K = U(2n), is commutative if and only if τ is the trivial representation.
Case B. The objects that we will describe in this paragraph can be found in item 12 of table 3 in [V] as well as in item 9 of table 13.4.1 in the book [Wo1].
Consider the two-step nilpotent Lie algebra Let τ = η ⊗ χ r where η ∈ Sp(n) and χ r a character of S 1 . First we consider the case where the factor in Sp(n) is trivial, that is τ = χ r . Since the convolution algebra L 1 τ,τ (K ⋉ N, End(W τ )) is naturally identified with the space of End(W τ )-valued integrable functions on N such that F (k · x) = τ (k)F (x)τ (k) −1 for all k ∈ K and x ∈ N (cf. e.g. [DL, RS]). In this situation, it totally coincides with the convolution algebra of K-invariant integrable scalars functions on N , which is commutative since we have a Gelfand pair. Therefore we have commutative triples. Now let η ∈ Sp(n) non-trivial.
We consider the linear functional λ with representative X λ := (0, q) ∈ z. Let ρ λ denote the irreducible representation class associated to it. From the formula of the Pfaffian given [Wo1,p. 340], P (λ) is a positive multiple of |q| 2n . Then for all imaginary non-zero quaternions q, the associated representation ρ λ is square integrable. The group K λ is easily calculated: If the quaternion q belongs to iR, K λ coincides with K (since q commutes with every complex); for the other cases K λ is Sp(n). We consider for example q = j ∈ Im(H) in order to fix K λ = Sp(n). Note that τ |K λ = η.
The metaplectic representation (ω, P(C 2n )) of K λ = Sp(n) decomposes as Therefore ω ⊗ η is not multiplicity free: η appears in the the factors η ⊗ η (0) and in η ⊗ η (2) by Lemma 2. Then, by Theorem 1, in these cases the triples are not commutative. To conclude this case we want to note that the analysis of the item 11 of table 3 of [V] is the same (or simpler) since the subgroup K is only Sp(n).
Case C. The objects that we will describe in this paragraph can be found in item 20 of table 3 in [V] as well as in item 2 of table 13.4.1 in the book [Wo1].
In this case we study an H-type group ( [Ka]). Consider the two-step nilpotent Lie algebra n = z ⊕ V where V is R 8 or the octonions O and the center z is R 7 or the imaginary octonions Im (O) with Lie bracket on V characterized by: [u, v], z z := J(z)u, v V , where (J, V ) is a real representation of z given by the product on the octonions J(z)v := zv for z ∈ Im (O) and v ∈ O.
Let K be the maximal connected group of orthogonal automorphisms of N . Precisely, K = Spin(7). (We mention that in general, for an H-type group, it group of automorphisms was determined by L. Saal in [S].) Let λ be a linear functional with representative X ∈ z. From [Wo1,p. 340], the Pfaffian P (λ) is not null almost everywhere. In this case, K λ is isomorphic to the spin group Spin(6). Therefore we will work with the Lie algebra so(6) .
Let τ be a non-trivial irreducible unitary representation of Spin(7). When we restrict τ to Spin(6) it decomposes as a sum of irreducible representations and we can pick one non-trivial factor. We will denote it asη ∈ Spin(6) and we will identify it with its derived representation and also we will view it as a representation of so(6).
Apart from that, it is easy to derive the decomposition of the metaplectic representation since Spin(6) is isomorphic to SU(4) and it is well known that its action on P(C 4 ) decomposes into ⊕ j∈Z ≥0 P j (C 4 ). It can be identified with whereη (j) corresponds to the partition of length one (j) of so(6). Here we decided to use an analogous notation to that of the representations of sp(n). As a consequence of the highest weight theorem, we can associate each irreducible representation σ of so(2m) with an n-tuple of integers (σ 1 , ..., σ m ) satisfying the condition σ 1 ≥ σ 2 ≥ ... ≥ σ m−1 ≥ |σ m | (cf. [FH, O]). The following fact can be found in [O,Theorem 3.2 and Remark 3.5].
Theorem 6. The triple (K ⋉ N, K, τ ), where N is the simply connected nilpotent Lie group with Lie algebra Im(O) ⊕ O and K = Spin (7), is commutative if and only if τ | Spin(6) is associated to a partition of the form (r, r, r) for r ∈ Z ≥0 .

Conclusions
At this point we sum up the following. Theorem 3 does not provide non-trivial commutative triples. Theorem 4 gives rise to a commutative triple only if τ is a character of S 1 . These cases are twisted Gelfand pairs. Finally, Theorem 6 gives a family of commutative triples. This family is similar to the case of the H-type group listed by J. Lauret in [L1]. There the nilpotent Lie algebra n is Im(H) ⊕ H n where Im(H) is its center z and the Lie bracket on H n is given, analogously as in case C, by the real representation of z, J(z)(v) := (zv 1 , ..., zv n ) for v = (v 1 , ..., v n ) ∈ H. Its (maximal connected) group of automorphisms is K = SU(2) × Sp(n). By [DS,Proposition 1], in that case we obtain commutative triples if and only if τ ∈ SU(2) or τ ∈ Sp(n) corresponding to a partition of the form (a, a, ..., a) of length at most n for a non-negative integer a.
In conclusion, denoting by H n the Heisenberg group H n = C n × R, if we exclude from the analysis the Gelfand pairs of the form (H n ⋉ K, K) where K is a proper subgroup of U(n), all the commutative triples of the form (K ⋉ N, K, τ ), where N is a simply connected nilpotent Lie group having square integrable representations and K is a compact subgroup of G, are the following: (1) ([S 1 × Sp(n)]⋉N, S 1 × Sp(n), τ ), where N is the simply connected nilpotent Lie group with Lie algebra [H 0 (H n ) ⊕ Im(H)] ⊕ H n , is commutative for τ ∈ S 1 . It corresponds to the preceding case C. It is a twisted Gelfand pair.
Where inítems 2, 3, 5, 6 and 7 the nilpotent Lie group N = N (g, V ) is endowed with a left-invariant Riemannian metric determined by an inner product ·, · on its Lie algebra n described as follows. Let (π, V ) be a faithful real representation of a compact Lie algebra g. We consider inner products ·, · g on g and ·, · V on V such that ·, · g is ad(g)-invariant and ·, · V is π(g)-invariant. Let n := g ⊕ V be the two-step nilpotent Lie algebra with center g and Lie bracket defined on V by [u, v], X g := π(X)u, v V for all u, v ∈ V , X ∈ g. These inner products define an inner product ·, · on n satisfying ·, · |g×g = ·, · g , ·, · |V ×V = ·, · V and g, V = 0. Let N (g, V ) be the two-step connected simply connected Lie group with Lie algebra n. Specifically, in item 2, C n denotes the standard representation of su(n), in item 3, C n denotes the standard representation of u(n) regarded as a real representation, in item 5, C 2 denotes the standard representation of su(2) regarded as a real representation, in item 6, (C 2 ) k ⊕ (C 2 ) n is an orthogonal sum, the center of u(2) acts non-trivially only on (C 2 ) k , (C 2 ) n denotes the representation of su(2) stated in the first item, u(2) acts component-wise on (C 2 ) k in the standard way regarded as a real representation and τ ∈ U(k) and in item 7, g is acting on V as follows: For each 1 ≤ i ≤ β + α, c has a unique subspace c i acting non-trivially on only the i-th component of V and the dimension of c i is 1. For 1 ≤ i ≤ β, su(m i )⊕c i (which is isomorphic to u(m i )) acts non-trivially only on C mi by standard. For β + 1 < i ≤ β + α, su(2) ⊕ c i acts non-trivially only on C 2ki+2ni as in the above case. For more details see [DS,L1,L2].