Motivated by the gene tree/species tree problem from statistical phylogenetics, we extend the class of Markov branching trees to a parametric family of distributions on fragmentation trees that satisfies a generalized Markov branching property. The main theorems establish important statistical properties of this model, specifically necessary and sufficient conditions under which a family of trees can be constructed consistently as sample size grows. We also consider the question of attaching random edge lengths to these trees.

In this paper we consider the beta(2 − α, α)-coalescents with 1 < α < 2 and study the moments of external branches, in particular, the total external branch length of an initial sample of n individuals. For this class of coalescents, it has been proved that nα-1T(n) →DT, where T(n) is the length of an external branch chosen at random and T is a known nonnegative random variable. For beta(2 − α, α)-coalescents with 1 < α < 2, we obtain limn→+∞n3α-5 𝔼(Lext(n) − n2-α𝔼T)2 = ((α − 1)Γ(α + 1))2Γ(4 − α) / ((3 − α)Γ(4 − 2α)).

In this paper we study a weak law of large numbers for the total internal length of the Bolthausen-Sznitman coalescent, thereby obtaining the weak limit law of the centered and rescaled total external length; this extends results obtained in Dhersin and Möhle (2013). An application to population genetics dealing with the total number of mutations in the genealogical tree is also given.

Considering a random binary tree with n labelled leaves, we use a pruning procedure on this tree in order to construct a β(3/2,1/2)-coalescent process. We also use the continuous analogue of this construction, i.e. a pruning procedure on Aldous's continuum random tree, to construct a continuous state space process that has the same structure as the β-coalescent process up to some time change. These two constructions enable us to obtain results on the coalescent process, such as the asymptotics on the number of coalescent events or the law of the blocks involved in the last coalescent event.

The diffusion-generator approximation technique developed by De Iorio and Griffiths (2004a) is a very useful method of constructing importance-sampling proposal distributions. Being based on general mathematical principles, the method can be applied to various models in population genetics. In this paper we extend the technique to the neutral coalescent model with recombination, thus obtaining novel sampling distributions for the two-locus model. We consider the case with subdivided population structure, as well as the classic case with only a single population. In the latter case we also consider the importance-sampling proposal distributions suggested by Fearnhead and Donnelly (2001), and show that their two-locus distributions generally differ from ours. In the case of the infinitely-many-alleles model, our approximate sampling distributions are shown to be generally closer to the true distributions than are Fearnhead and Donnelly's.

We present recursions for the total number, Sn, of mutations in a sample of n individuals, when the underlying genealogical tree of the sample is modelled by a coalescent process with mutation rate r>0. The coalescent is allowed to have simultaneous multiple collisions of ancestral lineages, which corresponds to the existence of large families in the underlying population model. For the subclass of Λ-coalescent processes allowing for multiple collisions, such that the measure Λ(dx)/x is finite, we prove that Sn/(nr) converges in distribution to a limiting variable, S, characterized via an exponential integral of a certain subordinator. When the measure Λ(dx)/x2 is finite, the distribution of S coincides with the stationary distribution of an autoregressive process of order 1 and is uniquely determined via a stochastic fixed-point equation of the form with specific independent random coefficients A and B. Examples are presented in which explicit representations for (the density of) S are available. We conjecture that Sn/E(Sn)→1 in probability if the measure Λ(dx)/x is infinite.

We study identities for the distribution of the number of edges at time t back (i.e. measured backwards) in a coalescent tree whose subtrees have no mutations. This distribution is important in the infinitely-many-alleles model of mutation, where every mutation is unique. The model includes, as a special case, the number of edges in a coalescent tree at time t back when mutation is ignored. The identities take the form of expected values of functions of Zt=eiXt, where Xt is distributed as standard Brownian motion. Associated identities are also found for the distributions of the time to the most recent common ancestor, the time until loss of ancestral lines by coalescence or mutation, and the age of a mutation. Hypergeometric functions play an important role in the identities. The identities are of mathematical interest, as well as potentially being formulae to use for numerical integration or simulation to compute distributions that are usually expressed as alternating-sign series expansions, which are difficult to compute.

Stephens and Donnelly (2000) constructed an efficient sequential importance-sampling proposal distribution on coalescent histories of a sample of genes for computing the likelihood of a type configuration of genes in the sample. In the current paper a characterization of their importance-sampling proposal distribution is given in terms of the diffusion-process generator describing the distribution of the population gene frequencies. This characterization leads to a new technique for constructing importance-sampling algorithms in a much more general framework when the distribution of population gene frequencies follows a diffusion process, by approximating the generator of the process.

De Iorio and Griffiths (2004) developed a new method of constructing sequential importance-sampling proposal distributions on coalescent histories of a sample of genes for computing the likelihood of a type configuration of genes in the sample by simulation. The method is based on approximating the diffusion-process generator describing the distribution of population gene frequencies, leading to an approximate sample distribution and finally to importance-sampling proposal distributions. This paper applies that method to construct an importance-sampling algorithm for computing the likelihood of samples of genes in subdivided population models. The importance-sampling technique of Stephens and Donnelly (2000) is thus extended to models with a Markov chain mutation mechanism between gene types and migration of genes between subpopulations. An algorithm for computing the likelihood of a sample configuration of genes from a subdivided population in an infinitely-many-alleles model of mutation is derived, extending Ewens's (1972) sampling formula in a single population. Likelihood calculation and ancestral inference in gene trees constructed from DNA sequences under the infinitely-many-sites model are also studied. The Griffiths-Tavaré method of likelihood calculation in gene trees of Bahlo and Griffiths (2000) is improved for subdivided populations.