We consider the class of bandit problems in which each of the n ≧ 2 independent arms generates rewards according to one of the same two reward distributions, and discounting is geometric over an infinite horizon. We show that the dynamic allocation index of Gittins and Jones (1974) in this context is strictly increasing in the probability that an arm is the better of the two distributions. It follows as an immediate consequence that myopic strategies are the uniquely optimal strategies in this class of bandit problems, regardless of the value of the discount parameter or the shape of the reward distributions. Some implications of this result for bandits with Bernoulli reward distributions are given.

Families of alternative bandit processes have been used as models for problems in a variety of areas. Optimal strategies for these decision processes are determined by dynamic allocation indices. These indices are here shown to play an important role in the evaluation of suboptimal strategies.

Stoppable families of alternative bandit processes are decision processes with the property that at each decision epoch the choice is between allocating service to one of the constituent bandit processes or stopping and deciding in favour of one of them. The problem is considered of finding optimal (or good suboptimal) strategies for such processes. The theory for non-stoppable families leads us to study the performance of a simple strategy. This is shown to be optimal under certain conditions. These conditions are discussed and an example relating to research planning is given.

The optimal strategy for a class of non-Markov decision processes is characterised and has the property that changes of action may occur between successive transitions of the process. Results are given which enable the optimal strategy to be computed iteratively.

The D.A.I. theorem of Gittins and Jones has proved a powerful tool in solving sequential statistical problems. A generalisation of this theorem is presented. This generalisation enables us to solve certain stochastic scheduling problems where the items or jobs to be scheduled have random times to completion, the random times having distributions dependent upon parameters to which prior distributions are allocated. Such problems are of interest in many areas where scheduling is important.

The problem of scheduling items for service with random service times is formulated as an optimal control problem. Pontryagin's maximum principle is used to determine the optimal schedule in certain cases.