A stationary Markovian decision model is considered with general state and action spaces where the transition probabilities are weakened to be bounded transition measures (this is useful for many applications). New and improved bounds are given for the optimal value of stationary problems with a large planning horizon if either only a few steps of iteration are carried out or, in addition, a solution of the infinite-stage problem is known. Similar estimates are obtained for the quality of policies which are composed of nearly optimal decisions from the first few steps or from the infinite-stage solution.

Given a finite number of different experiments with unknown probabilities p1, p2, ···, pk of success, the multi-armed bandit problem is concerned with maximising the expected number of successes in a sequence of trials. There are many policies which ensure that the proportion of successes converges to p = max (p1, p2, ···, pk), in the long run. This property is established for a class of decision procedures which rely on randomisation, at each stage, in selecting the experiment for the next trial. Further, it is suggested that some of these procedures might perform well over any finite sequence of trials.

The paper looks at the asymptotic properties of the finite Walsh–Fourier transform applied to a discrete-time stationary time series, and shows that in many ways we have analogous results to those obtained when using the finite trigonometric Fourier transform.

In this paper we arrive at a series of bounds for the availability and unavailability in the time interval I = [tA, tB] ⊂ [0, ∞), for a coherent system of maintained, interdependent components. These generalize the minimal cut lower bound for the availability in [0, t] given in Esary and Proschan (1970) and also most bounds for the reliability at time t given in Bodin (1970) and Barlow and Proschan (1975). In the latter special case also some new improved bounds are given. The bounds arrived at are of great interest when trying to predict the performance process of the system. In particular, Lewis et al. (1978) have revealed the great need for adequate tools to treat the dependence between the random variables of interest when considering the safety of nuclear reactors.

Satyanarayana and Prabhakar (1978) give a rapid algorithm for computing exact system reliability at time t. This can also be used in cases where some simpler assumptions on the dependence between the components are made. It seems, however, impossible to extend their approach to obtain exact results for the cases treated in the present paper.

We discuss a single-server queue whose input is the versatile Markovian point process recently introduced by Neuts [22] herein to be called the N-process. Special cases of the N-process discussed earlier in the literature include a number of complex models such as the Markov-modulated Poisson process, the superposition of a Poisson process and a phase-type renewal process, etc. This queueing model has great appeal in its applicability to real world situations especially such as those involving inhibition or stimulation of arrivals by certain renewals. The paper presents formulas in forms which are computationally tractable and provides a unified treatment of many models which were discussed earlier by several authors and which turn out to be special cases. Among the topics discussed are busy-period characteristics, queue-length distributions, moments of the queue length and virtual waiting time. We draw particular attention to our generalization of the Pollaczek–Khinchin formula for the Laplace–Stieltjes transform of the virtual waiting time of the M/G/1 queue to the present model and the resulting Volterra system of integral equations. The analysis presented here serves as an example of the power of Markov renewal theory.