We consider a two-stage service system with two types of servers, namely subordinates who perform the first-stage service and supervisors who have their own responsibilities in addition to collaborating with the subordinates on the second-stage service. Rewards are earned when first- or second-stage service is completed and when supervisors finish one of their own responsibilities. Costs are incurred when impatient customers abandon without completing the second-stage service. Our problem is to determine how the supervisors should distribute their time between their joint work with the subordinates and their own responsibilities. Under the assumptions that service times at both stages are exponentially distributed and that the customers waiting for second-stage service abandon after an exponential amount of time, we prove that one of two policies will maximize the long-run average profit. Namely, it is optimal for supervisors to start collaborating with subordinates either when subordinates can no longer serve new customers or as soon as there is a customer ready for second-stage service. Furthermore, we show that the optimality condition is a simple threshold on the system parameters. We conclude by proving that pooling supervisors (and their associated subordinates) improves system performance, but with limited returns as more supervisors are pooled.

This paper considers optimal admission and routing control in multi-class service systems in which customers can either receive quality regular service which is subject to congestion or can receive congestion-free but less desirable service at an alternative service station, which we call the self-service station. We formulate the problem within the Markov decision process framework and focus on characterizing the structure of dynamic optimal policies which maximize the expected long-run rewards. For this, value function and sample path arguments are used. The congestion sensitivity of customers is modeled with class-independent holding costs at the regular service station. The results show how the admission rewards of customer classes affect their priorities at the regular and self-service stations. We explore that the priority for regular service may not only depend on regular service admission rewards of classes but also on the difference between regular and self-service admission rewards. We show that optimal policies have monotonicity properties, regarding the optimal decisions of individual customer classes such that they divide the state space into three connected regions per class.

We consider the problem of routing and admission control in a loss system featuring two classes of arriving jobs (high-priority and low-priority jobs) and two types of servers, in which decision-making for high-priority jobs is forced, and rewards influence the desirability of each of the four possible routing decisions. We seek a policy that maximizes expected long-run reward, under both the discounted reward and long-run average reward criteria, and formulate the problem as a Markov decision process. When the reward structure favors high-priority jobs, we demonstrate that there exists an optimal monotone switching curve policy with slope of at least −1. When the reward structure favors low-priority jobs, we demonstrate that the value function, in general, lacks structure, which complicates the search for structure in optimal policies. However, we identify conditions under which optimal policies can be characterized in greater detail. We also examine the performance of heuristic policies in a brief numerical study.

We propose a discrete state-space model for storage of urban stormwater in two connected dams using an optimal pump-to-fill policy to transfer water from the capture dam to the holding dam. We assume stochastic supply to the capture dam and independent stochastic demand from the holding dam. We find new analytic formulae to calculate steady-state probabilities for the contents of each dam and thereby enable operators to better understand system behaviour. We illustrate our methods by considering some particular examples and discuss extension of our analysis to a series of three connected dams.

This paper argues that most of the problems that actuaries have to deal with in the context of non-life insurance can be usefully cast in the framework of computational intelligence (a.k.a. artificial intelligence), the discipline that studies the design of agents which exhibit intelligent behaviour. Finding an adequate framework for actuarial problems has more than a simply theoretical interest: it also allows a technological transfer from the computational intelligence discipline to general insurance, wherever techniques have been developed for problems which are common to both contexts. This has already happened in the past (neural networks, clustering, data mining have all found applications to general insurance) but not in a systematic way. One of the objectives of this paper will therefore be to introduce some useful techniques such as sparsity-based regularisation and dynamic decision networks that are not yet known to the wider actuarial community.

Whilst in the first part of this paper we dealt mainly with data-driven loss modelling under the assumption that all the data were accurate and fully relevant to the exercise, in this second part of the paper we explore how to deal with uncertain knowledge, whether this uncertainty comes from the fact that the data are not fully reliable (e.g. they are estimates) or from the fact that the knowledge is “soft” (e.g. expert beliefs) or not fully relevant (e.g. market information on a given risk). Most importantly, we will deal with the problem of making pricing, reserving and capital decisions under uncertainty. It will be concluded that a Bayesian framework is the most adequate for dealing with uncertainty, and we will present a number of computational intelligence techniques to do this in practice.

We consider a Markov decision process for an MX/M/1 queue that iscontrolled by batches of negative customers. More specifically, we deriveconditions that imply threshold-type optimal policies, under either thetotal discounted cost criterion or the average cost criterion. Theperformance analysis of the model when it operates under a giventhreshold-type policy is also studied. We prove a stability condition and acomplete stochastic comparison characterization for models operating underdifferent thresholds. Exact and asymptotic results concerning thecomputation of the stationary distribution of the model are also derived.

In this paper we study the assignment of packets to M parallel heterogeneous servers with no buffers. The controller does not have knowledge on the state of the servers and bases all decisions on the number of time slots ago that packets have been sent to the different servers. The objective of the controller is to minimize the expected average cost. We consider a general stationary arrival process, with no independence assumptions. We show that the problem can be transformed into a Markov Decision Process (MDP). We further show under quite general conditions on cost functions that only a finite number of states have to be considered in this MDP, which then implies the optimality of periodic policies. For the case of two servers we obtain a more detailed structure of the cost and optimal policies. In particular we show that the average cost function is multimodular, and we obtain expressions for the cost for MMPP and MAP processes. Finally we present an application to optimal robot scheduling for Web search engines.

We consider two parallel M/M/1 queues. The server at one of the queuesis subject to intermittent breakdowns. By the theory of dynamic programming, we determine a threshold optimal policy which consists to transfer, when it is necessary, the customers that arrive at the first queue towards the second queue in order to minimize an instantaneous costdepending of the two queue lengths.

Given a sequence of random variables (rewards), the Haviv–Puterman differential equation relates the expected infinite-horizon λ-discounted reward and the expected total reward up to a random time that is determined by an independent negative binomial random variable with parameters 2 and λ. This paper provides an interpretation of this proven, but previously unexplained, result. Furthermore, the interpretation is formalized into a new proof, which then yields new results for the general case where the rewards are accumulated up to a time determined by an independent negative binomial random variable with parameters k and λ.

The primary objective in the present paper is to gain fundamental understanding of the performance achievable in ATM networks as a function of the various system characteristics. We derive limit theorems that characterize the achievable performance in terms of the offered traffic, the admissible region, and the revenue measure. The insights obtained allow for substantial simplifications in the design of real-time connection admission control algorithms. In particular, we describe how the boundaries of admissible regions with convex complements may be linearized - thus reducing the admissible region - so as to obtain a convenient loss network representation. The asymptotic results for the achievable performance suggest that the potential reduction in revenue is immaterial in high-capacity networks. Numerical experiments confirm that the actual reduction is typically negligible, even in networks of moderate capacity.

This paper studies an admission control M/M/1 queueing system. It shows that the only gain (average) optimal stationary policies with gain and bias which satisfy the optimality equation are of control limit type, that there are at most two and, if there are two, they occur consecutively. Conditions are provided which ensure the existence of two gain optimal control limit policies and are illustrated with an example. The main result is that bias optimality distinguishes these two gain optimal policies and that the larger of the two control limits is the unique bias optimal stationary policy. Consequently it is also Blackwell optimal. This result is established by appealing to the third optimality equation of the Markov decision process and some observations concerning the structure of solutions of the second optimality equation.

We consider a problem similar in many respects to a finite horizon Markov decision process, except that the reward to the individual is a strictly concave functional of the distribution of the state of the individual at final time T. Reward structures such as these are of interest to biologists studying the fitness of different strategies in a fluctuating environment. The problem fails to satisfy the usual optimality equation and cannot be solved directly by dynamic programming. We establish equations characterising the optimal final distribution and an optimal policy π*. We show that in general π* will be a Markov randomised policy (or equivalently a mixture of Markov deterministic policies) and we develop an iterative, policy improvement based algorithm which converges to π*. We also consider an infinite population version of the problem, and show that the population cannot do better using a coordinated policy than by each individual independently following the individual optimal policy π*.

This paper presents a simple model of the decision problems which face many animals in the wild. Animals have an upper limit to the amount of energy reserves which can be stored as body fat. They constantly use energy and find food as a stochastic process. Death occurs if energy reserves fall to zero or if the animal is taken by a predator. Typically animals have a range of behavioural options which differ in the distribution of food gained and in the associated predation risk. There is often a trade-off between starvation and predation in that animals have to expose themselves to higher predation risks in order to gain more food.

The above situation is modelled as a finite-state, finite-time-horizon Markov decision problem. The policy which maximises long-term survival probability is characterised. As special cases two trade-offs are analysed. It is shown that an animal should take fewer risks in terms of predation as its reserves increase, and that an animal should reduce the variability of its food supply at the expense of its mean gain as its reserves increase.

In this paper we consider a queueing system having n exponential servers, each with its own queue and service rate. Customers arrive according to a Poisson process with rate λ, and upon arrival each customer must be routed to some server's queue. No jockeying amongst the queues is allowed and each server serves its queue according to a first-come-first-served discipline.

Each server i, 1 ≦ i ≦ n, provides service with a state-dependent rate μ(i)(k), k = 0, 1, …. In addition, at every queue i, there is a deterministic holding cost which occurs at rate h(i)(k) while there are k customers at the queue.

An admissible routing policy is a policy that assigns each arriving customer to one of the queues. A decision at time t may be randomized and dependent on the queue lengths and decisions till time t. An optimal routing policy is an admissible policy that minimizes the long-run average holding cost.

In this study, we bound the optimal cost from below, by considering an ideal system, where each server optimally selects a given proportion of customers, irrespective of other servers' selections. From this ideal system we construct a class of admissible routing policies, the overflow routing class, that approximates the ideal situation for each server. Finally, we evaluate the policies and compare them to the lower bound.

We study the effect of perturbations in the data of a discrete-time Markov reward process on the finite-horizon total expected reward, the infinite-horizon expected discounted and average reward and the total expected reward up to a first-passage time. Bounds for the absolute errors of these reward functions are obtained. The results are illustrated for a finite as well as infinite queueing systems (M/M/1/S and ). Extensions to Markov decision processes and other settings are discussed.

We study the problem of optimal customer admission to multiserver queues. These queues are assumed to live in an extraneous environment which changes in a semi-Markovian way. Arrivals, service mechanism and random reward/cost structure may all depend on these surroundings. Included as special cases are SM/M/c queues, in particular G/M/c queues, in a random environment. By a direct inductive approach we establish optimality of a generalized control-limit rule depending on the actual environment. Particular emphasis is laid on different applications that show the versatility of the proposed setup.

A general form of Green's theorem is used to derive relations between expected costs for an optimisation problem and the distribution of state variable. A characterisation of the optimal stopping set and an alternative proof of the Howard improvement lemma emerge as non-trivial consequences.

A collection of jobs is to be processed by a single machine. The amount of processing required by each job is a random variable with a known probability distribution. The jobs must be processed in a manner which is consistent with a precedence relation but the machine is free to switch from one job to another at any time; such switches are costly, however. This paper discusses conditions under which there is an optimal strategy for allocating the machine to the jobs which is given by a fixed permutation of the jobs indicating in which order they should be processed. When this is so, existing algorithms may be helpful in giving the best job ordering.