We consider the assignment of servers to two phases of service in a two-stage tandem queueing system when customers can abandon from each stage of service. New jobs arrive at both stations. Jobs arriving at station 1 may go through both phases of service and jobs arriving at station 2 may go through only one phase of service. Stage-dependent holding and lump-sum abandonment costs are incurred. Continuous-time Markov decision process formulations are developed that minimize discounted expected and long-run average costs. Because uniformization is not possible, we use the continuous-time framework and sample path arguments to analyze control policies. Our main results are conditions under which priority rules are optimal for the single-server model. We then propose and evaluate threshold policies for allocating one or more servers between the two stages in a numerical study. These policies prioritize a phase of service before “switching” to the other phase when total congestion exceeds a certain number. Results provide insight into how to adjust the switching rule to significantly reduce costs for specific input parameters as well as more general multi-server situations when neither preemption or abandonments are allowed during service and service and abandonment times are not exponential.

We propose and analyze a temporal concatenation heuristic for solving large-scale finite-horizon Markov decision processes (MDP), which divides the MDP into smaller sub-problems along the time horizon and generates an overall solution by simply concatenating the optimal solutions from these sub-problems. As a “black box” architecture, temporal concatenation works with a wide range of existing MDP algorithms. Our main results characterize the regret of temporal concatenation compared to the optimal solution. We provide upper bounds for general MDP instances, as well as a family of MDP instances in which the upper bounds are shown to be tight. Together, our results demonstrate temporal concatenation's potential of substantial speed-up at the expense of some performance degradation.

We consider an M/M/1 queue with a removable server that dynamically chooses its service rate from a set of finitely many rates. If the server is off, the system must warm up for a random, exponentially distributed amount of time, before it can begin processing jobs. We show under the average cost criterion, that work conserving policies are optimal. We then demonstrate the optimal policy can be characterized by a threshold for turning on the server and the optimal service rate increases monotonically with the number in system. Finally, we present some numerical experiments to provide insights into the practicality of having both a removable server and service rate control.

We consider the problem of routing and admission control in a loss system featuring two classes of arriving jobs (high-priority and low-priority jobs) and two types of servers, in which decision-making for high-priority jobs is forced, and rewards influence the desirability of each of the four possible routing decisions. We seek a policy that maximizes expected long-run reward, under both the discounted reward and long-run average reward criteria, and formulate the problem as a Markov decision process. When the reward structure favors high-priority jobs, we demonstrate that there exists an optimal monotone switching curve policy with slope of at least −1. When the reward structure favors low-priority jobs, we demonstrate that the value function, in general, lacks structure, which complicates the search for structure in optimal policies. However, we identify conditions under which optimal policies can be characterized in greater detail. We also examine the performance of heuristic policies in a brief numerical study.

From health care to maintenance shops, many systems must contend with allocating resources to customers or jobs whose initial service requirements or costs change when they wait too long. We present a new queueing model for this scenario and use a Markov decision process formulation to analyze assignment policies that minimize holding costs. We show that the classic cμ rule is generally not optimal when service or cost requirements can change. Even for a two-class customer model where a class 1 task becomes a class 2 task upon waiting, we show that additional orderings of the service rates are needed to ensure the optimality of simple priority rules. We then show that seemingly-intuitive switching curve structures are also not optimal in general. We study these scenarios and provide conditions under which they do hold. Lastly, we show that results from the two-class model do not extend to when there are n≥3 customer classes. More broadly, we find that simple priority rules are not optimal. We provide sufficient conditions under which a simple priority rule holds. In short, allowing service and/or cost requirements to change fundamentally changes the structure of the optimal policy for resource allocation in queueing systems.

Predicted crop yields and wind erosion rates from a multi-year/multi-crop growth simulation model provided input into a multi-period recursive QP model to evaluate erosion implications during the transition to dryland crop production on the Texas Southern High Plains. Three farm-program participation options were considered in this study. Participation in an extension of the current farm program resulted in an increase in net returns and wind erosion rates above nonparticipation. Imposition of a soil loss limit without consideration of a flexible base option can significantly reduce discounted present values. Increasing risk aversion across producers affects crop mix selection which can result in lower per acre wind erosion rates for this particular region.

Farm size and production costs are varied in a six state variable stochastic dynamic programming model that quantifies monthly hedging, storage, and cash cotton sale decisions for an Alabama cotton producer. State variables considered are: (1) cash cotton price; (2) basis level; (3) before-tax income level; (4) cotton holdings; (5) futures position; and (6) value of futures position. Results indicate that when farm size and production cost level differ, marketing decisions diverge the most for cash cotton sales at the end of the tax year and lower range of cash price (less than $.65/lb.), basis (less than -$.05/lb.), and before-tax income (less than $0.00) states.

Relying only on the classical Bahadur–Rao approximation for large deviations (LDs) of univariate sample means, we derive strong LD approximations for probabilities involving two sets of sample means. The main result concerns the exact asymptotics (as n→∞) of

$$ {\open P}\left({\max_{i\in\{1,\ldots,d_x\}}\bar X_{i,n} \les \min_{i\in\{1,\ldots,d_y\}}\bar Y_{i,n}}\right),$$

${\bar X}_{i,n}{\rm s}$

${\bar Y}_{i,n}{\rm s}$

${\open E}X \gt {\open E}Y$

In this paper, we study the almost sure convergence for sequences of asymptotically negative associated (ANA) random variables. As a result, we extend the classical Khintchine–Kolmogorov convergence theorem, Marcinkiewicz strong law of large numbers, and the three series theorem for sequences of independent random variables to sequences of ANA random variables without necessarily adding any extra conditions.

In many scheduling applications, minimizing delays is of high importance. One adverse effect of such delays is that the reward for completion of a job may decay over time. Indeed in healthcare settings, delays in access to care can result in worse outcomes, such as an increase in mortality risk. Motivated by managing hospital operations in disaster scenarios, as well as other applications in perishable inventory control and information services, we consider non-preemptive scheduling of jobs whose internal value decays over time. Because solving for the optimal scheduling policy is computationally intractable, we focus our attention on the performance of three intuitive heuristics: (1) a policy which maximizes the expected immediate reward, (2) a policy which maximizes the expected immediate reward rate, and (3) a policy which prioritizes jobs with imminent deadlines. We provide performance guarantees for all three policies and show that many of these performance bounds are tight. In addition, we provide numerical experiments and simulations to compare how the policies perform in a variety of scenarios. Our theoretical and numerical results allow us to establish rules-of-thumb for applying these heuristics in a variety of situations, including patient scheduling scenarios.

The knowledge gradient (KG) policy was originally proposed for online ranking and selection problems but has recently been adapted for use in online decision-making in general and multi-armed bandit problems (MABs) in particular. We study its use in a class of exponential family MABs and identify weaknesses, including a propensity to take actions which are dominated with respect to both exploitation and exploration. We propose variants of KG which avoid such errors. These new policies include an index heuristic, which deploys a KG approach to develop an approximation to the Gittins index. A numerical study shows this policy to perform well over a range of MABs including those for which index policies are not optimal. While KG does not take dominated actions when bandits are Gaussian, it fails to be index consistent and appears not to enjoy a performance advantage over competitor policies when arms are correlated to compensate for its greater computational demands.

The fundamental goal of conservation planning is biodiversity persistence, yet most reserve selection methods prioritize sites using occurrence data. Numerous empirical studies support the notion that defining and measuring objectives in terms of species richness (where the value of a site is equal to the number of species it contains, or contributes to an existing reserve network) can be inadequate for maintaining biodiversity in the long-term. An existing site-assessment framework that implicitly maximized the persistence probability of multiple species was integrated with a dynamic optimization model. The problem of sequential reserve selection as a Markov decision process was combined with stochastic dynamic programming to find the optimal solution. The approach represents a compromise between representation-based approaches (maximizing occurrences) and more complex tools, like spatially-explicit population models. The method, the inherent problems and interesting conclusions are illustrated with a land acquisition case study on the central Platte River.

Motivated by a wide range of applications, we consider a development of Whittle's restless bandit model in which project activation requires a state-dependent amount of a key resource, which is assumed to be available at a constant rate. As many projects may be activated at each decision epoch as resource availability allows. We seek a policy for project activation within resource constraints which minimises an aggregate cost rate for the system. Project indices derived from a Lagrangian relaxation of the original problem exist provided the structural requirement of indexability is met. Verification of this property and derivation of the related indices is greatly simplified when the solution of the Lagrangian relaxation has a state monotone structure for each constituent project. We demonstrate that this is indeed the case for a wide range of bidirectional projects in which the project state tends to move in a different direction when it is activated from that in which it moves when passive. This is natural in many application domains in which activation of a project ameliorates its condition, which otherwise tends to deteriorate or deplete. In some cases the state monotonicity required is related to the structure of state transitions, while in others it is also related to the nature of costs. Two numerical studies demonstrate the value of the ideas for the construction of policies for dynamic resource allocation, most especially in contexts which involve a large number of projects.

This paper concerns two families of Markov decision problem that fall within the family of (bi-directional) restless bandits, an intractable class of decision processes introduced by Whittle. The spinning plates problem concerns the optimal management of a portfolio of reward-generating assets whose yields grow with investment but otherwise tend to decline. In the model of asset exploitation called the squad system, the yield from an asset tends to decline when it is used but will recover when the asset is at rest. In all cases, simply stated conditions are given that guarantee indexability of the problem, together with conditions necessary and sufficient for its strict indexability. The index heuristics for asset activation that emerge from the analysis are assessed numerically and found to perform very strongly.

We consider a brokered market with heterogeneous suppliers and consumers of identical items or services. Each supplier determines a minimum acceptable price for the items it supplies, based on its costs and target rate of return. The broker begins with a fixed set of items from the suppliers. Customers arrive periodically to the market and offer a bid for an item. If a customer wants more than one item, a set of bids is made. If the broker accepts a bid it must be assigned an item with minimum accceptable price less than or equal to the bid, and the price paid is the bid price. The broker has the option of rejecting a bid. The broker makes acceptance and assignment decisions to maximize the total expected discounted revenue. We show the optimal policy has a simple threshold structure where thresholds can be easily determined by finding the individually optimal policies subject to priorities.

In the classical sequential assignment problem as introduced by Derman et al. (1972) there are n workers who are to be assigned a finite number of sequentially arriving jobs. If a worker of value p is assigned a job of value x the return is px, where we interpret the return as the probability that the given worker correctly completes the given job. The job value is a random value that is observed upon arrival, and jobs must be assigned or rejected when they arrive. Each worker can only do one job. Derman et al. showed that when the objective is to maximize the expected return, i.e., the expected number of correctly completed jobs, the optimal policy is a simple threshold policy, which does not depend on the worker values. Their result was extended by Albright (1974) to allow job arrivals according to a Poisson process and a single random deadline for job completion (which is equivalent to discounting). Righter (1987) further extended the result to permit workers to have independent random deadlines for job completions. Here we show that when there are independent deadlines a simple threshold policy that is independent of the worker values stochastically maximizes the number of correctly completed jobs, and therefore maximizes the expected number of correctly completed jobs. We also show that there is no policy that stochastically maximizes the number of correctly completed jobs when there is a single deadline. However, when there is single deadline and the objective is to maximize the probability that n jobs are done correctly by n workers, then the optimal policy is determined by a single threshold that is independent of n and of the worker values.

A simple proof is given for the best strategy when the game of pure strategy is played.