In this paper we investigate risk-sensitive semi-Markov decision processes with a Borel state space, unbounded cost rates, and general utility functions. The performance criteria are several expected utilities of the total cost in a finite horizon. Our analysis is based on a type of finite-horizon occupation measure. We express the distribution of the finite-horizon cost in terms of the occupation measure for each policy, wherein the discount is not needed. For unconstrained and constrained problems, we establish the existence and computation of optimal policies. In particular, we develop a linear program and its dual program for the constrained problem and, moreover, establish the strong duality between the two programs. Finally, we provide two special cases of our results, one of which concerns the discrete-time model, and the other the chance-constrained problem.

We address a queueing control problem considering service times and conversion timesfollowing normal distributions. We formulate the multi-server queueing control problem byconstructing a semi-Markov decision process (SMDP) model. The mechanism of statetransitions is developed through mathematical derivation of the transition probabilitiesand transition times. We also study the property of the queueing control system and showthat optimizing the objective function of the addressed queueing control problem isequivalent to maximizing the time-average reward.

Pharmaceutical companies have to face huge risks and enormous costs of production before they can produce a drug. Efficient allocation of resources is essential to help in maximizing profits. Yu and Gittins (2007) described a model and associated software for determining efficient allocations for a preclinical research project. This is the starting point for this paper. We provide explicit optimal policies for the selection of successive candidate drugs for two restricted versions of the Yu and Gittins (2007) model. To some extent these policies are likely to be applicable to the unrestricted model.

We study an optimal admission of arriving customers to a Markovian finite-capacity queue, e.g. an M/M/c/N queue, with several customer types. The system managers are paid for serving customers and penalized for rejecting them. The rewards and penalties depend on customer type. The goal is to maximize the average rewards per unit time subject to the constraint on the average penalties per unit time. We provide a solution to this problem based on Lagrangian optimization. For a feasible problem, we show the existence of a randomized trunk reservation optimal policy with the acceptance thresholds for different customer types ordered according to a linear combination of the service rewards and rejection costs. In addition, we prove that any 1-randomized stationary optimal policy has this structure. In particular, we establish the structure of an optimal policy that maximizes the average rewards per unit time subject to the constraint on the blocking probability of either one of the customer types or a group of customer types pooled together.

In this paper, we study a repair replacement model for a stochastically deteriorating system. For the expected discounted reward case, we show that the optimal replacement policy is of the form ‘replace at the time of the Nth failure'.

In this paper we consider a class of single-server queueing systems with compound Poisson arrivals, in which, at service completion epochs, the server has the option of taking off for one or several vacations of random length. The cost structure consists of a holding cost rate specified by a general non-decreasing function of the queue size, fixed costs for initiating and terminating service, and a variable operating cost incurred for each unit of time that the system is in operation. We show under some weak conditions with respect to the holding cost rate function and the service time, vacation time and arrival batch size distributions that it is either optimal among all feasible (stationary and non-stationary) policies never to take a vacation, or it is optimal to take a vacation when the system empties out and to resume work when, upon completion of a vacation, the queue size is equal to or in excess of a critical threshold. These optimality results are generalized for several variants of this model.

We consider a single-server queueing system with Poisson arrivals and general service times. While the server is up, it is subject to breakdowns according to a Poisson process. When the server breaks down, we may either repair the server immediately or postpone the repair until some future point in time. The operating costs of the system include customer holding costs, repair costs and running costs. The objective is to find a corrective maintenance policy which minimizes the long-run average operating costs of the system. The problem is formulated as a semi-Markov decision process. Under some mild conditions on the repair time and service time distributions and the customer holding cost rate function, we prove that there exists an optimal stationary policy which is characterized by a single threshold parameter: a repair is initiated if and only if the number of customers in the system exceeds this threshold. We also show how the average cost under such policies may be computed and how an optimal policy may efficiently be determined.

A new approach to the optimal control of the service rate in M/G/1 queues is introduced using the method of phases. Each customer's work requirement is approximated by a random number of exponential phases with (possibly) different parameters (a generalized hyper-Erlang distribution). Using a semi-Markov decision-process formulation, we establish monotonicity properties of optimal policies for the finite-horizon problem, by induction on the horizon length. The analysis is then extended to the discounted infinite-horizon and the long-run average-return problems. In contrast to the models in previous papers, our model is appropriate for situations where the system controller has partial information about the work requirement of a customer, specifically the number of phases (tasks) to be performed. Because it requires a multidimensional state description, the analysis of the phase-type control model may be viewed as a first step toward the solution of control models for networks of queues.

The problem of controlling input to a stochastic input-output system by accepting or rejecting arriving customers is analyzed as a semi-Markov decision process. Included as special cases are a GI/G/1 model and models with compound input and/or output processes, as well as several previously studied queueing-control models. We establish monotonicity of socially and individually optimal acceptance policies and the more restrictive nature of the former, with random rewards for acceptance and both customer-oriented and system-oriented non-linear waiting costs. Distinctive features of our analysis are (i) that it allows dependent interarrival times and (ii) that the monotonicity proofs do not rely on the standard concavity-preservation arguments.

During the course of an R and D project, it is often meaningful and possible to evaluate its status, so that this information may be used for making better financing decisions over time. The project status changes stochastically due to the internal (technological) and the external (market) uncertainties, the former being partially controlled by expenditure of resources. In addition to the resource expenditure strategy, the manager must also decide when to terminate the project. Once the project is terminated, a terminal return is collected, whose value depends on the final project status. It is shown that the project should be terminated if the current status is either too low or too high to make further expenditure worthwhile. Otherwise, for an intermediate (promising) status of the project, an aggressive investment strategy is shown to be optimal. Thus, the model unifies the problems of optimally undertaking, financing and terminating an R and D project in face of various uncertainties.