A new control strategy for a class of piecewise deterministic processes (PDP) is presented. In this class, PDP stochastic processes consist of ordinary differential equations that are subject to random switches corresponding to a discrete Markov process. The proposed strategy aims at controlling the probability density function (PDF) of the PDP. The optimal control formulation is based on the hyperbolic Fokker–Planck system that governs the time evolution of the PDF of the PDP and on tracking objectives of terminal configuration with a target PDF. The corresponding optimization problems are formulated as a sequence of open-loop hyperbolic optimality systems following a model predictive control framework. These systems are discretized by first-order schemes that guarantee positivity and conservativeness of the numerical PDF solution. The effectiveness of the proposed computational control framework is validated considering PDP with dichotomic noise.

In this paper, we consider a certain class of Markov renewal processes where the matrix of the transition kernel governing the Markov renewal process possesses some block-structured property, including repeating rows. Duality conditions and properties are obtained on two probabilistic measures which often play a key role in the analysis and computations of such a block-structured process. The method used here unifies two different concepts of duality. Applications of duality are also provided, including a characteristic theorem concerning recurrence and transience of a transition matrix with repeating rows and a batch arrival queueing model.

This paper deals with a service system in which the processor must serve two types of impatient units. In the case of blocking, the first type units leave the system whereas the second type units enter a pool and wait to be processed later.

We develop an exhaustive analysis of the system including embedded Markov chain, fundamental period and various classical stationary probability distributions. More specific performance measures, such as the number of lost customers and other quantities, are also considered. The mathematical analysis of the model is based on the theory of Markov renewal processes, in Markov chains of M/G/l type and in expressions of ‘Takács' equation' type.

Electric water heating loads, in power systems, can be adequately modeled by Markov processes comprising a mix of continuous and discrete states. A physically-based characterization of the dynamic behavior of large aggregates of electric water heating loads is obtained by deriving the forward Kolmogorov equations associated with the individual hybrid-state processes. In addition, by focusing on the discrete part of the state, a Markov renewal viewpoint of the processes is developed. Both viewpoints are used to analyze and predict the transient and steady-state behavior of these loads, of great importance in load management applications.

This paper identifies a probability structure for queues that belong to the class of vacation systems operating according to Markov schedules, admitting a wide variety of server scheduling disciplines including most disciplines associated with those M/GI/1/L vacation systems reported in the literature. The conditions that define these schedules are identified, and it is shown that when these conditions are satisfied, queueing behavior is governed by an underlying Markov renewal/semi-regenerative structure. A simple example is examined (the M/GI/1 vacation system with limited batch service) to demonstrate the usefulness and generality of the underlying probability structure.

We develop a theory of semiregenerative phenomena. These may be viewed as a family of linked regenerative phenomena, for which Kingman [6], [7] developed a theory within the framework of quasi-Markov chains. We use a different approach and explore the correspondence between semiregenerative sets and the range of a Markov subordinator with a unit drift (or a Markov renewal process in the discrete-time case). We use techniques based on results from Markov renewal theory.

Falin (1984) examined the quasi-input process (the flow of service starting times) in the M/G/1/∞ queue and raised the question as to whether this process is a renewal process. We show that, except in the trivial case of instantaneous service, the quasi-input process is never renewal.

Cross-covariances between the Bernoulli thinned processes of an arbitrary point process are determined. When the point process is renewal it is shown that zero correlation implies independence. An example is given to show that zero covariance between intervals does not imply zero covariance between counts. Mark-dependent thinning of Markov renewal processes is discussed and the results are applied to the overflow queue. Here we give an example of two uncorrelated but dependent renewal processes, neither of which is Poisson, which yield a Poisson process when superposed. Finally, we study Markov-chain thinning of renewal processes.

In Part I (Hunter) a study of feedback queueing models was initiated. For such models the queue-length process embedded at all transition points was formulated as a Markov renewal process (MRP). This led to the observation that the queue-length processes embedded at any of the ‘arrival', ‘departure', ‘feedback', ‘input', ‘output' or ‘external' transition epochs are also MRP. Part I concentrated on the properties of the embedded discrete-time Markov chains. In this part we examine the semi-Markov processes associated with each of these embedded MRP and derive expressions for the stationary distributions associated with their irreducible subspaces. The special cases of birth-death queues with instantaneous state-dependent feedback, M/M/1/N and M/M/1 queues with instantaneous Bernoulli feedback are considered in detail. The results obtained complement those derived in Part II (Hunter) for birth-death queues without feedback.

Queueing systems which can be formulated as Markov renewal processes with basic transitions of three types, ‘arrivals', ‘departures' and ‘feedbacks' are examined. The filtering procedure developed for Markov renewal processes by Çinlar (1969) is applied to such queueing models to show that the queue-length processes embedded at any of the ‘arrival', ‘departure', ‘feedback', ‘input', ‘output' or ‘external' transition epochs are also Markov renewal. In this part we focus attention on the derivation of stationary and limiting distributions (when they exist) for each of the embedded discrete-time processes, the embedded Markov chains. These results are applied to birth–death queues with instantaneous state-dependent feedback including the special cases of M/M/1/N and M/M/1 queues with instantaneous Bernoulli feedback.

In this part we extend and particularise results developed by the author in Part I (pp. 349–375) for a class of queueing systems which can be formulated as Markov renewal processes. We examine those models where the basic transition consists of only two types: ‘arrivals' and ‘departures'. The ‘arrival lobby' and ‘departure lobby' queue-length processes are shown, using the results of Part I to be Markov renewal. Whereas the initial study focused attention on the behaviour of the embedded discrete-time Markov chains, in this paper we examine, in detail, the embedded continuous-time semi-Markov processes. The limiting distributions of the queue-length processes in both continuous and discrete time are derived and interrelationships between them are examined in the case of continuous-time birth–death queues including the M/M/1/M and M/M/1 variants. Results for discrete-time birth–death queues are also derived.

Queues with delayed feedback have been little studied in queueing theory. Presented here is a rather complete discussion of such problems including queue length processes, busy period processes and several customer flow processes (e.g., departure processes). The case in which the delay mechanism is an M-server queue is studied in detail but it is shown later that many of the results carry over to a more general delay mechanism.

We show that if the increment distribution of a renewal process has some convolution non-singular with respect to Lebesgue measure, then the skeletons of the forward recurrence time process are φ-irreducible positive recurrent Markov chains. Known convergence properties of such chains give simple proofs of uniform versions of some old and new key renewal theorems; these show in particular that non-singularity assumptions on the increment and initial distributions enable the assumption of direct Riemann integrability to be dropped from the standard key renewal theorem. An application to Markov renewal processes is given.

It is shown that a class of infinite, block-partitioned, stochastic matrices has a matrix-geometric invariant probability vector of the form (x0, x1,…), where xk = x0Rk, for k ≧ 0. The rate matrix R is an irreducible, non-negative matrix of spectral radius less than one. The matrix R is the minimal solution, in the set of non-negative matrices of spectral radius at most one, of a non-linear matrix equation.

Applications to queueing theory are discussed. Detailed explicit and computationally tractable solutions for the GI/PH/1 and the SM/M/1 queue are obtained.

In this paper the limit behaviour of α-recurrent Markov renewal processes and semi-Markov processes is studied by using the recent results on the concept of α-recurrence for Markov renewal processes. Section 1 contains the preliminary results, which are needed later in the paper. In Section 2 we consider the limit behaviour of the transition probabilities Pij(t) of an α-recurrent semi-Markov process. Section 4 deals with quasi-stationarity. Our results extend the results of Cheong (1968), (1970) and of Flaspohler and Holmes (1972) to the case in which the functions to be considered are directly Riemann integrable. We also try to correct the errors we have found in these papers. As a special case from our results we consider continuous-time Markov processes in Sections 3 and 5.

Consider a semi-Markov process X(t) defined on a subset of the non-negative integers with zero as an absorbing state and the non-zero states forming an irreducible class with exit to zero being possible. Conditions are given for the existence of the limits: where Xj(t) is the amount of time prior to time t spent in state j.

The limits (which are independent of the initial state) are evaluated when the sufficient conditions are satisfied.