The Hoffman ratio bound, Lovász theta function, and Schrijver theta function are classical upper bounds for the independence number of graphs, which are useful in graph theory, extremal combinatorics, and information theory. By using generalized inverses and eigenvalues of graph matrices, we give bounds for independence sets and the independence number of graphs. Our bounds unify the Lovász theta function, Schrijver theta function, and Hoffman-type bounds, and we obtain the necessary and sufficient conditions of graphs attaining these bounds. Our work leads to some simple structural and spectral conditions for determining a maximum independent set, the independence number, the Shannon capacity, and the Lovász theta function of a graph.

We show how convergence to the Gumbel distribution in an extreme value setting can be understood in an information-theoretic sense. We introduce a new type of score function which behaves well under the maximum operation, and which implies simple expressions for entropy and relative entropy. We show that, assuming certain properties of the von Mises representation, convergence to the Gumbel distribution can be proved in the strong sense of relative entropy.

An extension of Shannon’s entropy power inequality when one of the summands is Gaussian was provided by Costa in 1985, known as Costa’s concavity inequality. We consider the additive Gaussian noise channel with a more realistic assumption, i.e. the input and noise components are not independent and their dependence structure follows the well-known multivariate Gaussian copula. Two generalizations for the first- and second-order derivatives of the differential entropy of the output signal for dependent multivariate random variables are derived. It is shown that some previous results in the literature are particular versions of our results. Using these derivatives, concavity of the entropy power, under certain mild conditions, is proved. Finally, special one-dimensional versions of our general results are described which indeed reveal an extension of the one-dimensional case of Costa’s concavity inequality to the dependent case. An illustrative example is also presented.

This paper provides a functional analogue of the recently initiated dual Orlicz–Brunn–Minkowski theory for star bodies. We first propose the Orlicz addition of measures, and establish the dual functional Orlicz–Brunn–Minkowski inequality. Based on a family of linear Orlicz additions of two measures, we provide an interpretation for the famous $f$-divergence. Jensen’s inequality for integrals is also proved to be equivalent to the newly established dual functional Orlicz–Brunn–Minkowski inequality. An optimization problem for the $f$-divergence is proposed, and related functional affine isoperimetric inequalities are established.

The proportional hazards (PH) model and its associated distributions provide suitable media for exploring connections between the Gini coefficient, Fisher information, and Shannon entropy. The connecting threads are Bayes risks of the mean excess of a random variable with the PH distribution and Bayes risks of the Fisher information of the equilibrium distribution of the PH model. Under various priors, these Bayes risks are generalized entropy functionals of the survival functions of the baseline and PH models and the expected asymptotic age of the renewal process with the PH renewal time distribution. Bounds for a Bayes risk of the mean excess and the Gini's coefficient are given. The Shannon entropy integral of the equilibrium distribution of the PH model is represented in derivative forms. Several examples illustrate implementation of the results and provide insights for potential applications.

Consider a family of Boolean models, indexed by integers n≥1. The nth model features a Poisson point process in ℝn of intensity e{nρn}, and balls of independent and identically distributed radii distributed like X̅n√n. Assume that ρn→ρ as n→∞, and that X̅n satisfies a large deviations principle. We show that there then exist the three deterministic thresholds τd, the degree threshold, τp, the percolation probability threshold, and τv, the volume fraction threshold, such that, asymptotically as n tends to ∞, we have the following features. (i) For ρ<τd, almost every point is isolated, namely its ball intersects no other ball; (ii) for τd<ρ<τp, the mean number of balls intersected by a typical ball converges to ∞ and nevertheless there is no percolation; (iii) for τp<ρ<τv, the volume fraction is 0 and nevertheless percolation occurs; (iv) for τd<ρ<τv, the mean number of balls intersected by a typical ball converges to ∞ and nevertheless the volume fraction is 0; (v) for ρ>τv, the whole space is covered. The analysis of this asymptotic regime is motivated by problems in information theory, but it could be of independent interest in stochastic geometry. The relations between these three thresholds and the Shannon‒Poltyrev threshold are discussed.

Consider a real-valued discrete-time stationary and ergodic stochastic process, called the noise process. For each dimension n, we can choose a stationary point process in ℝn and a translation invariant tessellation of ℝn. Each point is randomly displaced, with a displacement vector being a section of length n of the noise process, independent from point to point. The aim is to find a point process and a tessellation that minimizes the probability of decoding error, defined as the probability that the displaced version of the typical point does not belong to the cell of this point. We consider the Shannon regime, in which the dimension n tends to ∞, while the logarithm of the intensity of the point processes, normalized by dimension, tends to a constant. We first show that this problem exhibits a sharp threshold: if the sum of the asymptotic normalized logarithmic intensity and of the differential entropy rate of the noise process is positive, then the probability of error tends to 1 with n for all point processes and all tessellations. If it is negative then there exist point processes and tessellations for which this probability tends to 0. The error exponent function, which denotes how quickly the probability of error goes to 0 in n, is then derived using large deviations theory. If the entropy spectrum of the noise satisfies a large deviations principle, then, below the threshold, the error probability goes exponentially fast to 0 with an exponent that is given in closed form in terms of the rate function of the noise entropy spectrum. This is obtained for two classes of point processes: the Poisson process and a Matérn hard-core point process. New lower bounds on error exponents are derived from this for Shannon's additive noise channel in the high signal-to-noise ratio limit that hold for all stationary and ergodic noises with the above properties and that match the best known bounds in the white Gaussian noise case.

The depth of a trie has been deeply studied when the source which produces the words is a simple source (a memoryless source or a Markov chain). When a source is simple but not an unbiased memoryless source, the expectation and the variance are both of logarithmic order and their dominant terms involve characteristic objects of the source, for instance the entropy. Moreover, there is an asymptotic Gaussian law, even though the speed of convergence towards the Gaussian law has not yet been precisely estimated. The present paper describes a ‘natural’ class of general sources, which does not contain any simple source, where the depth of a random trie, built on a set of words independently drawn from the source, has the same type of probabilistic behaviour as for simple sources: the expectation and the variance are both of logarithmic order and there is an asymptotic Gaussian law. There are precise asymptotic expansions for the expectation and the variance, and the speed of convergence toward the Gaussian law is optimal. The paper first provides analytical conditions on the Dirichlet series of probabilities of a general source under which this Gaussian law can be derived: a pole-free region where the series is of polynomial growth. In a second step, the paper focuses on sources associated with dynamical systems, called dynamical sources, where the Dirichlet series of probabilities is expressed with the transfer operator of the dynamical system. Then, the paper extends results due to Dolgopyat, already generalized by Baladi and Vallée, and shows that the previous analytical conditions are fulfilled for ‘most’ dynamical sources, provided that they ‘strongly differ’ from simple sources. Finally, the present paper describes a class of sources not containing any simple source, where the trie depth has the same type of probabilistic behaviour as for simple sources, even with more precise estimates.

We describe a general framework for realistic analysis of sorting algorithms, and we apply it to the average-case analysis of three basic sorting algorithms (QuickSort, InsertionSort, BubbleSort). Usually the analysis deals with the mean number of key comparisons, but here we view keys as words produced by the same source, which are compared via their symbols in lexicographic order. The ‘realistic’ cost of the algorithm is now the total number of symbol comparisons performed by the algorithm, and, in this context, the average-case analysis aims to provide estimates for the mean number of symbol comparisons used by the algorithm. For sorting algorithms, and with respect to key comparisons, the average-case complexity of QuickSort is asymptotic to 2n log n, InsertionSort to n2/4 and BubbleSort to n2/2. With respect to symbol comparisons, we prove that their average-case complexity becomes Θ (n log2n), Θ(n2), Θ (n2 log n). In these three cases, we describe the dominant constants which exhibit the probabilistic behaviour of the source (namely entropy and coincidence) with respect to the algorithm.

In this paper, we introduce the minimum dynamic discrimination information (MDDI) approach to probability modeling. The MDDI model relative to a given distribution G is that which has least Kullback-Leibler information discrepancy relative to G, among all distributions satisfying some information constraints given in terms of residual moment inequalities, residual moment growth inequalities, or hazard rate growth inequalities. Our results lead to MDDI characterizations of many well-known lifetime models and to the development of some new models. Dynamic information constraints that characterize these models are tabulated. A result for characterizing distributions based on dynamic Rényi information divergence is also given.

A formal approach to produce a model for the data-generating distribution based on partial knowledge is the well-known maximum entropy method. In this approach, partial knowledge about the data-generating distribution is formulated in terms of some information constraints and the model is obtained by maximizing the Shannon entropy under these constraints. Frequently, in reliability analysis the problem of interest is the lifetime beyond an age t. In such cases, the distribution of interest for computing uncertainty and information is the residual distribution. The information functions involving a residual life distribution depend on t, and hence are dynamic. The maximum dynamic entropy (MDE) model is the distribution with the density that maximizes the dynamic entropy for all t. We provide a result that relates the orderings of dynamic entropy and the hazard function for distributions with monotone densities. Applications include dynamic entropy ordering within some parametric families of distributions, orderings of distributions of lifetimes of systems and their components connected in series and parallel, record values, and formulation of constraints for the MDE model in terms of the evolution paths of the hazard function and mean residual lifetime function. In particular, we identify classes of distributions in which some well-known distributions, including the mixture of two exponential distributions and the mixture of two Pareto distributions, are the MDE models.

Wyner and Ziv (1989) studied the asymptotic properties of recurrence times of stationary ergodic processes, and applied the results to obtain optimal data compression schemes in information transmission. Since then many data compression algorithms based upon string matching of a sequence from an information source with a database have been proposed and studied. In this paper we consider Gaussian stationary processes representing an information source and a database, and study problems of string matching with distortion. We prove theorems concerning the asymptotic behavior of the probability of string matching with distortion and the waiting time for the string matching.

We present an entropy conservation principle applicable to either discrete or continuous variables which provides a useful tool for aggregating observations. The associated method of modality grouping transforms a variable Z1 into a new variable Z2 such that the mutual information I(Z2,Y) between Y, a variable of interest, and Z2 is equal to I(Z1,Y).