Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Notation
- Contributors
- 1 Introduction to Information Theory and Data Science.
- 2 An Information-Theoretic Approach to Analog-to-Digital Compression
- 3 Compressed Sensing via Compression Codes
- 4 Information-Theoretic Bounds on Sketching
- 5 Sample Complexity Bounds for Dictionary Learning from Vector- and Tensor-Valued Data
- 6 Uncertainty Relations and Sparse Signal Recovery
- 7 Understanding Phase Transitions via Mutual Information and MMSE
- 8 Computing Choice: Learning Distributions over Permutations
- 9 Universal Clustering
- 10 Information-Theoretic Stability and Generalization
- 11 Information Bottleneck and Representation Learning
- 12 Fundamental Limits in Model Selection for Modern Data Analysis
- 13 Statistical Problems with Planted Structures: Information-Theoretical and Computational Limits
- 14 Distributed Statistical Inference with Compressed Data
- 15 Network Functional Compression
- 16 An Introductory Guide to Fano’s Inequality with Applications in Statistical Estimation
- Index
- References
6 - Uncertainty Relations and Sparse Signal Recovery
Published online by Cambridge University Press: 22 March 2021
Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Notation
- Contributors
- 1 Introduction to Information Theory and Data Science.
- 2 An Information-Theoretic Approach to Analog-to-Digital Compression
- 3 Compressed Sensing via Compression Codes
- 4 Information-Theoretic Bounds on Sketching
- 5 Sample Complexity Bounds for Dictionary Learning from Vector- and Tensor-Valued Data
- 6 Uncertainty Relations and Sparse Signal Recovery
- 7 Understanding Phase Transitions via Mutual Information and MMSE
- 8 Computing Choice: Learning Distributions over Permutations
- 9 Universal Clustering
- 10 Information-Theoretic Stability and Generalization
- 11 Information Bottleneck and Representation Learning
- 12 Fundamental Limits in Model Selection for Modern Data Analysis
- 13 Statistical Problems with Planted Structures: Information-Theoretical and Computational Limits
- 14 Distributed Statistical Inference with Compressed Data
- 15 Network Functional Compression
- 16 An Introductory Guide to Fano’s Inequality with Applications in Statistical Estimation
- Index
- References
Summary
This chapter provides an introduction to uncertainty relations underlying sparse signal recovery. We start with the seminal work by Donoho and Stark (1989), which defines uncertainty relations as upper bounds on the operator norm of the band-limitation operator followed by the time-limitation operator, generalize this theory to arbitrary pairs of operators, and then develop, out of this generalization, the coherence-based uncertainty relations due to Elad and Bruckstein (2002), plus uncertainty relations in terms of concentration of the 1-norm or 2-norm. The theory is completed with set-theoretic uncertainty relations which lead to best possible recovery thresholds in terms of a general measure of parsimony, the Minkowski dimension. We also elaborate on the remarkable connection between uncertainty relations and the “large sieve,” a family of inequalities developed in analytic number theory. We show how uncertainty relations allow one to establish fundamental limits of practical signal recovery problems such as inpainting, declipping, super-resolution, and denoising of signals corrupted by impulse noise or narrowband interference.
Keywords
- Type
- Chapter
- Information
- Information-Theoretic Methods in Data Science , pp. 163 - 196Publisher: Cambridge University PressPrint publication year: 2021
References
Heisenberg, W., The physical principles of the quantum theory. University of Chicago Press, 1930.Google Scholar
Faris, W. G., “Inequalities and uncertainty principles,” J. Math. Phys., vol. 19, no. 2, pp. 461–466, 1978.Google Scholar
Cowling, M. G. and Price, J. F., “Bandwidth versus time concentration: The Heisenberg–Pauli–Weyl inequality,” SIAM J. Math. Anal., vol. 15, no. 1, pp. 151–165, 1984.CrossRefGoogle Scholar
Benedetto, J. J., Wavelets: Mathematics and applications. CRC Press, 1994, ch. Frame decompositions, sampling, and uncertainty principle inequalities.Google Scholar
Folland, G. B. and Sitaram, A., “The uncertainty principle: A mathematical survey,” J. Fourier Analysis Applications, vol. 3, no. 3, pp. 207–238, 1997.Google Scholar
Donoho, D. L. and Stark, P. B., “Uncertainty principles and signal recovery,” SIAM J. Appl. Math., vol. 49, no. 3, pp. 906–931, 1989.Google Scholar
Donoho, D. L. and Logan, B. F., “Signal recovery and the large sieve,” SIAM J. Appl. Math., vol. 52, no. 2, pp. 577–591, 1992.Google Scholar
Elad, M. and Bruckstein, A. M., “A generalized uncertainty principle and sparse representation in pairs of bases,” IEEE Trans. Information Theory, vol. 48, no. 9, pp. 2558–2567, 2002.Google Scholar
Studer, C., Kuppinger, P., Pope, G., and Bölcskei, H., “Recovery of sparsely corrupted signals,” IEEE Trans. Information Theory, vol. 58, no. 5, pp. 3115–3130, 2012.Google Scholar
Kuppinger, P., Durisi, G., and Bölcskei, H., “Uncertainty relations and sparse signal recovery for pairs of general signal sets,” IEEE Trans. Information Theory, vol. 58, no. 1, pp. 263–277, 2012.CrossRefGoogle Scholar
Terras, A., Fourier analysis on finite groups and applications. Cambridge University Press, 1999.Google Scholar
Tao, T., “An uncertainty principle for cyclic groups of prime order,” Math. Res. Lett., vol. 12, no. 1, pp. 121–127, 2005.Google Scholar
Stotz, D., Riegler, E., Agustsson, E., and Bölcskei, H., “Almost lossless analog signal separation and probabilistic uncertainty relations,” IEEE Trans. Information Theory, vol. 63, no. 9, pp. 5445–5460, 2017.CrossRefGoogle Scholar
Foucart, S. and Rauhut, H., A mathematical introduction to compressive sensing. Birkhäuser, 2013.Google Scholar
Gabor, D., “Theory of communications,” J. Inst. Elec. Eng., vol. 96, pp. 429–457, 1946.Google Scholar
Bölcskei, H., Advances in Gabor analysis. Birkhäuser, 2003, ch. Orthogonal frequency division multiplexing based on offset QAM.Google Scholar
Fefferman, C., “The uncertainty principle,” Bull. Amer. Math. Soc., vol. 9, no. 2, pp. 129–206, 1983.Google Scholar
Evra, S., Kowalski, E., and Lubotzky, A., “Good cyclic codes and the uncertainty principle,” arXiv:1703.01080, 2017.Google Scholar
Bombieri, E., Le grand crible dans la théorie analytique des nombres. Société Mathématique de France, 1974.Google Scholar
Montgomery, H. L., Twentieth century harmonic analysis – a celebration. Springer, 2001, ch. Harmonic analysis as found in analytic number theory.Google Scholar
Baraniuk, R. G. and Wakin, M. B., “Random projections of smooth manifolds,” Found. Comput. Math., vol. 9, no. 1, pp. 51–77, 2009.CrossRefGoogle Scholar
Candès, E. J. and Recht, B., “Exact matrix completion via convex optimization,” Found. Comput. Math., vol. 9, no. 6, pp. 717–772, 2009.Google Scholar
Eldar, Y. C., Kuppinger, P., and Bölcskei, H., “Block-sparse signals: Uncertainty relations and efficient recovery,” IEEE Trans. Signal Processing, vol. 58, no. 6, pp. 3042–3054, 2010.Google Scholar
Candès, E. J. and Plan, Y., “Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements,” IEEE Trans. Information Theory, vol. 4, no. 57, pp. 2342–2359, 2011. uncertainty Relations and Sparse Signal Recovery 195Google Scholar
Alberti, G., Bölcskei, H., De Lellis, C., Koliander, G., and Riegler, E., “Lossless analog compression,” submitted to IEEE Trans. Information Theory, arXiv:1803.06887, 2018.Google Scholar
Riegler, E., Stotz, D., and Bölcskei, H., “Information -theoretic limits of matrix completion,” in Proc. IEEE International Symposium on Information Theory, 2015, pp. 1836–1840.CrossRefGoogle Scholar
Izenman, A. J., “Introduction to manifold learning,” WIREs Comput. Statist., vol. 4, pp. 439–446, 2012.Google Scholar
Lu, H., Fainman, Y., and Hecht-Nielsen, R., “Image manifolds,” in Proc. SPIE, 1998, vol. 3307, pp. 52–63.Google Scholar
Sochen, N. and Zeevi, Y. Y., “Representation of colored images by manifolds embedded in higher dimensional non-Euclidean space,” in Proc. IEEE International Conference on Image Processing, 1998, pp. 166–170.Google Scholar
Hinton, G. E., Dayan, P., and Revow, M., “Modeling the manifolds of images of handwritten digits,” IEEE Trans. Neural Networks, vol. 8, no. 1, pp. 65–74, 1997.Google Scholar
Leland, W. E., Taqqu, M. S., Willinger, W., and Wilson, D. V., “On the self-similar nature of Ethernet traffic (extended version),” IEEE/ACM Trans. Networks, vol. 2, no. 1, pp. 1–15, 1994.CrossRefGoogle Scholar
Abel, J. S. and Smith III, J. O., “Restoring a clipped signal,,” in Proc. IEEE International Conference on Acoustics Speech Signal Processing, 1991, pp. 1745–1748.Google Scholar
Mallat, S. G. and Yu, G., “Super-resolution with sparse mixing estimators,” IEEE Trans. Image Processing, vol. 19, no. 11, pp. 2889–2900, 2010.CrossRefGoogle ScholarPubMed
Elad, M. and Hel-Or, Y., “Fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur,” IEEE Trans. Image Processing, vol. 10, no. 8, pp. 1187–1193, 2001.Google Scholar
Bertalmio, M., Sapiro, G., v. Caselles, and Ballester, C., “Image inpainting,” in Proc. 27th Annual Conference on Computer Graphics and Interactive Techniques, 2000, pp. 417–424.Google Scholar
Elad, M., Starck, J.-L., Querre, P., and Donoho, D. L., “Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA),” Appl. Comput. Harmonic Analysis, vol. 19, pp. 340–358, 2005.Google Scholar
Donoho, D. L. and Kutyniok, G., “Microlocal analysis of the geometric separation problem,” Commun. Pure Appl. Math., vol. 66, no. 1, pp. 1–47, 2013.Google Scholar
Tropp, J. A., “On the conditioning of random subdictionaries,” Appl. Comput. Harmonic Analysis, vol. 25, pp. 1–24, 2008.Google Scholar
Candès, E. J., Romberg, J., and Tao, T., “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Information Theory, vol. 52, no. 2, pp. 489–509, 2006.Google Scholar
Pope, G., Bracher, A., and Studer, C., “Probabilistic recovery guarantees for sparsely corrupted signals,” IEEE Trans. Information Theory, vol. 59, no. 5, pp. 3104–3116, 2013.Google Scholar
Donoho, D. L., “Compressed sensing,” IEEE Trans. Information Theory, vol. 52, no. 4, pp. 1289–1306, 2006.Google Scholar
Tropp, J. A., “On the linear independence of spikes and sines,” J. Fourier Analysis Applications, vol. 14, no. 5, pp. 838–858, 2008.CrossRefGoogle Scholar
Donoho, D. L., Johnstone, I. M., Hoch, J. C., and Stern, A. S., “Maximum entropy and the nearly black object,” J. Roy. Statist. Soc. Ser. B., vol. 54, no. 1, pp. 41–81, 1992.Google Scholar
Feng, P. and Bresler, Y., “Spectrum -blind minimum-rate sampling and reconstruction of multiband signals,” in Proc. IEEE International Conference on Acoustics Speech Signal Processing, 1996, pp. 1688–1691.Google Scholar
Mishali, M. and Eldar, Y. C., “From theory to practice: Sub-Nyquist sampling of sparse wideband analog signals,” IEEE J. Selected Areas Commun., vol. 4, no. 2, pp. 375–391, 2010.Google Scholar
Heckel, R. and Bölcskei, H., “Identification of sparse linear operators,” IEEE Trans. Information Theory, vol. 59, no. 12, pp. 7985–8000, 2013.Google Scholar
Vaaler, J. D., “Some extremal functions in Fourier analysis,” Bull. Amer. Math. Soc., vol. 2, no. 12, pp. 183–216, 1985.Google Scholar
Mattila, P., Geometry of sets and measures in Euclidean space: Fractals and rectifiability. Cambridge University Press, 1999.Google Scholar