Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-q99xh Total loading time: 0 Render date: 2024-12-28T05:24:52.421Z Has data issue: false hasContentIssue false

2 - An Information-Theoretic Approach to Analog-to-Digital Compression

Published online by Cambridge University Press:  22 March 2021

By
Alon Kipnis ,
Yonina C. Eldar  and
Andrea J. Goldsmith
Edited by
Miguel R. D. Rodrigues  and
Yonina C. Eldar
Miguel R. D. Rodrigues
Affiliation:
University College London
Yonina C. Eldar
Affiliation:
Weizmann Institute of Science, Israel
Get access

Summary

Processing, storing, and communicating information that originates as an analog phenomenon involve conversion of the information to bits. This conversion can be described by the combined effect of sampling and quantization. The digital representation in this procedure is achieved by first sampling the analog signal so as to represent it by a set of discrete-time samples and then quantizing these samples to a finite number of bits. Traditionally, these two operations are considered separately. The sampler is designed to minimize information loss due to sampling based on prior assumptions about the continuous-time input. The quantizer is designed to represent the samples as accurately as possible, subject to the constraint on the number of bits that can be used in the representation. The goal of this chapter is to revisit this paradigm by considering the joint effect of these two operations and to illuminate the dependence between them.

Keywords

analog-to-digital conversiondigital-to-analog conversionsamplingdistortion raterate distortionNyquist samplingsub-Nyquist samplingaliasing ADX
Type
Chapter
Information
Information-Theoretic Methods in Data Science , pp. 44 - 71
Publisher: Cambridge University Press
Print publication year: 2021

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Eldar, Y. C., Sampling theory: Beyond bandlimited systems. Cambridge University Press, 2015.Google Scholar
Gray, R. M. and Neuhoff, D. L., “Quantization,” IEEE Trans. Information Theory, vol. 44, no. 6, pp. 2325–2383, 1998.Google Scholar
Shannon, C. E., “Communication in the presence of noise,” IRE Trans. Information Theory, vol. 37, pp. 10–21, 1949.Google Scholar
Landau, H., “Sampling, data transmission, and the Nyquist rate,” Proc. IEEE, vol. 55, no. 10, pp. 1701–1706, 1967.Google Scholar
Shannon, C. E., “A mathematical theory of communication,” Bell System Technical J., vol. 27, pp. 379–423, 623–656, 1948.Google Scholar
Shannon, C. E., “Coding theorems for a discrete source with a fidelity criterion,” IRE National Convention Record, vol. 4, no. 1, pp. 142–163, 1959.Google Scholar
Berger, T., Rate-distortion theory: A mathematical basis for data compression. Prentice-Hall, 1971.Google Scholar
Walden, R., “Analog -to-digital converter survey and analysis,” IEEE J. Selected Areas in Communications, vol. 17, no. 4, pp. 539–550, 1999.Google Scholar
Candy, J., “A use of limit cycle oscillations to obtain robust analog-to-digital converters,” IEEE Trans. Communications, vol. 22, no. 3, pp. 298–305, 1974.Google Scholar
Oliver, B., Pierce, J., and Shannon, C., “The philosophy of PCM,” IRE Trans. Information Theory, vol. 36, no. 11, pp. 1324–1331, 1948.Google Scholar
Neuhoff, D. L. and Pradhan, S. S., “Information rates of densely sampled data: Distributed vector quantization and scalar quantization with transforms for Gaussian sources,” IEEE Trans. Information Theory, vol. 59, no. 9, pp. 5641–5664, 2013.Google Scholar
Matthews, M., “On the linear minimum-mean-squared-error estimation of an undersampled wide-sense stationary random process,” IEEE Trans. Signal Processing, vol. 48, no. 1, pp. 272–275, 2000.Google Scholar
Chan, D. and Donaldson, R., “Optimum pre- and postfiltering of sampled signals with application to pulse modulation and data compression systems,” IEEE Trans. Communication Technol., vol. 19, no. 2, pp. 141–157, 1971.Google Scholar
Dobrushin, R. and Tsybakov, B., “Information transmission with additional noise,” IRE Trans. Information Theory, vol. 8, no. 5, pp. 293–304, 1962.Google Scholar
Kolmogorov, A., “On the Shannon theory of information transmission in the case of continuous signals,” IRE Trans. Information Theory, vol. 2, no. 4, pp. 102–108, 1956.Google Scholar
Lapidoth, A., “On the role of mismatch in rate distortion theory,” IEEE Trans. Information Theory, vol. 43, no. 1, pp. 38–47, 1997.Google Scholar
Kontoyiannis, I. and Zamir, R., “Mismatched codebooks and the role of entropy coding in lossy data compression,” IEEE Trans. Information Theory, vol. 52, no. 5, pp. 1922–1938, 2006.Google Scholar
Zemanian, A. H., Distribution theory and transform analysis: An introduction to generalized functions, with applications. Courier Corporation, 1965.Google Scholar
Wolf, J. and Ziv, J., “Transmission of noisy information to a noisy receiver with minimum distortion,” IEEE Trans. Information Theory, vol. 16, no. 4, pp. 406–411, 1970.Google Scholar
Witsenhausen, H., “Indirect rate distortion problems,” IEEE Trans. Information Theory, vol. 26, no. 5, pp. 518–521, 1980.Google Scholar
Landau, H., “Necessary density conditions for sampling and interpolation of certain entire functions,” Acta Mathematica, vol. 117, no. 1, pp. 37–52, 1967.Google Scholar
Beurling, A. and Carleson, L., The collected works of Arne Beurling: Complex analysis. Birkhäuser, 1989, vol. 1.Google Scholar
Beutler, F. J., “Sampling theorems and bases in a Hilbert space,” Information and Control, vol. 4, nos. 2–3, pp. 97–117, 1961.Google Scholar
Kipnis, A., Eldar, Y. C., and Goldsmith, A. J., “Fundamental distortion limits of analog-to-digital compression,” IEEE Trans. Information Theory, vol. 64, no. 9, pp. 6013–6033, 2018.Google Scholar
Bennett, W., “Statistics of regenerative digital transmission,” Bell Labs Technical J., vol. 37, no. 6, pp. 1501–1542, 1958.Google Scholar
Kipnis, A., Goldsmith, A. J., and Eldar, Y. C., “The distortion rate function of cyclostationary Gaussian processes,” IEEE Trans. Information Theory, vol. 64, no. 5, pp. 3810–3824, 2018.Google Scholar
Kipnis, A., Goldsmith, A. J., Eldar, Y. C., and Weissman, T., “Distortion rate function of sub-Nyquist sampled Gaussian sources,” IEEE Trans. Information Theory, vol. 62, no. 1, pp. 401–429, 2016.Google Scholar
Kipnis, A., “Fundamental distortion limits of analog-to-digital compression,” Ph.D. dissertation, Stanford University, 2018.Google Scholar
Kipnis, A., Goldsmith, A. J., and Eldar, Y. C., “The distortion-rate function of sampled Wiener processes,” in IEEE Transactions on Information Theory, vol. 65, no.1, pp. 482-499, Jan. 2019. doi: 10.1109/TTT.2018.2878446Google Scholar
Kipnis, A., Reeves, G., and Eldar, Y. C., “Single letter formulas for quantized compressed sensing with Gaussian codebooks,,” in 2018 IEEE International Symposium on Information Theory (ISIT), 2018, pp. 71–75.Google Scholar
Kipnis, A., Reeves, G., Eldar, Y. C., and Goldsmith, A. J., “Compressed sensing under optimal quantization,,” in 2017 IEEE International Symposium on Information Theory (ISIT), 2017, pp. 2148–2152.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×