Course description
Information Theory is the science for measuring, preserving, transmitting, and estimating
information in random data. It was initially proposed by Shannon as a mathematical theory of communication more than five decades ago. It provides the fundamental limits of performance for transmission of messages generated by a random source over a noisy communication channel. On the one hand, Information Theory has been the driving force behind the revolution in digital communication and has led to various practical data compression and error correcting codes that meet the fundamental theoretical limits of performance. On the other hand, over the years, techniques and concepts from Information Theory have found applications well beyond communication theory. In this course, we will introduce the basic notions and results of Information Theory, keeping in mind both its fundamental role in communication theory and its varied applications beyond communication theory. This course, and the follow-up advanced courses to be offered in the future, will be of interest to students from various backgrounds.
Broad list of topics
- Compression of random data: fixed and variable length source coding theorems and entropy
- Hypothesis testing: Stein's lemma and Kullback-Leibler divergence
- Measures of randomness: Properties of Shannon entropy, Kullback-Leibler divergence, and Mutual Information
- Transmission over a noisy channel: channel coding theorems, joint source-channel coding theorem, the Gaussian channel
- Lower bounds on minimax cost in parameter estimation using Fano's inequality
- Quantisation and lossy compression: rate-distortion theorem
Who should take this course
- Communication engineer
- Computer scientists interested in data compression
- Computer scientists interested in complexity theory
- Cryptographers interested in notions of security and quantum cryptography
- Data scientists interested in information theoretic methods and measures of information
- Statisticians interested in information theoretic lower bounds
- Physicists interested in Quantum Information Theory