Entropy and Information
Measure of uncertainty.
Key idea
Measure of uncertainty.
In depth
Entropy H = −Σ p log₂ p measures average uncertainty in bits. A fair coin has 1 bit; a certain event has 0. It sets the limit for lossless compression.
Worked example
A biased coin with p = 0.9 has H ≈ 0.47 bits.
Formula
Real-world uses
- Compressing files in ZIP and JPEG formats
- Measuring surprise in messages and passwords
Common mistake
Using the wrong log base, which changes units but not the meaning.
How to learn this
- Read the key idea and say it in your own words.
- Work through the worked example, then try a similar problem.
- Use the interactive lesson’s visual and mini-quiz to check the pattern.
Frequently asked questions
What is Entropy and Information?
Measure of uncertainty. It is a topic at the expert level in Probability & Statistics.
What is the formula for Entropy and Information?
The key formula is H = −Σp log p.
Where is Entropy and Information used in real life?
Real-world examples: Compressing files in ZIP and JPEG formats; Measuring surprise in messages and passwords.
What should I learn before Entropy and Information?
Make sure you are comfortable with Data Ethics and Misleading Graphs, the previous topic in Probability & Statistics.
What is a common mistake with Entropy and Information?
Using the wrong log base, which changes units but not the meaning.
How can I practise Entropy and Information?
Open the interactive lesson, play with the visual, then answer the three-question mini-quiz. Getting two right marks the topic complete.
What comes after Entropy and Information?
Move on to Gambler’s Fallacy next.
Related expert topics in Probability & Statistics
- Skewness and Kurtosis – Measures of asymmetry and tail heaviness.
- Residuals and R² – Error of a fit and proportion of variance explained.
- Simpson’s Paradox – A trend reverses when groups are combined.
- Law of Total Probability – Split an event over a partition.
- Bayes’ Theorem – Update beliefs with evidence.
- Medical Tests and Base Rates – Why positive results can still be probably false.