12 minute lessonReady to learn

Measure uncertainty with entropy

Turn a binary probability into a reproducible information quantity.

Course lessons · 1 of 5

Your goal

Calculate simple binary entropies and identify their units and limiting cases.

Useful background: Decide which inputs are valid

01 · Understand the idea

A distribution describes possible outcomes

A Bernoulli variable has two outcomes with probabilities p and 1 − p. Entropy adds the uncertainty contribution of each outcome. In this lesson, logarithms are base two, so the result is in bits. The bare expression with natural logarithms instead gives nats; dividing that result by ln 2 converts it to bits.

Read the implementation as well as the displayed expression. The linked C++ source calculates natural-log entropy and divides by LN2, so its returned value is in bits. The linked Go Shannon-entropy implementation uses math.Log2 and also returns bits. At an equal split, the mathematical entropy is one bit or ln 2 nats; these are different units for the same uncertainty.

H_2(p)=-p\log_2 p-(1-p)\log_2(1-p)

Predict the shape before calculating

When p is zero or one, the outcome is certain and mathematical binary entropy is zero. The continuous extension assigns 0 log 0 its limiting value zero. The laboratory uses this continuous mathematical model. It does not directly evaluate log zero.

The linked C++ implementation also returns zero whenever p ≤ 0.001 or p ≥ 0.999. That cutoff changes the returned value near the endpoints: small positive mathematical entropy can be replaced with zero. The linked Go entropy code likewise skips terms under its own small-probability cutoff. Those numerical choices are separate from the mathematical endpoint limit.

Uncertainty is greatest at p = 0.5. The curve is symmetric because exchanging the labels of the two outcomes exchanges p and 1 − p without changing uncertainty. Entropy measures a distribution; it does not directly measure intelligence or experience.

02 · Work through an example

Calculate an equal binary split

1Write both probabilities

At p = 0.5, each outcome has probability one half.

2Calculate each contribution

The base-two logarithm of one half is −1, so each contribution is one half bit.

-(1/2)\log_2(1/2)=1/2

3Add the contributions

Two contributions of one half give one bit. With natural logarithms, the result would instead be ln 2 nats.

H_2(1/2)=1

03 · Change something

Make the idea respond

This lab uses the continuous base-two binary-entropy formula and reports bits. It handles p = 0 and p = 1 by their mathematical limits. It does not reproduce the linked C++ source’s additional 0.001/0.999 cutoff or the Go source’s small-probability term cutoff.

Change one input at a time. Predict what should happen, then compare the computed result with your prediction.

Teaching model · computed in your browser

Uncertainty in one bit

Move from a certain outcome to an even chance. See exactly where each bit of uncertainty comes from.

H(p)=-p\log_2 p-(1-p)\log_2(1-p)

p is the probability of one binary outcome, and 1 − p is the probability of the other. Logarithms use base 2, so the result is in bits. At the endpoints, 0 log₂ 0 is assigned its limiting value 0.

Try this: compare p = 0.1 and p = 0.9. Then explain why p = 0.5 gives the maximum.

Experiment controls

Probability p 0.50 to 1 · step 0.01

The default worked example is shown below.

Probability of outcome 10.5

Probability of outcome 00.5

Binary entropy1 bits

p = 0.5 gives 1 bits. Entropy is zero at certain outcomes and reaches one bit at p = 0.5.

Two contributions: −p log₂ p = 0.5; −(1 − p) log₂(1 − p) = 0.5. Add them to obtain 1 bits.

Probability and uncertainty
p	1 − p	H(p), bits
0	1	0
0.1	0.9	0.469
0.25	0.75	0.8113
0.5	0.5	1
0.75	0.25	0.8113
0.9	0.1	0.469
1	0	0

Open the full lab

04 · Check your understanding

Your turn.

Try an answer before opening a hint. Feedback will explain the result, and you can practice again.

Interactive practice is loading. The lesson, worked example, and printable questions remain available below.

View printable practice and solutions

Use these questions when studying without interactive controls. Opening a written solution does not earn progress.

What is binary entropy at p = 0.5 using base-two logarithms?
Read the solution
The total is 0.5 + 0.5 = 1 bit.
What is binary entropy at p = 1, using the continuous endpoint convention?
Read the solution
Both contributions are zero, so entropy is 0 bits.
How does H₂(0.2) compare with H₂(0.8)?
- Both must be zero.
- They are equal.
- The first is always larger.
- The second is always larger.
Read the solution
The binary entropy is symmetric under p ↔ 1 − p.
A bare binary-entropy expression uses natural logarithms and is NOT divided by ln 2. What unit does that expression produce?
- A probability between zero and one by definition
- Nats
- Bits without conversion
- Seconds
Read the solution
The bare natural-logarithm expression gives nats. The linked C++ code subsequently divides by LN2, so that implementation returns bits instead.

05 · Apply it to the archive

Follow the original record

Use the definitions you just practiced to read the project’s own notation, assumptions, and evidence.

Binary Shannon entropy Shannon entropy with truncated small probabilities

Take it with you

The ideas to keep

Specify the logarithm base, any conversion, and the returned unit.
Distinguish the mathematical endpoint limit from a source implementation’s probability cutoff.
Entropy describes uncertainty in a distribution.

Reading completion is separate from your practice evidence.

Progress is kept in this browser. How learning works