12 minute lessonReady to learn

Measure shared information

Connect marginal and joint entropy without confusing matching labels with dependence.

Course lessons · 2 of 5

Your goal

Calculate mutual information from a consistent joint model and explain independence.

Useful background: Measure uncertainty with entropy

01 · Understand the idea

Start with the joint distribution

A joint probability table gives the probability of each pair of outcomes. Summing its rows and columns produces the two marginal distributions. Those marginals must come from the same table for the entropy identity to describe mutual information.

The recovered implementation accepts marginal and joint arrays separately. Its caller must ensure consistency and normalization. Arbitrary entropy values can produce a negative number even though genuine discrete mutual information is nonnegative.

I(X;Y)=H(X)+H(Y)-H(X,Y)

Compare independence and a perfect copy

Two independent fair binary variables have four equally likely pairs. Each marginal has entropy one bit, while the pair has entropy two bits. Their mutual information is zero.

If Y is an exact copy of a fair binary X, only the pairs (0,0) and (1,1) occur. The pair has entropy one bit, so mutual information is one bit. This quantifies dependence; it does not establish which variable causes the other.

02 · Work through an example

Evaluate a copied bit

1Specify the allowed pairs

Assign probability one half to (0,0) and one half to (1,1). The other pairs have probability zero.

2Find the entropies

Each marginal is fair, and the pair also has only two equally likely states. All three entropies are one bit.

3Apply the identity

Subtract the joint uncertainty from the sum of marginal uncertainties.

I(X;Y)=1+1-1=1\text{ bit}

03 · Change something

Make the idea respond

The entropy lab changes one binary marginal probability. It has no joint probability table and does not compute mutual information. Two independent fair bits and one fair bit copied twice both have marginal entropy one bit; their joint entropies differ. Use the written joint models and questions to calculate that difference, and use the lab to inspect each marginal’s uncertainty.

Change one input at a time. Predict what should happen, then compare the computed result with your prediction.

Teaching model · computed in your browser

Uncertainty in one bit

Move from a certain outcome to an even chance. See exactly where each bit of uncertainty comes from.

H(p)=-p\log_2 p-(1-p)\log_2(1-p)

p is the probability of one binary outcome, and 1 − p is the probability of the other. Logarithms use base 2, so the result is in bits. At the endpoints, 0 log₂ 0 is assigned its limiting value 0.

Try this: compare p = 0.1 and p = 0.9. Then explain why p = 0.5 gives the maximum.

Experiment controls

Probability p 0.50 to 1 · step 0.01

The default worked example is shown below.

Probability of outcome 10.5

Probability of outcome 00.5

Binary entropy1 bits

p = 0.5 gives 1 bits. Entropy is zero at certain outcomes and reaches one bit at p = 0.5.

Two contributions: −p log₂ p = 0.5; −(1 − p) log₂(1 − p) = 0.5. Add them to obtain 1 bits.

Probability and uncertainty
p	1 − p	H(p), bits
0	1	0
0.1	0.9	0.469
0.25	0.75	0.8113
0.5	0.5	1
0.75	0.25	0.8113
0.9	0.1	0.469
1	0	0

Open the full lab

04 · Check your understanding

Your turn.

Try an answer before opening a hint. Feedback will explain the result, and you can practice again.

Interactive practice is loading. The lesson, worked example, and printable questions remain available below.

View printable practice and solutions

Use these questions when studying without interactive controls. Opening a written solution does not earn progress.

For two independent fair bits, H(X) = 1, H(Y) = 1, and H(X,Y) = 2. What is I(X;Y)?
Read the solution
The mutual information is zero bits, consistent with independence.
If Y copies a fair bit X, all three entropies in the identity are 1 bit. What is I(X;Y)?
Read the solution
The result is 1 + 1 − 1 = 1 bit.
How should the marginal distributions be obtained for a valid calculation?
- By choosing any two arrays of the right length.
- By assigning every marginal entropy the value one.
- By reading a family label.
- By summing the same joint table.
Read the solution
Marginals must agree with the joint distribution used in the calculation.
A positive mutual information value establishes which conclusion directly?
- X necessarily causes Y.
- The variables have the same physical units.
- The system is conscious.
- Statistical dependence in the specified joint model.
Read the solution
Mutual information quantifies dependence without establishing causal direction by itself.

05 · Apply it to the archive

Follow the original record

Use the definitions you just practiced to read the project’s own notation, assumptions, and evidence.

Mutual information from consistent marginals and a joint distribution Reading G16, DNA and the implemented mathematics

Take it with you

The ideas to keep

Derive consistent marginals from a joint distribution.
Independent variables have zero mutual information.
Dependence and causation are different claims.

Reading completion is separate from your practice evidence.

Progress is kept in this browser. How learning works