============================================
; Title:       BU 320 Week 11 — Bayes' Theorem and Shannon Entropy
; Author:      Hali Sarah Parsons
; Date:        24 May 2026
; Description: Three problems covering Bayes' Theorem applied to a
;              manufacturing probability tree, a custom Shannon entropy
;              function built from the Hadrien (2021) formula on p. 143,
;              and a comparison of entropy across two discrete distributions.
============================================

BU 320 Week 11 — Bayes' Theorem and Shannon Entropy

Hali Sarah Parsons | Bellevue University | May 2026

---

Overview

This notebook covers three problems from the Week 11 assignment:

1. Bayes' Theorem — two conditional probability calculations from a manufacturing probability tree
2. Entropy Function — a custom Python function implementing the Shannon entropy formula from Hadrien (2021, p. 143)
3. Comparing Entropies — applying that function to two distributions and interpreting the difference

All three connect to the same underlying idea: how much uncertainty exists in a probability distribution, and how Bayes' Theorem updates that uncertainty when new evidence arrives.

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

print("Imports loaded.")

Imports loaded.

---

Problem 1: Bayes' Theorem — Manufacturing Defects

Setup

A graphics card is manufactured by three processes with the following prior probabilities and defect rates from the tree diagram:

Process	Prior P(Process)	P(Def | Process)	P(Not Def | Process)
A	0.5	0.03	0.97
B	0.3	0.02	0.98
C	0.2	0.04	0.96

Bayes' Theorem

Bayes' Theorem updates a prior probability using observed evidence:

[P(Process | Evidence) = (P(Process) × P(Evidence | Process)) / P(Evidence)]

The denominator [P(Evidence)] is the total probability of observing that evidence across all possible processes — calculated by summing each process's contribution:

[P(Evidence) = P(A) × P(Evidence|A) + P(B) × P(Evidence|B) + P(C) × P(Evidence|C)]

This is the law of total probability. It ensures the denominator accounts for every path through the tree that could have produced the observed outcome.

# ── Prior probabilities (from tree diagram) ──────────────────────────────
P_A = 0.5
P_B = 0.3
P_C = 0.2

# ── Conditional probabilities (from tree diagram) ────────────────────────
P_Def_given_A    = 0.03;  P_NotDef_given_A = 0.97
P_Def_given_B    = 0.02;  P_NotDef_given_B = 0.98
P_Def_given_C    = 0.04;  P_NotDef_given_C = 0.96

# ── Joint probabilities: each branch of the tree ─────────────────────────
# Multiply along each branch: P(Process) × P(Outcome | Process)
P_A_and_Def    = P_A * P_Def_given_A
P_B_and_Def    = P_B * P_Def_given_B
P_C_and_Def    = P_C * P_Def_given_C

P_A_and_NotDef = P_A * P_NotDef_given_A
P_B_and_NotDef = P_B * P_NotDef_given_B
P_C_and_NotDef = P_C * P_NotDef_given_C

print("Joint probabilities (branch × leaf):")
print(f"  P(A ∩ Def)     = {P_A:.1f} × {P_Def_given_A:.2f} = {P_A_and_Def:.4f}")
print(f"  P(B ∩ Def)     = {P_B:.1f} × {P_Def_given_B:.2f} = {P_B_and_Def:.4f}")
print(f"  P(C ∩ Def)     = {P_C:.1f} × {P_Def_given_C:.2f} = {P_C_and_Def:.4f}")
print(f"  P(A ∩ Not Def) = {P_A:.1f} × {P_NotDef_given_A:.2f} = {P_A_and_NotDef:.4f}")
print(f"  P(B ∩ Not Def) = {P_B:.1f} × {P_NotDef_given_B:.2f} = {P_B_and_NotDef:.4f}")
print(f"  P(C ∩ Not Def) = {P_C:.1f} × {P_NotDef_given_C:.2f} = {P_C_and_NotDef:.4f}")

Joint probabilities (branch × leaf):
  P(A ∩ Def)     = 0.5 × 0.03 = 0.0150
  P(B ∩ Def)     = 0.3 × 0.02 = 0.0060
  P(C ∩ Def)     = 0.2 × 0.04 = 0.0080
  P(A ∩ Not Def) = 0.5 × 0.97 = 0.4850
  P(B ∩ Not Def) = 0.3 × 0.98 = 0.2940
  P(C ∩ Not Def) = 0.2 × 0.96 = 0.1920

# ── Total probability of Defective and Not Defective ─────────────────────
# Sum down the defective column — the law of total probability
P_Def    = P_A_and_Def + P_B_and_Def + P_C_and_Def
P_NotDef = P_A_and_NotDef + P_B_and_NotDef + P_C_and_NotDef

print(f"Total P(Defective)     = {P_A_and_Def:.4f} + {P_B_and_Def:.4f} + {P_C_and_Def:.4f} = {P_Def:.4f}")
print(f"Total P(Not Defective) = {P_A_and_NotDef:.4f} + {P_B_and_NotDef:.4f} + {P_C_and_NotDef:.4f} = {P_NotDef:.4f}")
print(f"\nVerification (should sum to 1.0): {P_Def + P_NotDef:.4f}")

Total P(Defective)     = 0.0150 + 0.0060 + 0.0080 = 0.0290
Total P(Not Defective) = 0.4850 + 0.2940 + 0.1920 = 0.9710

Verification (should sum to 1.0): 1.0000

Part (a): P(A | Defective)

A randomly chosen card is defective. What is the probability it came from Process A?

Applying Bayes' Theorem:

[P(A | Def) = P(A ∩ Def) / P(Def) = (P(A) × P(Def|A)) / P(Def)]

# Part (a): P(A | Defective)
P_A_given_Def = P_A_and_Def / P_Def

print("Part (a): P(A | Defective)")
print(f"  Numerator:   P(A) × P(Def|A) = {P_A} × {P_Def_given_A} = {P_A_and_Def:.4f}")
print(f"  Denominator: P(Def)           = {P_Def:.4f}")
print(f"  P(A | Def)  = {P_A_and_Def:.4f} / {P_Def:.4f} = {P_A_given_Def:.6f}")
print(f"\n  Answer: P(A | Defective) ≈ {P_A_given_Def:.4f}  ({P_A_given_Def*100:.2f}%)")

Part (a): P(A | Defective)
  Numerator:   P(A) × P(Def|A) = 0.5 × 0.03 = 0.0150
  Denominator: P(Def)           = 0.0290
  P(A | Def)  = 0.0150 / 0.0290 = 0.517241

  Answer: P(A | Defective) ≈ 0.5172  (51.72%)

Interpretation: Even though Process A manufactures 50% of all cards, given that a card is defective, the probability it came from Process A is about 51.7%. That's above its prior share because Process A produces the largest absolute number of cards — so even with a low defect rate (3%), its volume means it contributes the most defective units overall. Bayes' Theorem balances both the prior (how many cards each process makes) and the likelihood (how often each process produces defects).

Part (b): P(C | Not Defective)

A randomly chosen card is not defective. What is the probability it came from Process C?

[P(C | Not Def) = P(C ∩ Not Def) / P(Not Def) = (P(C) × P(Not Def|C)) / P(Not Def)]

# Part (b): P(C | Not Defective)
P_C_given_NotDef = P_C_and_NotDef / P_NotDef

print("Part (b): P(C | Not Defective)")
print(f"  Numerator:   P(C) × P(Not Def|C) = {P_C} × {P_NotDef_given_C} = {P_C_and_NotDef:.4f}")
print(f"  Denominator: P(Not Def)            = {P_NotDef:.4f}")
print(f"  P(C | Not Def) = {P_C_and_NotDef:.4f} / {P_NotDef:.4f} = {P_C_given_NotDef:.6f}")
print(f"\n  Answer: P(C | Not Defective) ≈ {P_C_given_NotDef:.4f}  ({P_C_given_NotDef*100:.2f}%)")

Part (b): P(C | Not Defective)
  Numerator:   P(C) × P(Not Def|C) = 0.2 × 0.96 = 0.1920
  Denominator: P(Not Def)            = 0.9710
  P(C | Not Def) = 0.1920 / 0.9710 = 0.197734

  Answer: P(C | Not Defective) ≈ 0.1977  (19.77%)

Interpretation: Process C has a prior probability of 20% and a 96% non-defect rate. Given that a card is not defective, the posterior probability it came from Process C is about 19.77% — very close to, but slightly below, its prior. That small drop happens because Processes A and B have slightly higher non-defect rates (97% and 98%), so knowing a card is good tilts the probability very slightly toward them. The effect is subtle because the non-defect rates across all three processes are close together.

# Summary table — posterior probabilities for both parts
print("Full Posterior Summary")
print("=" * 55)
print(f"{'Process':<10} {'P(Process|Def)':>16} {'P(Process|Not Def)':>20}")
print("-" * 55)
for process, prior, p_def_giv, p_nd_giv in [
    ('A', P_A, P_Def_given_A, P_NotDef_given_A),
    ('B', P_B, P_Def_given_B, P_NotDef_given_B),
    ('C', P_C, P_Def_given_C, P_NotDef_given_C),
]:
    post_def    = (prior * p_def_giv) / P_Def
    post_notdef = (prior * p_nd_giv)  / P_NotDef
    print(f"{process:<10} {post_def:>16.6f} {post_notdef:>20.6f}")
print("=" * 55)

Full Posterior Summary
=======================================================
Process      P(Process|Def)   P(Process|Not Def)
-------------------------------------------------------
A                  0.517241             0.499485
B                  0.206897             0.302781
C                  0.275862             0.197734
=======================================================

---

Problem 2: Shannon Entropy Function

Shannon entropy measures the uncertainty in a probability distribution. A distribution where all outcomes are equally probable has maximum entropy — there is no way to predict which outcome will occur. A distribution where one outcome dominates has low entropy — the outcome is mostly predictable.

The formula from Hadrien (2021, p. 143):

[H(X) = -Σ pᵢ × log₂(pᵢ)]

Where [pᵢ] is the probability of outcome [i], and the sum runs over all outcomes. The negative sign flips the result positive — since [log₂(pᵢ)] is always negative or zero for valid probabilities (0 < p ≤ 1), the negative makes [H(X)] always non-negative.

Units are bits — entropy measured in base-2 logarithms.

Two edge cases to handle:

• [pᵢ = 0]: [0 × log₂(0)] is defined as 0 by convention (a zero-probability outcome contributes nothing to uncertainty).
• [pᵢ = 1]: [1 × log₂(1) = 0], so a certain outcome yields [H = 0] — no uncertainty.

def entropy(probs):
    """
    Calculates Shannon entropy of a discrete probability distribution.

    Formula (Hadrien, 2021, p. 143):
        H(X) = -sum(p_i * log2(p_i)) for all i

    Parameters
    ----------
    probs : array-like
        Probability values p1, p2, ..., pn for a discrete distribution.
        Values must be non-negative and should sum to 1.

    Returns
    -------
    float
        Shannon entropy in bits (base-2 logarithm).
    """
    probs = np.array(probs, dtype=float)

    # Filter out zero probabilities — log2(0) is undefined, but
    # the convention 0 * log2(0) = 0 means they contribute nothing
    probs = probs[probs > 0]

    return -np.sum(probs * np.log2(probs))

# ── Sanity checks ────────────────────────────────────────────────────────
# A fair coin: two equally probable outcomes → maximum entropy for 2 outcomes = 1 bit
H_fair_coin = entropy([0.5, 0.5])
print(f"Fair coin H([0.5, 0.5]) = {H_fair_coin:.4f} bits  (expected: 1.0000)")

# A certain outcome: all probability on one value → no uncertainty → H = 0
H_certain = entropy([1.0, 0.0])
print(f"Certain   H([1.0, 0.0]) = {H_certain:.4f} bits  (expected: 0.0000)")

# Uniform over 4 outcomes → H = log2(4) = 2 bits
H_uniform4 = entropy([0.25, 0.25, 0.25, 0.25])
print(f"Uniform-4 H([0.25×4])   = {H_uniform4:.4f} bits  (expected: 2.0000)")

Fair coin H([0.5, 0.5]) = 1.0000 bits  (expected: 1.0000)
Certain   H([1.0, 0.0]) = -0.0000 bits  (expected: 0.0000)
Uniform-4 H([0.25×4])   = 2.0000 bits  (expected: 2.0000)

---

Problem 3: Comparing Entropies of X and Y

Two random variables each take values 1 through 5:

Value	P(X = x)	P(Y = y)
1	0.2	0.1
2	0.2	0.4
3	0.2	0.1
4	0.2	0.3
5	0.2	0.1

X is uniform — every outcome is equally probable. Y is concentrated — outcomes 2 and 4 together carry 70% of the probability.

Part (a): Calculate H(X) and H(Y)

# Define distributions
p_X = [0.2, 0.2, 0.2, 0.2, 0.2]   # uniform over 5 outcomes
p_Y = [0.1, 0.4, 0.1, 0.3, 0.1]   # concentrated — Y=2 and Y=4 dominate

H_X = entropy(p_X)
H_Y = entropy(p_Y)

print(f"H(X) = {H_X:.6f} bits")
print(f"H(Y) = {H_Y:.6f} bits")
print(f"\nH(X) > H(Y): {H_X > H_Y}")
print(f"Difference: H(X) - H(Y) = {H_X - H_Y:.6f} bits")

# Verify H(X) = log2(5) for a uniform distribution over 5 outcomes
print(f"\nVerification: log2(5) = {np.log2(5):.6f}  (should match H(X))")

H(X) = 2.321928 bits
H(Y) = 2.046439 bits

H(X) > H(Y): True
Difference: H(X) - H(Y) = 0.275489 bits

Verification: log2(5) = 2.321928  (should match H(X))

Part (b): Compare H(X) and H(Y) — Intuitive Explanation

H(X) = 2.3219 bits > H(Y) = 2.0464 bits

X has higher entropy. The intuition follows directly from what entropy measures — uncertainty before observing an outcome.

For X, every outcome is equally likely at 20%. There is no way to make a better-than-random guess at what value X will take. Each outcome carries the same amount of surprise. That equal spread produces the maximum possible entropy for a five-outcome distribution — [H(X) = log₂(5) ≈ 2.3219] bits.

For Y, outcomes 2 and 4 together account for 70% of the probability. Before observing Y, a reasonable guess is already "probably 2 or 4." The distribution is already telling us something — and that advance information reduces entropy. Outcomes 1, 3, and 5 each carry only 10%, so they add less uncertainty to the mix. The result is lower entropy even though Y has the same five possible values as X.

The study guide framing from JDataLab (2017) helps here: entropy measures the average number of yes/no questions needed to identify the outcome. For X, every question eliminates exactly one outcome — five equally weighted options require the most questions on average. For Y, the first question "Is it 2 or 4?" eliminates 70% of the probability mass in one step — fewer questions needed on average, lower entropy.

# Visualize the two distributions and their entropies side by side
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
values = [1, 2, 3, 4, 5]

# X — uniform
axes[0].bar(values, p_X, color='#457B9D', edgecolor='white', width=0.6)
axes[0].axhline(y=0.2, color='tomato', linestyle='--', alpha=0.7, label='uniform baseline (0.2)')
axes[0].set_title(f'Distribution of X  |  H(X) = {H_X:.4f} bits', fontweight='bold')
axes[0].set_xlabel('Value')
axes[0].set_ylabel('Probability')
axes[0].set_ylim(0, 0.55)
axes[0].set_xticks(values)
axes[0].legend()
for v, p in zip(values, p_X):
    axes[0].text(v, p + 0.01, f'{p}', ha='center', fontsize=10)

# Y — concentrated
axes[1].bar(values, p_Y, color='#2A9D8F', edgecolor='white', width=0.6)
axes[1].axhline(y=0.2, color='tomato', linestyle='--', alpha=0.7, label='uniform baseline (0.2)')
axes[1].set_title(f'Distribution of Y  |  H(Y) = {H_Y:.4f} bits', fontweight='bold')
axes[1].set_xlabel('Value')
axes[1].set_ylabel('Probability')
axes[1].set_ylim(0, 0.55)
axes[1].set_xticks(values)
axes[1].legend()
for v, p in zip(values, p_Y):
    axes[1].text(v, p + 0.01, f'{p}', ha='center', fontsize=10)

plt.suptitle('Entropy Comparison: Uniform X vs. Concentrated Y', fontsize=13, fontweight='bold')
plt.tight_layout()
plt.show()

print(f"H(X) = {H_X:.6f} bits — maximum entropy for 5 outcomes (uniform)")
print(f"H(Y) = {H_Y:.6f} bits — lower entropy due to concentration at Y=2 and Y=4")
print(f"H(X) is {H_X - H_Y:.6f} bits higher than H(Y)")

H(X) = 2.321928 bits — maximum entropy for 5 outcomes (uniform)
H(Y) = 2.046439 bits — lower entropy due to concentration at Y=2 and Y=4
H(X) is 0.275489 bits higher than H(Y)

---

Summary

Problem	Result
P(A | Defective)	0.5172 (~51.72%)
P(C | Not Defective)	0.1977 (~19.77%)
H(X) — uniform distribution	2.3219 bits
H(Y) — concentrated distribution	2.0464 bits
Which is larger?	H(X) > H(Y) — uniform distributions always carry maximum entropy

All three problems connect through the idea of information and uncertainty. Bayes' Theorem updates probability estimates when evidence arrives — which is precisely what reduces entropy in a decision tree: each split selects the feature that most reduces uncertainty (maximizes information gain). Shannon entropy quantifies exactly how much uncertainty exists before that update happens.

---

References

Hadrien, J. (2021). Essential math for data science. O'Reilly Media.

JDataLab. (2017, February 22). Measuring uncertainty by calculating Shannon entropy. https://www.jdatalab.com/data_science_and_data_mining/2017/02/22/shannon-entropy.html