C64 · Topic 5.3 · 40 minutes · IOC-1.D
Testing a proposed mitigation
By the end you can:
- Use each group’s denominator.
- Compare aggregate and group outcomes.
- Explain trade-offs and remaining uncertainty.
Before this lesson: topics 2.3, 5.1.
- Computing bias
- Systematic differences arising through data, design or use of a computing system.
- Error rate
- Errors divided by the relevant number of evaluated cases.
Understand
An aggregate accuracy can hide unequal outcomes because groups may have different sizes. Compute the relevant denominator for each group and the weighted overall result rather than averaging group percentages without considering counts. Compare both the level and distribution of errors when evaluating a proposed change.
A mitigation may improve one group while worsening another or leaving an overall metric unchanged. Explain those effects without asserting that one statistic proves fairness on every dimension. Use evidence about the identified mechanism and relevant outcomes, and state where uncertainty or missing data remains.
Worked example
Group A has 20 cases/8 errors;B has 80 cases/4 errors. Proposed errors become 4 and 8 with the same group sizes.
- Baseline accuracies are 60%and 95%; overall 88%.
- Proposed accuracies are 80%and 90%; overall remains 88%.
- The gap falls from 35 to 10 percentage points, but B accuracy falls and the causal mechanism is still not established by these counts alone.
Explore each concept
Choose a concept, change its inputs and follow the explanation. Each example models the stated rule or scenario; completing it is practice rather than evidence of mastery.
- Sampling, labels, rules and deployment
- More data versus better representation
- Evaluate group outcomes after change
Change it · step through · explain
Change an input, predict what will happen, then advance one step. Read why the state changes, reset and experiment. This fixed teaching model does not save progress or award XP.
Ready · Make a prediction
- Bias mechanismsamplingWaiting
- Proposed responsemore-of-same-dataWaiting
Ready to explore: Locate bias before choosing a mitigation
Why: Change an input, predict the first effect, then advance one step.
Try this: Change one input or choose another listed case. Predict the effect, reset and compare the states and governing rule.
Step through and experiment
Change it · step through · explain
Change an input, predict what will happen, then advance one step. Read why the state changes, reset and experiment. This fixed teaching model does not save progress or award XP.
Ready · Make a prediction
- Input 120,80Waiting
- Input 28,4Waiting
Ready to experiment
Why: Predict the first change, then advance one step.
Try this: Change group sizes while keeping very different error rates. Does a high overall accuracy hide a gap?
Check a misconception
Claim: Equal overall accuracy means the two systems affect groups identically.
The distribution of errors can change while the total remains the same.
Guided practice
Show the baseline overall calculation.
Work with a partner. Make a prediction, compare your reasoning, then identify the rule that supports your answer.
Independent transfer
Evaluate the proposal with at least one benefit, one cost and one limitation.
Exit check
What input check should a rate tool make before calculating?
Write your reasoning before comparing with a classmate. If your teacher assigned this task, submit it through My classes to save your work and receive feedback.
Check yourself
Your practice so far
Check an answer to see its rationale.