Skip to main content
CSP LabBy Educator Today
AP CSP course

C15 · Topic 2.4 · 40 minutes · DAT-2.D, DAT-2.E

Cleaning and filtering data

By the end you can:

  • Document a cleaning policy.
  • Distinguish missing from zero.
  • Compute results using the valid denominator.

Before this lesson: topics 2.3, 3.10.

Cleaning
Applying documented rules to inconsistent, duplicate or missing data.
Join key
A field used to match records across datasets.

Understand

Programmes make data processing reproducible when rules are explicit. Decide how to handle duplicates, missing values and inconsistent formats before calculating. A missing measurement is unknown; replacing it with 0 asserts a particular value and can change a result. Preserve a log of exclusions so a summary does not hide who or what was removed.

Cleaning depends on metadata and the question. A duplicate ID is not automatically an error if several events per ID are intended, but it is a problem in this table’s one-row-per-trip contract. Excluding missing time from a mean need not mean deleting that trip from every other analysis.

Worked example

Rows A 12,B 18,C 24,C 24,D missing,E 48 minutes; IDs identify trips and only one row per trip is intended.

  1. Deduplicate C, and exclude D from calculations requiring a known time.
  2. Valid sum 12+18+24+48=102 and valid count 4 give mean 25.5.
  3. Keeping D for a separate missingness count preserves evidence that one unique trip lacks a time.

Explore each concept

Choose a concept, change its inputs and follow the explanation. Each example models the stated rule or scenario; completing it is practice rather than evidence of mastery.

  • Keys and unit conversion
  • Normalisation
  • Transformation and classification
  • Tool capacity and scaling
Open this example on its own

Change it · step through · explain

Change an input, predict what will happen, then advance one step. Read why the state changes, reset and experiment. This fixed teaching model does not save progress or award XP.

Ready · Make a prediction

Ready to explore: Join, transform, classify and scale data
  1. PipelinejoinWaiting
  2. Join keystable-idWaiting

Ready to explore: Join, transform, classify and scale data

Why: Change an input, predict the first effect, then advance one step.

Try this: Change one input or choose another listed case. Predict the effect, reset and compare the states and governing rule.

Step through and experiment

Change it · step through · explain

Change an input, predict what will happen, then advance one step. Read why the state changes, reset and experiment. This fixed teaching model does not save progress or award XP.

Ready · Make a prediction

Ready to experiment
  1. Input 1deduplicateWaiting
  2. Input 2excludeWaiting

Ready to experiment

Why: Predict the first change, then advance one step.

Try this: Change only one cleaning policy. Which numerator or denominator changes?

Check a misconception

Claim: Cleaning means deleting every record with any missing field from every analysis.

Choose a policy appropriate to each question and preserve what was excluded and why.

Guided practice

Calculate the mean if the duplicate is kept but missing time is excluded.

Work with a partner. Make a prediction, compare your reasoning, then identify the rule that supports your answer.

Independent transfer

Calculate the mean if the duplicate is removed but missing time is invented as 0.

Exit check

Why must a mean use the count of values actually included in its sum?

Write your reasoning before comparing with a classmate. If your teacher assigned this task, submit it through My classes to save your work and receive feedback.

Check yourself

  1. Question 1 of 2Under the documented cleaning rule, how many known unique trip times remain?

    Select one answer.

  2. Question 2 of 2Which TWO practices support reproducibility?

    Select exactly two answers (0 of 2 selected).

Your practice so far

Check an answer to see its rationale.

Teacher guidance and scoring