AP® Computer Science A · Unit 4 · Topic 4.1

Data Collection, Privacy, and Bias

A data set is never automatically fair just because it is large or machine-collected — it reflects whoever and whatever its collection process happened to include, and leaving groups out produces programs that work worse for them.

A data set looks complete when you're staring at rows and rows of entries, but completeness is about coverage, not row count. Every data set comes from some collection process — a sensor, a survey, an app's sign-up form — and that process always points at some population and not another. If the process systematically reaches certain people more than others, the data inherits that imbalance no matter how many entries pile up. A face-detection app trained on photos scraped mostly from one source will see mostly the people who used that source.

This becomes a program problem once you remember how these systems learn: they find patterns in whatever examples they're given. If 90 percent of training images show one demographic, the program gets heavy practice recognizing patterns common to that group and comparatively little practice on the rest. The output isn't a neutral tool — it's a tool that performs well for the 90 percent and noticeably worse for everyone else, because its 'knowledge' is really just a reflection of its training data's proportions. A machine did the sorting, but the humans who built the collection process are why the skew exists.

This connects to the unit's bigger theme of collections holding real data about real people. Arrays and ArrayLists are neutral structures — they'll happily store whatever you put in them — but the question of what you put in them, and who agreed to be in there, sits outside the code. A data set built from people who never consented to be tracked raises a separate problem: even an accurate, unbiased sample can be an ethical one only if the people inside it had a say.

Key terms

Sampling bias
A systematic skew in a data set caused by a collection process that overrepresents or underrepresents certain groups.
Underrepresented group
A group that appears in a data set far less often than its actual share of the real-world population.
Training data
The data set a program learns patterns from before being used on new, unseen cases.
Consent
Agreement from the people being recorded or counted to have their data collected and used.

Watch it explained

  1. How it worksHow does a skewed sample turn into a program that treats some users worse than others?
  2. Worked exampleTrace how a data set that is 90 percent one group shifts a program's outputs.
  3. Common mistakeWhy isn't data neutral just because a machine was the thing that collected it?
  4. Going furtherWhat changes about a data set when the people inside it never agreed to be counted?

Check yourself

Your turn1 of 2
Why isn't data neutral just because a machine was the thing that collected it?

End of topic.