Filtering and Transforming Data with a Program

Filtering and transforming are both operations that process a data set one record at a time, but they do different jobs. A filter asks a yes-or-no question of each record, something like "is this value greater than 5?", and builds a new, usually shorter list out of only the records that answer yes. The records themselves never change; some just get dropped. A transform, by contrast, touches every single record and changes it in some way, like doubling every number, but keeps the same number of records you started with. This is why filtering and sorting are so different too: sorting rearranges every record without removing or changing any of them, while filtering removes records outright based on a test.

Because filtering shrinks the list and transforming changes the values, the order you apply them in can change the final answer. If you transform first and then filter, the filter test is applied to the new, changed values. If you filter first and then transform, the filter test is applied to the old, original values, which can let in or throw out different records than you intended. Once a record is filtered out, it is gone for good, so an early filter based on the wrong values can permanently discard something that should have survived.

This matters because real programs chain these steps together constantly, such as cleaning a spreadsheet of sensor readings before graphing it. Getting the order backwards is a quiet bug: the program still runs and still produces a list, just the wrong one, which makes this an easy mistake to miss without checking a small example by hand.