Clustering, Classifying, and the Scale of Big Data

A data set grows past what one computer can handle for simple reasons: limited memory, limited processing speed, and limited time. A million records might fit on a laptop. A billion transactions, images, or sensor readings will not, at least not in any reasonable time. Once that happens, the only way forward is to split the work across many machines running at once, which is called parallel or distributed computing.

Clustering is how a computer finds structure in data with no help from a human. Feed it ten thousand unlabeled customer records, and a clustering algorithm looks at patterns, like spending habits or age, and groups similar records together into segments. Nobody told it what the segments should be in advance; it discovers them from similarity alone. Classifying is different: it starts from labeled examples. Show an algorithm thousands of past emails already marked spam or not spam, and it learns which patterns, like certain words or sender addresses, tend to go with each label. Then it applies that pattern to sort new incoming email into the same two piles.

Both tasks scale badly as data grows, which connects straight back to distributed computing. Splitting n records evenly across k machines gives each machine n divided by k records to process, so the work per machine shrinks as more machines join in. But splitting the data also means combining partial results afterward, coordinating machines, and tolerating the fact that any one machine might fail. This tradeoff between speed gained and coordination cost is the real story behind why big data needs big, distributed systems rather than one very powerful computer.