Sequential, Parallel, and Distributed Computing

Sequential computing does one step, then the next, then the next, so a task of n equal steps takes n time units. Parallel computing breaks the task into independent pieces that run at the same moment on different processors, so n steps split evenly across p processors takes roughly n divided by p time units. The speedup only works if the pieces truly don't depend on each other's results. If step 5 needs the answer from step 4, those two steps can't run at the same time no matter how many processors you have.

Parallel computing usually means multiple cores inside one machine, sharing memory and swapping data almost instantly. Distributed computing means separate machines, each with its own memory, passing data over a network instead. Distributed systems scale to far more workers and survive a single machine failing, but network communication is slower and less reliable than one machine talking to itself, so coordinating the pieces costs more time.

This connects to the rest of the unit because the same fault-tolerant, packet-based network that lets the internet survive a cut cable is what distributed processors use to send pieces of a job back and forth. It also explains a limit the unit keeps circling back to: real speedup is bounded not by hardware alone, but by how much of a task is genuinely parallel. The sequential portion of a task, and the overhead of splitting work and collecting results, both eat into the gains four workers were supposed to deliver.