Residuals

Every regression line makes a prediction, y-hat, for each x-value. But real data rarely sits exactly on the line. The residual captures that gap: residual = observed y - predicted y-hat. If the point sits above the line, the residual is positive, the line underestimated. If the point sits below the line, the residual is negative, the line overestimated. A residual of zero means the line nailed that point exactly.

The direction matters as much as the size. Residuals are always measured vertically, straight up or down from the point to the line, because y-hat is a prediction of y based on x, and we care about how far off that prediction was in the y-direction. The shortest distance from a point to the line would usually cut across at an angle, but that distance mixes up errors in x and y in a way that doesn't match what the line is actually predicting. Only the vertical gap isolates the prediction error.

Residuals are where the real diagnostic work of regression happens. A least-squares line is chosen specifically to make the sum of squared residuals as small as possible, so every residual is a piece of that minimization. Plotting all the residuals against x, in a residual plot, is how you check whether a straight line was ever the right model: patternless scatter around zero means the line fits well, while curves or fans in the residuals expose problems the scatterplot alone might hide.