Outliers and Influential Points

Least-squares regression picks the line that minimizes the sum of squared residuals, and that minimizing process treats points unequally. A point far from the mean of x sits at the end of a long lever. Tilt the line even slightly near that point and its residual swings wildly, so the fitting process bends the whole line toward it to keep that one squared residual small. A point that's unusual in y but sits near the mean of x has no such lever arm — pulling the line toward it barely changes its distance from other points, so the line resists moving much.

This is why leverage, not strangeness, decides influence. A point can be a dramatic outlier in y, sitting far above or below the cloud, and still have little effect on the slope if its x-value is ordinary — the line just absorbs it as one more residual among many. Influence specifically means the slope or intercept would change substantially if that point were deleted. Some high-leverage points are not influential either, if their y-value happens to already fit the trend; they pull on a line that was already headed their way.

This sets up residual plots and the idea of checking a fit: an influential point can make r and the slope look stronger or weaker than the bulk of the data actually supports, so any regression summary needs a second look at individual points, not just the overall line.