← Linear Algebra

Unit 5: Orthogonality, Projections, and Least Squares

Dropping a perpendicular onto a subspace, and why that single idea is the best-fit answer to a system with no exact solution.

12 topics · 44 questions

Each topic is a path, not a menu: work down it in order and every question builds on the one above. Click any of them to watch the answer draw itself.

Orthogonal Vectors

  1. How it worksHow does the Pythagorean theorem fall straight out of the dot product formula for perpendicular vectors?
  2. Worked exampleShow that the vectors (3, 4) and (4, -3) are orthogonal.
  3. Going furtherWhat happens to the dot-product test for orthogonality once you move into 3-space or higher?
Your turn1 of 2
The vectors (4, 3) and (3, -4) are orthogonal for any a and b. Find the length of the first vector.

Orthogonal Complements of a Subspace

  1. How it worksHow does the orthogonal complement collect every vector perpendicular to a whole subspace, not just one vector in it?
  2. Worked exampleFind the orthogonal complement of the line spanned by (1, 2) in the plane.
  3. Common mistakeWhy isn't the orthogonal complement of a plane in 3-space also a plane?
Your turn1 of 2
The orthogonal complement of the line spanned by (4, 3) in the plane is spanned by (-b, a). Find the length of this complementary direction vector.

Orthogonality of the Four Subspaces

  1. How it worksHow does every equation in Ax = 0 say that x is perpendicular to one row of A?
  2. Common mistakeWhy doesn't the column space end up orthogonal to the null space, the way the row space does?
  3. Ties togetherHow does this orthogonality split the input space and the output space each into two perpendicular pieces?
Your turn1 of 2
A matrix has a row (4, 2, 7) (so any null-space vector (x, y, z) must satisfy ax + by + cz = 0). If a null-space vector has y = -4 and z = -7, find x.

Orthonormal Bases

  1. How it worksHow does finding a vector's coordinates in an orthonormal basis collapse into simple dot products, with no system to solve?
  2. Worked exampleFind the coordinates of the vector (3, 5) in the orthonormal basis (0.6, 0.8) and (-0.8, 0.6).
  3. Common mistakeWhy isn't every orthogonal basis automatically an orthonormal one?
Your turn1 of 2
The vectors (cos(3.89), sin(3.89)) and (-sin(3.89), cos(3.89)) form an orthonormal basis of the plane. Find the coordinate of the vector (3, 8) along the first basis vector.

Projecting a Vector onto a Line

  1. How it worksHow does dropping a perpendicular from a vector down onto a line locate its projection?
  2. Worked exampleFind the projection of the vector (3, 4) onto the line spanned by (1, 0).
  3. Common mistakeWhy isn't the projection of a vector onto a line just the vector scaled down to fit it?
Your turn1 of 2
Find the length of the projection of the vector (4, 3) onto the line spanned by (8, 3).

Projecting a Vector onto a Subspace

  1. How it worksHow does the projection formula built from A transpose A generalize dropping a perpendicular onto any subspace?
  2. Worked exampleFind the projection of the vector (1, 1, 1) onto the plane spanned by (1, 0, 0) and (0, 1, 0).
  3. Worked exampleFind how far the vector (2, 3, 4) sits from its shadow on the plane spanned by (1, 1, 0) and (0, 1, 1).
  4. Going furtherWhat happens to the projection formula once the columns of A are already orthonormal?
Your turn1 of 2
Find the length of the projection of the vector (4, 3, 8) onto the plane spanned by (1, 0, 0) and (0, 1, 0).

The Projection Matrix

  1. How it worksHow does the matrix P equal A times the inverse of A transpose A times A transpose always find a vector's shadow?
  2. Worked exampleFind the projection matrix onto the line spanned by the vector (1, 2).
  3. Common mistakeWhy does applying a projection matrix twice in a row land you on the exact same answer as applying it once?
  4. Going furtherWhat happens to the projection matrix P when the subspace you're projecting onto is the entire space?
Your turn1 of 2
Find the (1,1) entry of the projection matrix onto the line spanned by (4, 3).

Least Squares and the Normal Equations

  1. How it worksHow does multiplying both sides of Ax = b by A transpose turn an unsolvable system into a solvable one?
  2. Worked exampleFind the normal equations for the inconsistent system x = 1, x = 2, x = 4.
  3. Common mistakeWhy doesn't solving the normal equations give you back an exact solution to the original system?
Your turn1 of 2
Find the right-hand side of the normal equation for the inconsistent system x = 4, x = 3, x = 9.

Fitting a Line by Least Squares

  1. How it worksHow does minimizing the sum of squared vertical errors pick out one specific line from all the possible lines?
  2. Worked exampleFind the least squares line fitting the points (0, 1), (1, 1), and (2, 3).
  3. Common mistakeWhy doesn't minimizing the errors directly, instead of their squares, give the same best-fit line?
  4. Going furtherWhat happens to the least squares line once one of the data points is a huge outlier?
  5. Ties togetherHow does the error between the data points and the best-fit line show up as a vector orthogonal to the column space?
Your turn1 of 2
Fit a least squares line y = mx + b through the three points (0, 4), (1, 3), and (2, 9). Find the slope m.

Gram-Schmidt Orthogonalization

  1. How it worksHow does subtracting off the projection onto the earlier vectors straighten each new vector one at a time?
  2. Worked exampleFind an orthogonal basis from the vectors (1, 1) and (1, 0) using Gram-Schmidt.
  3. Worked exampleFind an orthogonal set built from the three vectors (1, 1, 0), (1, 0, 1), and (0, 1, 1) using Gram-Schmidt.
  4. Common mistakeWhy doesn't Gram-Schmidt ever change the span of the original set of vectors?
  5. Going furtherWhat happens during Gram-Schmidt when one of the input vectors already lies in the span of the earlier ones?
Your turn1 of 2
Using Gram-Schmidt on the vectors (6, 2) and (7, -4) (keeping the first vector unchanged as v1), find the first entry of the orthogonalized second vector: v2 = w2 minus its projection onto v1.

QR Factorization

  1. How it worksHow does Q hold the orthonormal vectors from Gram-Schmidt while R records how the columns built from them?
  2. Worked exampleFind the QR factorization of the matrix [[1, 1], [1, 0]].
  3. Common mistakeWhy doesn't the R matrix in a QR factorization end up with zeros anywhere on its diagonal?
  4. Ties togetherHow does QR factorization turn solving a least squares problem into a much easier triangular system?
Your turn1 of 2
The matrix with columns (7, 6) and (7, -4) has QR factorization A = QR. Find the (1,1) entry of R — the length of the first column.

Orthogonal Matrices

  1. How it worksHow does having orthonormal columns force a matrix's transpose to equal its own inverse?
  2. Worked exampleShow that the rotation matrix [[0, -1], [1, 0]] is orthogonal.
  3. Common mistakeWhy doesn't an orthogonal matrix have to look like a rotation — a reflection counts too?
  4. Going furtherWhat happens to the length of a vector once it's multiplied by an orthogonal matrix?
Your turn1 of 2
A vector (4, 3) is multiplied by the rotation matrix [[cos(5.14), -sin(5.14)], [sin(5.14), cos(5.14)]]. Find the length of the resulting vector.