PCA and SVD

1 · Projection

Keeping one number per point

Center the data (subtract the mean). Choose a unit direction \(u\). Each point \(x\) keeps one number, its coordinate along \(u\): \(z = x \cdot u\). Its reconstruction is \(z\,u\), and what's lost is the residual \(x - z\,u\), which is perpendicular to \(u\).

Two goals: make the kept coordinates \(z\) as spread out as possible, or make the residuals as small as possible. Do they pick the same direction?

The residual is perpendicular to \(u\), so each point splits by Pythagoras:

\[ \lVert x\rVert^2 = \underbrace{(x\cdot u)^2}_{\text{kept}} + \underbrace{\lVert x - (x\cdot u)u\rVert^2}_{\text{lost}} \]

The left side doesn't depend on \(u\). Every bit of variance you keep is a bit of error you don't make, so maximizing one minimizes the other, for any data.

2 · Play it

Find the axis

Drag anywhere on the plot to rotate the line. Blue dots are the kept coordinates; dashed red segments are what's lost. Keep as much variance as possible: within 0.5% of the best.

kept: projection onto the linelost: residual

3 · The answer

The best axis is an eigenvector

With centered data \(X\) (one row per point), the variance along a unit direction \(u\) is

\[ \operatorname{Var}(Xu) = \frac1n \lVert Xu\rVert^2 = u^\top \Sigma\, u, \qquad \Sigma = \frac1n X^\top X \]

Maximizing \(u^\top\Sigma u\) over unit vectors (a Rayleigh quotient) gives the top eigenvector of the covariance \(\Sigma\), and the variance it keeps is the top eigenvalue \(\lambda_1\). The next best direction, perpendicular to the first, is the second eigenvector, and so on. Those directions are the principal components.

A 2-D covariance has eigenvalues 4 and 1. What fraction of the total variance does the first principal component keep?

Total variance is the trace, \(\lambda_1 + \lambda_2 = 5\). PC1 keeps \(\lambda_1/(\lambda_1+\lambda_2) = 80\%\), and the residual error is exactly the discarded eigenvalue, 1. In \(d\) dimensions, keeping \(k\) components keeps \(\sum_{i\le k}\lambda_i / \sum_i \lambda_i\).

4 · SVD and low rank

The same thing, computed directly from the data

Any matrix factors as \(X = U S V^\top\): orthonormal columns in \(U\) and \(V\), and non-negative singular values on the diagonal of \(S\). Substitute it into the covariance:

\[ \Sigma = \frac1n X^\top X = V\,\frac{S^2}{n}\,V^\top \]

The columns of \(V\) are the principal directions, the eigenvalues are \(s_i^2/n\), and \(US\) holds each point's coordinates. In practice PCA is computed with the SVD of \(X\) rather than an eigendecomposition of \(X^\top X\). Forming \(X^\top X\) squares the condition number and throws away precision.

Keeping only the top \(k\) singular values gives the best rank-\(k\) approximation of \(X\) in squared error (Eckart–Young), and the error is exactly what you dropped:

\[ \min_{\operatorname{rank}(A)\le k} \lVert X - A\rVert_F^2 = \sum_{i>k} s_i^2 \]

A 4096 × 4096 weight matrix is fine-tuned with LoRA at rank 16: \(\Delta W = BA\) with \(B\): 4096 × 16 and \(A\): 16 × 4096. What fraction of the parameters of a full update does it train?

\(2 \times 4096 \times 16 = 131{,}072\) against \(4096^2 = 16{,}777{,}216\): 0.78%. LoRA bets that the change a fine-tune needs has low rank. Matrix factorization for recommendations makes the same bet about the user–item matrix: a few latent factors explain most preferences. Low rank is a recurring structural prior.

5 · How many to keep

Signal, then noise

Here are 20-dimensional data generated from 3 hidden factors plus noise in every dimension. The eigenvalues of its covariance, largest first:

Eigenvalues (bars) and cumulative variance kept (line)

noise level

Computed in your browser: 500 samples, sample covariance, Jacobi eigendecomposition.

Three eigenvalues stand out, then a flat floor of noise. Turn the noise up and the floor rises until the third factor sinks into it. Rules like "keep 95% of the variance" are heuristics. The better question is what the components are for: plotting (2–3), compression (bytes vs error), or denoising (stop where the signal meets the noise floor).

6 · New cases, no hints

Apply it somewhere else

Your features are revenue in dollars (spread in the thousands) and click rate (spread about 0.01). You run PCA on the raw features. What is PC1?

PCA maximizes variance, and variance depends on units: revenue's variance is millions of times larger. Standardize (PCA on the correlation matrix) unless the units are genuinely comparable. It's the same issue as k-means in lesson 6.

You fit PCA on the full dataset, then split into train and test to evaluate a model built on the components. What's wrong?

Unsupervised fitting still leaks the test distribution (its mean, its directions) into training. It's usually small, but it's real and can be large with few samples. Fit every preprocessing step on training data only, inside the cross-validation loop. Lesson 11 is about exactly this kind of mistake.

You factorize a user × item click matrix with SVD, filling every missing entry with 0. What's the conceptual problem?

Most zeros mean "never shown", not "not wanted" (lesson 9). Use weighted matrix factorization (low confidence on unobserved entries, as in implicit-feedback ALS), or fit only the observed entries with regularization. Evaluate by ranking held-out interactions, not by reconstruction error.

7 · Implement, unaided

PCA from the SVD

numpy's np.linalg.svd is allowed; sklearn isn't. ⌘/Ctrl+Enter runs.

Exercise 1: PCA

Exercise 2: best rank-k approximation

8 · Staff-level follow-ups

Answer out loud, then check

What does PCA preserve, and what does it discard?

It keeps the directions of largest variance, which gives the best linear reconstruction in squared error, and the Euclidean geometry within that subspace.
It discards everything in the low-variance directions, even if that's where the label signal lives (unsupervised). It can't capture nonlinear structure.
It's sensitive to scale and outliers. Variance isn't importance.

PCA vs an autoencoder vs t-SNE/UMAP: when would you use each?

PCA: linear, fast, deterministic, invertible, and reusable on new data. A good first baseline for compression and denoising.
A linear autoencoder with squared loss learns the same subspace as PCA. Nonlinear autoencoders can capture curved structure but need training and tuning.
t-SNE/UMAP: for 2-D visualization of local neighborhoods. Distances between far clusters and cluster sizes aren't meaningful, so don't use them as features without care.

Why compute PCA with SVD instead of eigendecomposing XᵀX, and how do you scale it?

Forming \(X^\top X\) squares the condition number, so small singular values lose precision. SVD works on \(X\) directly.
At scale: randomized SVD (random projections, then a small exact SVD), incremental or streaming PCA, or distributed Gram matrices when \(d\) is small.
For only the top few components, power or Lanczos iterations cost roughly one matrix-vector product per iteration.

Why does LoRA work, and what rank would you pick?

The empirical finding: fine-tuning updates have low intrinsic rank. \(\Delta W = BA\) with small \(r\) trains about \(r(d+k)\) parameters instead of \(dk\), and can be merged into \(W\) for inference at no extra cost.
Rank is task-dependent. Small ranks (4–16) often suffice for style or format adaptation; knowledge-heavy or very different domains may need more, or full fine-tuning.
Pick it by sweeping rank against held-out quality and memory. Also decide which matrices to adapt: attention only, or the MLP too.

Sources: Eckart & Young, The approximation of one matrix by another of lower rank (1936); Halko, Martinsson & Tropp, Finding structure with randomness (2009); Hu et al., LoRA (2021); Hu, Koren & Volinsky, Collaborative Filtering for Implicit Feedback Datasets (ICDM 2008).

PCA and SVD: what a projection keeps