Dimensionality Reduction
Images can contain thousands or millions of numerical features. In this codelab, we will learn why too many features can become a problem and how PCA, LDA, and ICA create smaller, more useful representations.
↓
Many features
↓
High-dimensional feature space
↓
Dimensionality reduction
↓
Smaller useful representation
↓
Faster analysis / visualization / classification
Previously, we represented objects using feature vectors. Now we ask an important question: what happens when that feature vector becomes extremely large?
What Does “Dimension” Mean?
In machine learning, each feature usually represents one dimension of the feature space.
A tiny image example
Consider a grayscale image of size 3 × 3:
10 12 11
50 52 51
90 91 92If we flatten it into a feature vector:
X = [10, 12, 11, 50, 52, 51, 90, 91, 92]There are 9 values, therefore this simple representation has 9 dimensions.
10,000 pixel features → 10,000 dimensions.
More than 6 million channel values if every pixel channel is used directly.
More features means a higher-dimensional feature space.
Curse of Dimensionality
As the number of dimensions increases, data becomes harder to represent, learn from, compare, and visualize.
Why is it called a “curse”?
Imagine dividing every dimension into only 10 possible regions.
| Dimensions | Possible regions |
|---|---|
| 1D | 10 |
| 2D | 10² = 100 |
| 3D | 10³ = 1,000 |
| 10D | 10¹⁰ = 10,000,000,000 |
The space grows extremely quickly, but the number of training samples usually does not. The data therefore becomes sparse.
Main problems
Samples occupy only a tiny portion of the huge feature space.
More values must be stored, processed, compared, and learned.
A model can learn accidental patterns when the number of features is large compared with the number of samples.
In very high dimensions, distances can become less useful for distinguishing samples.
What is one major effect of increasing dimensionality?
Feature Reduction
Feature reduction means representing the data using fewer features while trying to preserve the information that matters.
[x₁, x₂, x₃, x₄, x₅, ... x₁₀₀₀₀]
↓ feature reduction
[z₁, z₂, z₃, ... z₁₀₀]
Why can features be reduced?
Many features may be redundant, correlated, noisy, or unimportant.
If a dataset stores a person's height in centimeters, meters, and millimeters, those three columns mostly represent the same underlying information.
Two common approaches
Keep some of the original features and remove others.
x1 x2 x3 x4 x5
↓
x1 x3 x5Create new features by transforming or combining the original features.
x1 x2 x3 x4 x5
↓ transform
z1 z2Selection chooses old features. Extraction creates new features. PCA, LDA, and ICA are mainly feature-extraction techniques.
Principal Component Analysis
PCA finds new directions in the data where the samples vary the most.
“Which directions contain the largest amount of variation in my data?”
Imagine height and weight
Weight ↑
80 | ●
75 | ●
70 | ●
65 | ●
60 | ●
+--------------------------→ HeightHeight and weight are correlated. Most of the variation lies along one diagonal direction.
↓ PCA finds a new direction
PC1 = direction of maximum variance
↓
Project points onto PC1
↓
2 dimensions → possibly 1 useful dimension
A principal component is a new feature formed as a weighted combination of the original features.
How PCA Works
PCA centers the data, measures how features vary together, finds important directions, and projects the data onto those directions.
Step 1 — Center the data
X_centered = X - mean(X)The mean is moved to approximately zero so PCA focuses on variation around the center.
Step 2 — Compute covariance
covariance_matrix = covariance(X_centered)Covariance tells us how features change together.
Step 3 — Find eigenvectors and eigenvalues
| Term | Intuition |
|---|---|
| Eigenvector | Which direction? |
| Eigenvalue | How much variance exists in that direction? |
Step 4 — Sort and keep the strongest components
PC1 → largest eigenvalue
PC2 → second largest eigenvalue
PC3 → third largest eigenvalue
...Step 5 — Project the original data
↓ choose top principal components
Projection
↓
Lower-dimensional Z
Explained Variance
Explained variance tells us how much of the dataset's variation is captured by each principal component.
If the eigenvalues correspond to:
| Component | Explained variance |
|---|---|
| PC1 | 70% |
| PC2 | 20% |
| PC3 | 7% |
| PC4 | 3% |
Then the first two components preserve:
We can reduce 4 dimensions to 2 dimensions while keeping about 90% of the variance.
explained_variance_ratio = eigenvalue_i / sum(all_eigenvalues)What does PCA try to preserve?
PCA with Python
Use scikit-learn to reduce a simple four-feature dataset to two principal components.
import numpy as np
from sklearn.decomposition import PCA
X = np.array([
[2.5, 2.4, 1.2, 0.8],
[0.5, 0.7, 0.3, 0.2],
[2.2, 2.9, 1.1, 0.9],
[1.9, 2.2, 0.9, 0.7],
[3.1, 3.0, 1.5, 1.0],
[2.3, 2.7, 1.0, 0.8]
])
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)
print("Original shape:", X.shape)
print("Reduced shape:", X_reduced.shape)
print("Explained variance:", pca.explained_variance_ratio_)↓ PCA(n_components=2)
6 samples × 2 principal components
Important methods
| Code | Meaning |
|---|---|
PCA(n_components=2) | Keep two principal components. |
fit_transform(X) | Learn the PCA directions and transform the data. |
explained_variance_ratio_ | Shows how much variance each component preserves. |
PCA for Images and Eigenfaces
An image can be flattened into a long vector, and PCA can compress that vector into a much smaller set of components.
↓ flatten
10,000 pixel features
↓ PCA
100 principal components
↓
Compact face representation
Eigenfaces
When PCA is applied to a collection of face images, the principal directions can be visualized as face-like patterns often called eigenfaces.
Historically, eigenfaces were used for face representation and recognition. PCA can also be useful for image compression, visualization, noise reduction, and preprocessing.
PCA does not use class labels. It only cares about variance. The direction with maximum variance is not always the direction that best separates classes.
Want to write code, inspect eigenfaces, and compress real images? Check out the companion practical codelab: Implement Dimensionality Reduction using PCA for Images →
Linear Discriminant Analysis
LDA finds directions that make different classes easier to separate.
“Which direction separates my known classes the best?”
Suppose the data has two labeled classes:
Class A: ● ● ● ●
○ ○ ○ ○ : Class BLDA looks for a projection where:
● ● ● ● ○ ○ ○ ○
compact Class A compact Class B
← large separation →How spread out the samples are inside the same class. LDA tries to make this smaller.
How far the class centers are from each other. LDA tries to make this larger.
LDA = maximize between-class separation and minimize within-class spread.
How LDA Works
Unlike PCA, LDA needs class labels because its objective explicitly depends on class separation.
↓
Calculate class means
↓
Measure within-class scatter Sᵂ
↓
Measure between-class scatter Sᵇ
↓
Find directions maximizing separation
↓
Project data onto discriminant components
Objective intuition
Large distance between class centers
------------------------------------
Small spread inside each classImportant limitation
If there are C classes, classical LDA can produce at most C - 1 discriminant components.
| Classes | Maximum LDA components |
|---|---|
| 2 | 1 |
| 3 | 2 |
| 10 | 9 |
LDA with Python
Use the Iris dataset to reduce four input features to two discriminant components using known class labels.
from sklearn.datasets import load_iris
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
iris = load_iris()
X = iris.data
y = iris.target
lda = LinearDiscriminantAnalysis(n_components=2)
X_reduced = lda.fit_transform(X, y)
print("Original shape:", X.shape)
print("Reduced shape:", X_reduced.shape)fit_transform(X, y) receives both the features X and labels y. This is the key difference from PCA.
Why is LDA supervised?
Independent Component Analysis
ICA tries to separate observed data into statistically independent underlying source signals.
The cocktail-party problem
Imagine two people speaking at the same time while two microphones record mixtures of both voices.
Person A ─┐ ┌→ Microphone 1 = mixture
├→ room mixing ┤
Person B ─┘ └→ Microphone 2 = mixturex1 = 0.7 * voice_A + 0.3 * voice_B
x2 = 0.4 * voice_A + 0.6 * voice_BICA tries to recover the original independent sources:
↓ ICA
Estimated independent components
↓
Source A + Source B
Independence. ICA searches for components that are as statistically independent as possible.
ICA Model and Practical
ICA assumes that the observed signals are mixtures of hidden independent sources.
X = A · S| Symbol | Meaning |
|---|---|
X | Observed mixed signals. |
S | Unknown independent source signals. |
A | Unknown mixing matrix. |
ICA tries to estimate an unmixing matrix W so that:
S_estimated = W · XPython example
import numpy as np
from sklearn.decomposition import FastICA
X = np.array([
[1.0, 1.5],
[2.0, 1.0],
[3.0, 2.5],
[4.0, 2.0],
[5.0, 3.5]
])
ica = FastICA(n_components=2, random_state=0)
S = ica.fit_transform(X)
print(S)PCA creates uncorrelated components. ICA aims for the stronger condition of statistical independence.
PCA vs LDA vs ICA
All three create new representations, but they optimize very different objectives.
| Property | PCA | LDA | ICA |
|---|---|---|---|
| Full name | Principal Component Analysis | Linear Discriminant Analysis | Independent Component Analysis |
| Main goal | Maximum variance | Maximum class separation | Independent components |
| Uses labels? | No | Yes | No |
| Learning style | Unsupervised | Supervised | Unsupervised |
| Component property | Orthogonal / uncorrelated directions | Discriminative directions | Statistically independent sources |
| Typical use | Compression, visualization, preprocessing | Classification preprocessing | Source separation |
Find the directions where the data varies most.
Find the directions where labeled classes separate best.
Find independent hidden factors or signals.
LDA = Labels
ICA = Independence
One Face Dataset, Three Different Goals
The best way to understand the difference is to imagine applying all three techniques to the same face dataset.
Dataset
↓
10,000 pixel features
↓
Need a smaller representation
Finds directions that explain the largest variations across all face images. It does not care who each person is.
Uses identity labels and searches for directions that best separate Person A, Person B, Person C, and so on.
Searches for statistically independent underlying patterns or source-like factors in the observations.
You have labeled images of three people and want a projection that separates their identities. Which method directly targets that goal?
Dimensionality Reduction Revision
Review the core ideas before moving to non-parametric methods and clustering.
Completion checklist
- I understand what a dimension means in feature space.
- I can explain the curse of dimensionality.
- I can distinguish feature selection and feature extraction.
- I can explain PCA using variance, eigenvectors, and eigenvalues.
- I understand explained variance.
- I can explain why LDA is supervised.
- I know the C − 1 component limit of LDA.
- I understand ICA and blind source separation.
- I can compare PCA, LDA, and ICA.
Practice
- A grayscale image is 200 × 200 pixels. If every pixel is one feature, how many dimensions does it have?
- Give one example of redundant features in a real-world dataset.
- Explain feature selection and feature extraction using your own example.
- If PCA components explain 60%, 25%, 10%, and 5% variance, how much is preserved by the first two components?
- A dataset has 5 labeled classes. What is the maximum number of classical LDA components?
- For each scenario, choose PCA, LDA, or ICA: image compression; class separation; separating mixed audio sources.
Module 4.2 complete! 🎉
You can now explain why dimensionality becomes a problem and how PCA, LDA, and ICA solve different representation problems.