Pattern Classification Fundamentals
In this codelab, we will understand how machines identify patterns, represent objects using features, and learn to classify data using supervised, unsupervised, and semi-supervised learning.
↓
Data / Image
↓
Feature extraction
↓
Feature vector
↓
Pattern analysis
↓
Classification decision
This lesson builds the foundation for later topics such as PCA, LDA, ICA, K-Means, K-Medoids, and Gaussian Mixture Models.
What is Pattern Analysis?
Pattern analysis means finding meaningful similarities, differences, and structures in data so that a machine can make decisions.
What is a pattern?
A pattern is a measurable representation of an object or event.
A fruit, face, handwritten digit, pedestrian, vehicle, medical scan, or any other item we want to analyze.
Measurable properties such as color, height, width, texture, shape, area, weight, or pixel intensity.
Example: Fruit recognition
| Fruit | Weight | Redness | Length |
|---|---|---|---|
| Apple | 150 g | 0.90 | 7 cm |
| Orange | 180 g | 0.40 | 8 cm |
| Banana | 120 g | 0.10 | 18 cm |
The computer does not understand “apple” directly. It works with numerical features that describe the apple.
Represent Objects as Feature Vectors
Machine-learning algorithms usually work with numbers, so we represent each object using a vector of features.
X = [150, 0.90, 7]
# 150 -> weight in grams
# 0.90 -> redness score
# 7 -> length in cmThis is called a feature vector.
↓ extract measurable properties
[weight, redness, length]
↓
[150, 0.90, 7]
A feature vector is simply a list of numerical values used to describe one sample.
What is Classification?
Classification assigns an input sample to one of several predefined categories.
↓
Feature extraction
↓
Classifier
↓
Cat / Dog / Bird
Example
Suppose we already know two possible classes:
Cat
Dog
The classifier receives a new input and decides which predefined class it belongs to.
Recognition identifies what an object is. Classification is the decision process that assigns it to a known category.
Understand Feature Space
When we use two features, every sample can be visualized as a point on a 2D graph.
Feature 2 = Weight
Each person → one point in 2D feature space
Weight ↑
80 | ●
75 | ●
70 | ●
65 |
60 | ○
55 | ○
50 | ○
+----------------------------→ Height
○ = Class A
● = Class BThe complete region containing all possible feature values is called the feature space.
Classification becomes the problem of deciding which region of the feature space a new point belongs to.
Discriminant Function
A discriminant function gives a score to each class. The class with the highest score is selected.
Two-class example
g_cat(x) = score for Cat
g_dog(x) = score for Dog
if g_dog(x) > g_cat(x):
prediction = "Dog"0.35
0.82
Since 0.82 > 0.35, the sample is classified as Dog.
Linear discriminant function
g(x) = wᵀx + b| Symbol | Meaning |
|---|---|
x |
Feature vector of the input sample. |
w |
Weights that determine the importance of each feature. |
b |
Bias term that shifts the decision boundary. |
g(x) |
Discriminant score used for the decision. |
Decision Boundary
A decision boundary separates regions belonging to different classes.
DOGS
● ● ●
● ● ●
---------------------- ← Decision Boundary
○ ○ ○
○ ○
CATSA new point above the boundary may be classified as Dog, while a point below it may be classified as Cat.
The decision boundary is often the location where two class scores become equal.
What does a decision boundary do?
Supervised Learning
In supervised learning, every training sample is provided together with its correct answer or label.
| Input | Label |
|---|---|
| Cat image | Cat |
| Dog image | Dog |
| Cat image | Cat |
| Dog image | Dog |
↓
Learning algorithm
↓
Trained model
↓
New sample
↓
Predicted class
Common supervised tasks
- Cat vs dog classification
- Spam vs not spam
- Pedestrian vs non-pedestrian
- Normal vs abnormal medical image
Supervised learning = labeled training data.
A Tiny Supervised Classifier
Use K-Nearest Neighbors only as a simple demonstration of how labeled samples can classify a new point.
from sklearn.neighbors import KNeighborsClassifier
# Two features for each sample
X = [
[1, 1],
[1, 2],
[2, 1],
[8, 8],
[8, 9],
[9, 8]
]
# Correct labels
y = [
"Class A",
"Class A",
"Class A",
"Class B",
"Class B",
"Class B"
]
model = KNeighborsClassifier(n_neighbors=3)
model.fit(X, y)
prediction = model.predict([[2, 2]])
print(prediction)y = known labels
↓ model.fit(X, y)
model learns from labeled examples
↓ model.predict([[2, 2]])
↓ predicted class
KNN itself will be discussed later under non-parametric methods. Here, it is only being used to demonstrate supervised learning.
Unsupervised Learning
In unsupervised learning, the algorithm receives data without correct labels and tries to discover structure by itself.
↓
Learning algorithm
↓
Discover similarities
↓
Groups / clusters / structure
Example
Input points:
● ● ●
● ●
●
● ●
No labels are provided.
The algorithm may discover:
Cluster 1
Cluster 2
Cluster 3Classification usually uses known class labels. Clustering discovers groups when labels are not given.
Which type of data is used in unsupervised learning?
Semi-Supervised Learning
Semi-supervised learning combines a small amount of labeled data with a much larger amount of unlabeled data.
├→ Semi-supervised learning → Model
Large unlabeled dataset ┘
Why is it useful?
Collecting data can be easy, but labeling it can be expensive and time-consuming.
Manually labeled by experts.
Available but unlabeled.
A semi-supervised approach tries to learn from both.
A hospital may have thousands of scans, but only a smaller subset has been carefully labeled by doctors.
Supervised vs Unsupervised vs Semi-Supervised
These three terms mainly differ in how much labeled information is available during training.
| Type | Labels | Main Goal | Example |
|---|---|---|---|
| Supervised | Yes | Predict known classes | Cat vs Dog |
| Unsupervised | No | Discover hidden structure | Grouping similar pixels |
| Semi-Supervised | Some | Learn from labeled + unlabeled data | Medical image classification |
Unsupervised: no teacher, discover patterns.
Semi-supervised: teacher gives only some answers.
Pattern Classification Pipeline
Now connect all the concepts into one end-to-end flow.
↓
2. Preprocess data
↓
3. Extract useful features
↓
4. Build a feature vector
↓
5. Feed it to a classifier
↓
6. Calculate class scores / discriminant functions
↓
7. Compare scores
↓
8. Cross a decision boundary
↓
9. Output predicted class
An image of a pedestrian can be converted into features, passed to a trained model, scored against possible classes, and finally classified as “Pedestrian” or “Not Pedestrian”.
Pattern Classification Revision
Review the core definitions before moving to dimensionality reduction.
Completion checklist
- I can explain pattern analysis.
- I know what a feature and feature vector are.
- I can explain classification.
- I understand feature space.
- I understand a discriminant function.
- I understand a decision boundary.
- I can distinguish supervised, unsupervised, and semi-supervised learning.
- I understand how these ideas connect in a classification pipeline.
Practice
- Choose any real-world classification problem and list at least four useful features.
- Explain whether your problem should use supervised, unsupervised, or semi-supervised learning.
- Draw a simple 2D feature-space diagram containing two classes and a decision boundary.
- For a three-class problem, write three imaginary discriminant scores and decide the predicted class.
Module 4.1 complete! 🎉
You now have the foundation required for the next codelab: dimensionality reduction using PCA, LDA, and ICA.