- How a digital image stores colours
- What a pixel and a mask are
- How segmentation differs from other vision tasks
- How people naturally group visual objects
How can a computer divide a picture?
This codelab starts from zero. You will learn what an image is, what segmentation means, and how simple OpenCV programs can find useful parts of an image.
Imagine colouring a map. You use one colour for water, another for land,
and another for roads. Image segmentation does something similar. It gives every tiny part of a picture a
label such as “background,” “car,” or “damaged area.”
What do you need?
Anyone can learn from this lesson. It uses plain language, illustrations, and interactive sliders to build up every idea from scratch. You do not need previous experience with computer vision, machine learning, or advanced mathematics.
- Separating an object from its background
- Grouping pixels by colour and position
- Joining small breaks in a shape
- Finding damaged areas on a product
Tools we use
PythonThe programming language used in all four examples.
OpenCVA free library with many ready-made tools for working with
images and video. We import it using the short name
cv2.NumPyA Python library for working with large grids of numbers. An
image is stored as one of these grids.
How this codelab works
Everything here runs right inside this page. Every diagram, slider, and toggle button below is a small live illustration of an idea — there is nothing to install and nothing to run in a terminal.
How to use the sliders and buttons: whenever you see a slider or a
toggle button as you go through the steps, drag it or click it. The picture next to it changes immediately,
so you can watch an idea happen instead of only reading about it.
Before you continue
Make sure you can see the sidebar on the left (or the menu button on a phone), so you can move between steps.
Pictures are grids of tiny squares
Before we divide an image, we need to understand how a computer stores one.
What is a pixel?
A pixel is one tiny square in a digital picture. The word comes from “picture element.” A photo may contain millions of pixels. When the pixels are small enough, our eyes see a smooth picture instead of separate squares.
A pixel stores numbers
In a grey image, one number tells us how bright the pixel is. 0 means black and
255 means white. Values in between are different shades of grey.
In a colour image, OpenCV normally stores three numbers: blue, green, and red. This order is called BGR.
What is image segmentation?
Image segmentation means dividing an image into useful parts. The computer looks at every pixel and gives it a label. Pixels with the same label belong to the same group.
Input image
Pixels store colours
Pixels store colours
→
Segmentation rule
Decide which pixels belong together
Decide which pixels belong together
→
Mask
Pixels store labels
Pixels store labels
What is a mask?
A mask is a second image with the same width and height as the original. But a mask does not show normal colours. Each position contains a label that tells us which group the matching pixel belongs to.
Think of a small object
The four centre pixels are the object. The other pixels are the background.
→
Its binary mask
0000011001100000
0 = background · 1 = object
Binary segmentation
Binary means “two choices.” A pixel is either background or foreground. Examples: healthy/damaged, road/not road, tumour/not tumour.
Multiclass segmentation
There are more than two labels. For a street image, 0 could mean sky, 1 road,
2 car, and 3 person.
A stencil is like a physical mask. Paint passes through the open part and is blocked
everywhere else. In OpenCV, pixels with mask value
255 are often the open part.Why can two answers both be correct?
The useful answer depends on the goal. A map app may group a whole forest as one region. A plant scientist may need a separate region for every tree or even every leaf. The image is the same, but the question is different.
Try a tiny calculation
The mask above has 16 pixels in total. Four have the value 1.
Foreground fraction = foreground pixels ÷ all pixels = 4 ÷ 16 = 0.25 = 25%.
Check your understanding
Say this in your own words: “A mask has the same size as the input image, and each mask pixel stores a group label.”
What answer do we need?
Computer vision has several tasks. The best task depends on how much location detail we need.
Imagine asking questions about a class photo. “Are students present?”
needs only yes or no. “Where is each student?” needs boxes. “Which exact pixels belong to each student?”
needs segmentation.
| Task | Simple question | Computer's answer |
|---|---|---|
| Classification | What is in this whole image? | A name, such as “cat” |
| Object detection | What objects are here, and roughly where? | A rectangle and name for each object |
| Semantic segmentation | What type of thing is each pixel? | One class label for every pixel |
| Instance segmentation | Which individual object owns each pixel? | A separate mask for each object |
| Panoptic segmentation | Can every pixel have both a class and an object identity? | One complete class-and-instance map |
Semantic versus instance segmentation
Semantic: same type, same label
If an image has three cars, all car pixels receive the label “car.” It does not tell us which pixels belong to car 1 or car 2.
Instance: separate each object
Every car gets its own identity and its own mask. This lets us count the cars and measure each car separately.
What does “panoptic” mean here?
It joins the other two ideas. Background areas such as sky and road get class labels. Countable objects such as people and cars also get separate identities.
A factory wants to measure the exact damaged area of a product. Which answer is most useful?
Correct. A mask shows the exact damaged pixels, so we can count them and calculate an area.
Simple rule
Use classification for “what,” detection for “what and roughly where,” and segmentation for “which exact pixels.”
Separate light pixels from dark pixels
We will build a simple binary mask by hand. The main idea is called thresholding.
What is a threshold?
A threshold is a dividing value. Imagine that pixel brightness goes from 0 to 255. If our threshold is 130, we can place every pixel into one of two groups:
Brightness 0 to 130
Call it background and put 0 in the mask.
Brightness 131 to 255
Call it foreground and put 255 in the mask.
A teacher may say, “A score of 40 or more is a pass.” Here, 40 is the threshold. A
brightness threshold works in the same way: it divides numbers into two groups.
Why blur the image first?
Real images contain small, unwanted changes called noise. Two nearby pixels on the same surface may have slightly different values. A small blur replaces each pixel with a gentle average of nearby pixels. This reduces tiny changes and makes the mask cleaner.
How do we choose the threshold?
One simple way is Otsu's method. Picture a chart that counts how many pixels have each brightness value — this is called a histogram. If an image really has a dark group and a bright group, the chart shows two humps with a valley between them. Otsu's method looks for that valley and places the threshold right there.
Two humps (dark pixels on the left, bright pixels on the right) with a low valley in the middle — the coral bar marks where Otsu's method places the threshold.
Make a grey image
→
Blur small noise
→
Choose threshold
→
Create mask
Explore · Thresholding, livedrag the slider
Below is a strip of pixels with random brightness values, from very dark to very bright. Drag the slider to set a threshold and watch every pixel decide, on its own, whether it becomes foreground or background.
Brightness values (grey strip):
Resulting binary mask (black = background, white = foreground):
Foreground pixels above this threshold: —
Important limit: thresholding does not know what an object is. It
only compares numbers. If a bright background has the same brightness as the object, it may also become
foreground.
Try it
Drag the slider to a very low value, then a very high value. Notice how a threshold that is too low or too high makes almost every pixel — or almost none of them — count as foreground.
Your brain does more than see pixels
People quickly turn separate marks into objects and patterns. Gestalt principles describe some common ways our brains do this.
What does “Gestalt” mean?
Gestalt refers to a set of psychological principles used to help computer algorithms group visual elements, organize pixels, and recognize objects as unified wholes rather than isolated parts.
Figure and ground
The figure is the part we treat as the main object. The ground is the background behind it. Sometimes this is not clear. In a black-and-white drawing, you may see either a pale vase or two dark faces. The pixels have not changed—only which part you label as the figure has.
Right now the dark shape in the middle looks like the figure (a vase), and the pale sides are the ground.
Computer idea: use pixel positions
x and y, or only compare nearby pixels.
Computer idea: compare their numerical features.
Computer idea: join small gaps with a morphology operation.
4. ConnectednessSimple
meaning: things physically joined together strongly look like one
object.
Computer idea: label pixels that touch one another.
Computer idea: prefer smooth boundaries.
Computer idea: compare motion in video.
These are clues, not complete programs
A principle such as similarity tells us a useful idea, but it leaves questions: Similar in colour or shape? How similar is enough? Should distant pixels be grouped? We must turn the human idea into clear numerical rules.
Students wearing the same uniform look similar, but they are not automatically one person
or even one friend group. In the same way, same-colour pixels are not automatically one object.
Pause and look around
Find one example of proximity and one example of similarity near you. For example, books close together on one shelf show proximity; books with matching covers show similarity.
Group pixels by colour and position
Now we will turn similarity and proximity into numbers that a computer can compare.
What is a feature?
A feature is a number that describes something useful. A pixel's blue, green, and red values are colour features. Its horizontal and vertical positions are location features.
B, G, RThree numbers that describe the pixel's colour in
OpenCV.
xHow far the pixel is from the left side of the image.
yHow far the pixel is from the top of the image.
Feature vectorA small list of features for one item. For example,
[B, G, R, x, y].What is clustering?
Clustering means placing similar items into groups without already knowing their class names. We use an algorithm named K-means. The letter K means the number of groups we ask it to find.
Choose K starting centres
If K is 3, the algorithm starts with three possible group centres.
Assign every pixel
Each pixel joins the closest centre based on its feature numbers.
Move each centre
The centre moves to the average position of the pixels in its group.
Repeat
Assignment and movement repeat until very little changes.
Explore · Group the two circlesclick a
button
The two circles below have the exact same colour but sit in different places. Choose what the grouping is allowed to look at, and watch how the groups change.
Why ask K-means for three groups?
We would ask for one group for the background, one for the left circle, and one for the right circle. Colour alone cannot separate the circles because their colours are identical, so the extra colour-only group would not be useful. Adding position gives the algorithm information that can tell the two circle locations apart.
Remember: K-means does not know that the shapes are circles. It
only sees rows of numbers. We decide what it can compare by choosing the features.
Why include x and y?
Colour values describe what a pixel looks like. The x and y values describe where it is. Including both lets K-means use similarity and proximity together.
Colour only: [B, G, R]
Two orange circles in different places may join the same group because their colours match.
Colour + place: [B, G, R, x, y]
The x and y values affect the distance between rows, so the two circles can become separate groups.
Ask these questions for any grouping method
- What makes two pixels similar? It may be colour, brightness, texture, motion, or another feature.
- Which pixels can affect one another? Every pixel, or only nearby pixels?
- What result is allowed? Must a region be connected? Must its edge be smooth?
Small experiment: use the buttons above to switch back and forth a
few times. Imagine the two circles were drawn much closer together — would position alone still be enough
to tell them apart?
Join small gaps and find separate objects
A mask is often imperfect. We can clean its shapes before we count its regions.
What is morphology?
Morphology means changing shapes in a binary mask. It works using a small shape called a kernel. The kernel moves over the mask and checks a small neighbourhood at a time.
Imagine using the tip of a thick marker over a broken outline. A small marker fills tiny
gaps. A very thick marker may accidentally join two different objects. The kernel size has a similar effect.
DilationMakes white areas grow. It can fill tiny gaps, but it also
makes objects thicker.
ErosionMakes white areas shrink. It removes thin parts and can
separate weak connections.
ClosingDilation followed by erosion. It joins small breaks, then
brings the shape close to its old size.
OpeningErosion followed by dilation. It removes small white noise,
then restores the main shapes.
What is a connected component?
A connected component is one group of foreground pixels that touch. Imagine islands on a map. Every piece of connected land is one island. Connected-component analysis gives each island a different number.
Broken mask
→
Close small gaps
→
Fill the shapes
→
Label each region
Explore · Choose a kernel sizedrag the
slider
The dark shape on the left below has one small gap in it. A separate small block sits on the right. Drag the slider to try different kernel sizes and see what closing does at each size.
Choosing the kernel size
Kernel too small
It cannot reach across the gap. One object may stay broken into several parts.
Kernel too large
It may reach across the space between different objects and wrongly join them.
Try it
Move the slider through every position, one at a time. Notice the exact point where the two problems swap: the shape goes from "still broken" to "just right" to "wrongly merged" as the kernel grows.
Why a simple rule often fails
Our programs work well on carefully made images. Real photographs contain many more problems.
Light changes colour
The same white wall can look bright near a window and grey in a shadow. Its pixel numbers change even though it is still one wall.
Different objects look alike
A brown dog and a brown floor may have similar colours. Colour alone cannot understand their meaning.
One object has many looks
A patterned shirt has several colours. A shiny car contains reflections. One threshold may split one object into many parts.
Noise changes pixels
Cameras and compressed image files introduce small unwanted changes. A clean flat surface may not have one exact value.
Objects hide each other
Occlusion happens when one object blocks part of another. The hidden pixels give us no direct information.
The useful size changes
A forest can be one region, many tree regions, or thousands of leaf regions. The correct level depends on the question.
Learn to diagnose a bad mask
Do not immediately change random numbers. First describe what is wrong, then choose a tool that matches the problem.
| What you see | Possible reason | Simple thing to try |
|---|---|---|
| Many tiny white dots | Noise | Blur before thresholding, or use opening |
| Holes inside the object | Some object pixels look like background | Use closing or fill closed contours |
| One object becomes many pieces | Gaps or large colour changes | Use closing or a better feature |
| Two objects become one | They look similar or the kernel is too large | Use position information or a smaller kernel |
| A shadow becomes foreground | Darkness was confused with object identity | Use colour, texture, shape, context, or a learned model |
What does “semantic meaning” mean?
It means understanding what something actually is. A threshold understands only “brighter than 130.” It does not understand “this is a car.” Modern trained models can learn more meaningful visual features from many labelled examples.
Think like an engineer
If a shadow appears in your mask, say why: “My rule selects dark pixels, and the shadow is also dark.” This explanation points toward a better feature.
Find damage on a factory product
Our factory needs more than a “damaged” label. It needs the exact damaged pixels and the percentage of the product that is damaged.
Understand the problem first
Input
A picture of one light-coloured product. Scratches and stains are darker than the normal surface.
Output
A binary damage mask, an easy-to-read red overlay, the number of damage regions, and the damaged percentage.
Our important assumption
Damage is dark and the healthy product surface is light. This makes a simple threshold possible. A real factory system must test whether this assumption is true under different lights, materials, and camera settings.
The complete plan
Start with a picture
A photo of one light-coloured product with two scratches and one stain.
Make a product mask
We only want to search inside the product. This is called a region of interest, or ROI.
Select dark pixels
Thresholding returns white for grey values below the chosen threshold, and black for everything else.
Clean the damage mask
Closing fills small gaps in damage. Opening removes small isolated dots.
Find boundaries
A contour is a line that follows the outside edge of a shape. We draw contours so people can easily see the result.
Measure the result
Count damage pixels and divide by the total number of product pixels.
Explore · Choose a damage thresholddrag the
slider
The picture below is a light-coloured product with two scratches and one stain. Drag the slider to choose how dark a pixel must be before we call it "damaged," and watch the red damage mask change.
Do not confuse mask and overlay. A real damage mask stores exact labels
for computation. A red overlay like the one above is only a friendly picture for a person to view.
Try it
Move the slider through all four positions in order. Notice the trade-off: a threshold that is too strict misses real damage, and a threshold that is too loose flags healthy surface as damaged.
Choose the best answer
It is fine to make mistakes. Read the explanation after every answer and return to an earlier step when something is unclear.
1. What is a pixel?
A pixel is one small picture element. An image is a grid containing many pixels.
2. What is a segmentation mask?
A mask matches the input image's width and height. Each position stores a label such as background or foreground.
3. When do we need segmentation instead of classification?
Segmentation gives a pixel-level result, so it can show an exact shape or area.
4. What does a threshold do?
A threshold compares numbers. It does not understand object meaning.
5. Which Gestalt principle says nearby things look grouped?
Proximity means nearness. Pixel coordinates can represent this idea numerically.
6. Why add x and y to a pixel's feature vector?
The x and y values tell the algorithm where the pixel is, so proximity can affect grouping.
7. What does morphological closing help with?
Closing grows and then shrinks the foreground. It can bridge small gaps.
8. What does instance segmentation add for three cars?
Instance segmentation separates individual objects, even when they have the same class.
Can you explain the full pipeline?
Input image → choose useful features → create a mask → clean the mask → label or measure its regions.
You can now turn pixels into regions
You started with no computer-vision knowledge and built four small OpenCV programs. Review the main ideas in plain language.
The story of this codelab
An image is a grid
Every position is a pixel. A grey pixel stores one brightness value; an OpenCV colour pixel usually stores B, G, and R values.
Segmentation gives every pixel a label
The result is a mask with the same width and height as the input image.
Human grouping ideas can guide algorithms
Similarity, proximity, closure, connectedness, smooth continuation, and common motion are useful clues.
Algorithms need exact numerical rules
We represented colour with BGR values, position with x and y, closure with morphology, and connectedness with component labels.
Every simple method has assumptions
Thresholding works only when pixel values separate the wanted region. Good testing asks when that assumption fails.
Small glossary
PixelOne tiny square in a digital image.
SegmentationDividing an image into useful pixel regions.
MaskAn image-sized grid containing region labels.
FeatureA number that describes something useful, such as colour or
position.
ThresholdA value used to divide numbers into groups.
KernelA small shape used to check and change local mask
areas.
ContourA line around the edge of a shape.
Connected componentOne group of foreground pixels that
touch.
A good next lesson: learn watershed segmentation for objects that
touch. After that, explore trained semantic-segmentation models, which learn useful features from labelled
examples.
Final reflection
- Why is a mask more useful than a box when we need exact area?
- Why can two same-colour objects still be different regions?
- What happens when a morphology kernel is too large?
- Which assumption in the factory project may fail on a real photograph?
One-sentence summary
Image segmentation assigns a useful label to every pixel, using rules or learned features that match the problem we want to solve.