STEP 1 · BASIC IDEA

Segmentation Across Time

In an image, segmentation separates useful regions. In a video, we do the same thing across many frames.

For exams: Remember two simple applications: shot boundary detection and background subtraction.

Shot Boundary Detection

Find where one scene suddenly changes to another scene.

Background Subtraction

Separate moving foreground objects from the background.

Video
↓
Frame 1 → Frame 2 → Frame 3 → Frame 4
↓
Compare frames over time

What extra thing does video give us compared with one image?

STEP 2 · GET READY

Install OpenCV

We only need Python and OpenCV for this codelab.

pip install opencv-python

Video file:

Keep a small video in the same folder and name it video.mp4.

import cv2

video = cv2.VideoCapture("video.mp4")

while True:
    success, frame = video.read()

    if not success:
        break

    cv2.imshow("Video", frame)

    if cv2.waitKey(30) == ord("q"):
        break

video.release()
cv2.destroyAllWindows()

REMEMBER

A video is processed frame by frame.

STEP 3 · APPLICATION 1

What Is a Shot Boundary?

A shot boundary is the point where one camera shot ends and another shot begins.

Shot A
Frame 1 → Frame 2 → Frame 3
HARD CUT
Frame 4 → Frame 5 → Frame 6
Shot B

Hard shot boundary:

The change happens suddenly in one frame. This is the easiest type to detect.

Main idea:

If two consecutive frames look very different, a new shot may have started.

What is a hard cut?

STEP 4 · COMPARE COLOUR

Use an HSV Histogram

Instead of comparing every pixel directly, we can compare the colour distribution of two frames.

HSV

A colour format. H is hue, S is saturation, V is brightness.

Histogram

A count of how much of each colour appears in the frame.

Why normalize?

Normalization puts histogram values on a common scale, so two frames can be compared more fairly.

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

hist = cv2.calcHist(
    [hsv],
    [0, 1],
    None,
    [50, 60],
    [0, 180, 0, 256]
)

cv2.normalize(hist, hist)

EXAM LINE

An HSV histogram represents the colour distribution of a video frame.

STEP 5 · SIMPLE PROGRAM

Detect Hard Shot Boundaries

We compare the current frame histogram with the previous frame histogram.

Previous frame histogram
↓ compare
Current frame histogram
↓
Very different? → Possible hard cut

import cv2

video = cv2.VideoCapture("video.mp4")

previous_hist = None
frame_number = 0

while True:
    success, frame = video.read()

    if not success:
        break

    frame_number += 1

    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

    hist = cv2.calcHist(
        [hsv],
        [0, 1],
        None,
        [50, 60],
        [0, 180, 0, 256]
    )

    cv2.normalize(hist, hist)

    if previous_hist is not None:

        score = cv2.compareHist(
            previous_hist,
            hist,
            cv2.HISTCMP_CORREL
        )

        if score < 0.5:
            print("Possible shot boundary at frame:",
                  frame_number)

    previous_hist = hist

video.release()

Very simple meaning of the score:

A score close to 1 means the histograms are similar. A much smaller score means the frames are different.

Threshold:

Here we used 0.5 only as an example. The best threshold depends on the video.

With histogram correlation, what may indicate a shot change?

STEP 6 · APPLICATION 2

What Is Background Subtraction?

Background subtraction tries to separate moving objects from the mostly stable background.

Video frame
↓
Learn background
↓
Compare current frame with background
↓
Moving parts become foreground mask

Background

Usually the part that stays mostly unchanged.

Foreground

The moving object, such as a person or vehicle.

Example:

A fixed CCTV camera watches a road. The road is background. Moving cars are foreground.

STEP 7 · USE MOG2

MOG2 Background Subtraction

OpenCV already provides a background subtractor called MOG2.

background_subtractor =
    cv2.createBackgroundSubtractorMOG2()

Simple idea:

MOG2 watches many frames, learns what the background usually looks like, and marks changing regions as foreground.

EXAM LINE

MOG2 is a background-subtraction method used to detect moving foreground regions in video.

STEP 8 · SIMPLE PROGRAM

Extract the Moving Foreground

Apply MOG2 to every frame. The result is a foreground mask.

import cv2

video = cv2.VideoCapture("video.mp4")

background_subtractor =
    cv2.createBackgroundSubtractorMOG2()

while True:
    success, frame = video.read()

    if not success:
        break

    mask = background_subtractor.apply(frame)

    cv2.imshow("Original Video", frame)
    cv2.imshow("Foreground Mask", mask)

    if cv2.waitKey(30) == ord("q"):
        break

video.release()
cv2.destroyAllWindows()

What will you see?

The mask is mostly black for background. Moving foreground regions appear brighter.

Optional: reduce shadow pixels

_, clean_mask = cv2.threshold(
    mask,
    200,
    255,
    cv2.THRESH_BINARY
)

cv2.imshow("Clean Mask", clean_mask)

Why?

MOG2 may mark shadows as gray. A simple threshold can remove many gray shadow pixels.

STEP 9 · IMPORTANT LIMITATIONS

Things That Affect the Result

For college exams, you mainly need to remember why these methods may fail or give extra detections.

Problem	What happens?
Threshold too strict	Shot detector may report too many boundaries.
Threshold too loose	Real shot changes may be missed.
Camera motion	Much of the frame moves, so MOG2 may think the whole scene is foreground.
Lighting change	A sudden brightness change may look like motion or a shot change.
Shadows	MOG2 may detect shadows along with the moving object.
Gradual transition	A fade or dissolve changes slowly, so a hard-cut detector may miss it.

Very important:

These simple programs are good for understanding the concept. Real video systems usually need better tuning and extra processing.

STEP 10 · EXAM REVISION

What You Need to Remember

These short answers are enough for a basic college exam or viva.

1. What is shot boundary detection?

It finds the point where one video shot ends and another shot begins.

2. What is a hard shot boundary?

A sudden change from one shot to another in consecutive frames.

3. Why use HSV histograms?

They give a simple representation of the colour distribution of each frame.

4. Why normalize histograms?

Normalization puts histogram values on a common scale before comparison.

5. How do we detect a hard cut?

Compare histograms of consecutive frames. A large difference or low similarity can indicate a shot boundary.

6. What is background subtraction?

It separates moving foreground objects from a mostly stable background.

7. What is MOG2?

An OpenCV background-subtraction method that learns the background and detects moving foreground regions.

8. What is a foreground mask?

A binary-like image where moving regions are highlighted and background regions are dark.

9. Why can camera movement cause problems?

Because many pixels change at once, so the algorithm may think most of the frame is foreground.

10. Why are gradual transitions harder?

Because fades and dissolves change slowly instead of producing one sudden hard cut.

Completion checklist

I know what video segmentation means.
I can explain a hard shot boundary.
I know why HSV histograms are compared.
I understand that a threshold decides when a change is large enough.
I can explain background subtraction.
I know what MOG2 does.
I can explain foreground masks.
I know the effects of camera motion, lighting, shadows, and gradual transitions.

Codelab complete! 🎉

You now know the two basic video segmentation applications needed for this topic: hard shot boundary detection and moving foreground extraction.