## Description

## Overview

In this project, you will gain experience with the perspective transformation operation (discussed in detail in

the lectures), bilinear interpolation, and histogram equalization. You will use the perspective transform to

replace a portion of an existing image with an alternate image (the ‘hack’).

The goals are to:

• aid in understanding perspective transformations (or 2D homographies) and to help to visualize their

application to images;

• experiment with inverse image warping and bilinear interpolation, to insert one image into another

(respecting the appropriate geometry); and

• apply histogram equalization to improve the overall appearance (contrast) of an image.

The due date for project submission is

All submissions

will be in Python 3.8 via Autolab (more details will be provided in class and on Quercus); you may submit as

many times as you wish until the deadline. To complete the project, you will need to review some material

that goes beyond that discussed in the lectures—more details are provided below.

(a) Yonge & Dundas Square (b) Soldiers’ Tower

Your main project task is to perform some billboard hacking (this is a basic demonstration of the use of

computer vision and shows that it can be fairly easy to change ‘reality’).

There are two images above: image

(a) is of Yonge and Dundas Square, an area that contains several large billboards, while image (b) is of

Soldiers’ Tower on the University of Toronto campus. Conveniently, the image of Yonge and Dundas Square

has very limited radial distortion, which makes it suitable for our purposes.

Your assignment (should you

choose to accept it) is to replace the billboard advertisement for the “CN Tower Edge Walk” with the photo

of Soldiers’ Tower, such that the result looks natural (i.e., like the image of Soldiers’ Tower is meant to be

there). The project has four parts, worth a total of 50 points.

Please clearly comment your code and ensure that you only make use of the Python modules and functions listed

at the top of the code templates. We will view and run your code.

## Part 1: Perspective Transformations via the DLT

To carry out this exercise (Part 1), you will need to determine the perspective homography that transforms or

maps pixels from the (rectangular) Soldiers’ Tower image to the appropriate coordinates in the Y&D Square

image, and vice versa.

The homography can be computed using the Direct Linear Transform (DLT) algorithm,

given four point correspondences between the two images.

We did not review the DLT algorithm in the lectures, however it is straightforward and easy to implement in

Python using NumPy. Details can be found in Section 2.1 of the (very useful) M.A.Sc. thesis written by Elan

Dubrofsky of UBC, which is available on Quercus.

For the moment, we will consider the four point correspondences to be exact—in later lectures, we will show how an overdetermined system of correspondences can be

solved to produce an optimal estimate.

For this part of the project, you should submit:

• a single function, dlt_homography.py, that computes the perspective homography between two

images, given four point correspondences (n.b., the ordering of the points is important).

Note that we are using four matching points in the DLT algorithm, and each point provides two constraints

on the homography. However, there are nine numbers in the 3 × 3 homography matrix—recall that the a

homography is defined up to scale only (any multiple of all the values in the homography is the same homography), and so you should normalize your matrix by scaling all entries such that the lower right entry in the

matrix is 1.

## Part 2: Bilinear Interpolation

With the perspective homography in hand, you can make use of the inverse warping and bilinear interpolation

operations (discussed in the lectures and in the Szeliski text) to determine the best pixel value from the

Soldiers’ Tower image to replace a pixel value in the Y&D Square image.

Note that the Y&D Square image

is in colour (it has three bands: R, G, and B), and so the same transform must be applied to each band (the

Soldiers’ Tower image is a greyscale image).

For this part of the project, you should submit:

• a single function, bilinear_interp.py, that performs bilinear interpolation to produce a pixel intensity

value, given an image and a subpixel location (point).

## Part 3: Histogram Equalization

You will notice that the image file provided, uoft_soldiers_tower_light.png, is quite bright (overexposed) and has relatively low contrast. To fix this, you should implement the simple (discrete) histogram

equalization algorithm discussed on page 115 of the Szeliski text (and in the course lectures).

For this part

of the project, you should submit:

• a single function in histogram_eq.py, which performs discrete histogram equalization on the input

image (which will be 8-bit and greyscale only).

## Part 4: Billboard Hacking

You’re now ready to perform the billboard hack! Using the components you’ve built, you should: enhance

the contrast of the Soldiers’ Tower image, compute the perspective homography (once) that defines the warp

between the Y&D Square image and the Soldiers’ Tower image, and then perform bilinear interpolation over

all of the corresponding pixels to place Soldiers’ Tower in the billboard position.

Some portions of the code

have already been filled in for you—in particular, the bounding box for the Edge Walk billboard, and the

four pixel-to-pixel correspondences between the images, are available.

For this (final) part of the project, you

should submit:

• a single function in billboard_hack.py, that uses the other functions above to produce the composite,

‘hacked’ image.

The composite image must be stored in colour and which must be exactly the same size as the original Y&D

image (in terms of rows and columns, i.e., do not change the image size!).

Grading

Points for each portion of the project will be assigned as follows:

• Perspective homography DLT function – 15 points (3 tests × 5 points per test)

Each test uses a different set of point correspondences. The root of the sum of squared projection errors

(compared to the reference homography) must be below 0.1 (pixels) to pass.

• Bilinear interpolation function – 10 points (5 tests × 2 points per test)

Each test relies on a varied reference image and a different point location in that image. The absolute

value of the brightness difference (between the reference interpolated brightness) must be less than or

equal to 1 to pass (e.g., if the reference value is 212, your function must report 211, 212, or 213 to pass

the test).

• Histogram equalization function – 10 points (2 tests; 2 points and 8 points)

There are two tests, one using the over-exposed version of the Soldiers’ Tower image, and one using

a hidden reference image (see points allocated above). To pass either test, only 10% of less of the

equalized pixel intensity values may be greater than 2 units (of intensity) away from the reference

intensity values (this is a fairly generous bound).

• Image composition script – 15 points (3 tests × 5 points per test)

There are three tests, each of which applies a more stringent criterion for matching between your hacked

image and the reference solution (in terms of the absolute intensity difference between pixels in the

warped region only, evaluated by the mean and standard deviation).

For now, the exact threshold

parameters are being kept under wraps—if your support functions are working correctly, you should be

able to pass the hardest test!

Total: 50 points

Grading criteria include: correctness and succinctness of the implementation of support functions, proper

overall program operation and code commenting, and a correct composite image output (subject to some

variation). Please note that we will test your code and it must run successfully. Code that is not properly

commented or that looks like ‘spaghetti’ may result in an overall deduction of up to 10%.

3 of 3