## Description

For Assignment 2, you will implement an image stitcher that uses image warping

and homographies to automatically create an image mosaic. We will focus on the

case where we have two input images that should form the mosaic, where we warp

one image into the plane of the second image and display the combined views. This

problem will give some practice manipulating homogeneous coordinates, computing

homography matrices, and performing image warps. For simplicity, we’ll specify

corresponding pairs of points manually using mouse clicks.

Note: There are some built-in Matlab functions that could do much of the work for

this project. However, to get practice with the workings of the algorithms, we want

you to write your own code. Specifically, you may not use any of these functions in

your implementation: cp2tform, imtransform, tformarray, tformfwd, tforminv,

maketform.

Provided files:

Two image files that can be used for the mosaic.

What You Have to Do

Task 1 (15 points) Getting correspondences:

Write code to get manually identified corresponding points from two views. Look at

Matlab’s ginput function for an easy way to collect mouse click positions. The

results will be sensitive to the accuracy of the corresponding points; when providing

clicks, choose distinctive points in the image that appear in both views.

Task 2 (20 points) Computing the homography parameters:

Write a function that takes a set of corresponding image points and computes the

associated 3 x 3 homography matrix H. This matrix transforms any point pi in one

view to its corresponding homogeneous coordinates in the second view, pi’, such

that

λ* pi = H * pi’

Note that pi and pi’ are both vectors with 3 elements. The function should take a list

of pairs of corresponding points (what is the minimum number of points you

need?) from the two views, where each point is specified with its 2d image

coordinates.

Useful Matlab functions: ‘\’ operator (help mldivide), reshape

Verify that the homography matrix your function computes is correct by mapping

the clicked image points from one view to the other, and displaying them on top of

each respective image. Be sure to handle homogenous and non-homogenous

coordinates correctly.

Task 3 (20 points) Warping between image planes:

Write a function that can take the recovered homography matrix and an image, and

return a new image that is the warp of the input image using H . Since the

transformed coordinates will typically be sub-pixel values, you will need to sample

the pixel values from nearby pixels. Feel free to use the sampleBilinear.m you

developed for the first assignment. For color images, warp each RGB channel

separately and then stack together to form the output. To avoid holes in the output,

use inverse warp rather than direct mapping.

To compute the bounding box of the destination image, you will need to warp the

points from the source image into the reference frame of the destination. Then

sample all points in that destination bounding box from the proper coordinates in

the source image. Note that transforming all the points will generate an image of a

different shape / dimensions than the original input. It is ok to have some areas of

the new image be black (0).

Useful Matlab functions: round, interp2, meshgrid, isnan.

Task 4 (20 points) Create the output mosaic:

Once we have the source image warped into the destination image’s frame of

reference, we can create a merged image showing the mosaic. Create a new image

large enough to hold both (registered) views; overlay one view onto the other,

simply leaving it black wherever no data is available. Don’t worry about artifacts

that result at the boundaries.

You are free to use a method/convention of your own choosing for the overlap

areas.

Task 5 (25 points) After writing and debugging your system:

1. [5 pts] Apply your system to the provided pair of images, and display the output

mosaic.

2. [10 pts] Show two additional examples of mosaics you created using images that

you have taken. You can make a mosaic from two or more images of a broad scene

that requires a wide angle view to see well. Or, make a mosaic using two images

from the same room where the same person appears in both.

3. [10 pts] Warp one image into a “frame” region in the second image. To do this, let

the points from the one view be the corners of the image you want to insert in the

frame, and the let the corresponding points in the second view be the clicked points

of the frame (rectangle) into which the first image should be warped. Use this idea

to replace one surface in an image with an image of something else. For example —

overwrite a billboard with a picture of your dog, or project a drawing from one

image onto the street in another image, or replace a portrait on the wall with

someone else’s face, or paste a Powerpoint slide onto a movie screen, …

For all examples, play around a bit with the choice of points for the correspondence

pairs until you get a reasonable alignment.

[OPTIONAL] Extra credit

Implement RANSAC for robustly estimating the homography matrix from noisy

correspondences. Your function should be able to handle more than 4 point

correspondences. Show with an example where it successfully gives good results

even when there are outlier (bad) correspondences given as input. Compare the

robust output to the original (non-RANSAC, only 4 point correspondences)

implementation where all correspondences are used.

Note: feel free to attempt using David Lowe’s SIFT Matlab demo

(http://www.cs.ubc.ca/~lowe/keypoints/siftDemoV4.zip). Warning: it only works

in Linux and Windows, it does not work on Mac.

The result of calling the sift.m function will be 2 vectors, one of features and one

of keypoints. Make sure you understand what is saved in these two variables so that

you can potentially use them for the RANSAC implementation. If you can’t get SIFT

to work, you will need to select make than 4 pairs of points using the function

developed in Task 1.

Submitting the assignment:

Make sure each script or function file is well commented and it includes a block

comment with your name, course number, assignment number and instructor name.

Zip all the .m and image files together and submit the resulting .zip file through

Moodle as Assignment 2 by Friday, September 30th, by 11:55pm.

Acknowledgement: project description and test images courtesy of Kristen

Grauman, University of Texas at Austin.

Tips:

• It can be useful when debugging to plot the corners and clicked points from

one view on top of the second view after transforming them via H. Use

axis([minx, maxx, miny, maxy]); to adjust the viewing window so that you

can see all points.

• You will need the inverse of the homography matrix to transform

“backwards”.

• Be aware that Matlab’s image (matrix) indices are specified in (row,col)

order, i.e., (y,x), whereas the plot and ginput functions use (col,row)

order, i.e., (x,y).

• Check the order of the clicked corresponding points, to make sure your code

uses the intended corresponding point pairs.

• As usual, be careful with how images are cast for computations and display

(double vs. uint8). In particular, for your sampleBilinear.m function, be

sure to pass a matrix of doubles for the image input.

• When collecting your own images, be sure to either maintain the same center

of projection (hold the camera at one location, but rotate between views), or

else take shots of a scene with a large planar component (a building, maybe).

In either case, use a static scene. Textured images that have distinctive points

you can click on are good. Also ensure that there is an adequate overlap

between the two views.