## Description

## Part I: Camera Calibration using 3D calibration object (50 points)

We want to calibrate the camera of a robot vehicle. We will use a linear method as described in the lectures.

We place a large cubic frame of size 4 meters on the road several meters in front of the vehicle.

The positions

of the eight corners of the cubic frame are dened with respect to a world coordinate system with its axes

parallel to the cube edges and with its origin at the center of the cube.

The world coordinates of the cube

vertices are:

2 2 2,

-2 2 2,

-2 2 -2,

2 2 -2,

2 -2 2,

-2 -2 2,

-2 -2 -2,

2 -2 -2

We detect the corresponding cube corners at the following pixel positions in the camera image:

422 323; 178 323; 118 483; 482 483; 438 73; 162 73; 78 117; 522 117

1. Draw the image points, using small circles for each image point.

2. Write a Matlab function that takes as argument the homogeneous coordinates of one cube corner and

the homogeneous coordinates of its image, and returns 2 rows of the matrix P (slide 30 of the Camera

Calibration pdf document). This matrix P will be used to compute the 12 elements of the projection

matrix M such that λpi = MPi

3. Use this Matlab function to generate 2 rows of the matrix P for each cube corner and its image and

obtain a matrix with 16 rows and 12 columns. Print matrix P.

1

4. Now we need to solve the system Pm = 0. Find the singular value decomposition of matrix P using

matlab svd function. The last column vector of V obtained by svd(P) should be the 12 elements in row

order of the projection matrix that transformed the cube corner coordinates into their images. Print

the matrix M.

5. Now we need to recover the translation vector which is a null vector of M. Find the singular value

decomposition of matrix M = UΣV

T

. The 4 elements of the last column of V are the homogeneous

coordinates of the position of the camera center of projection in the frame of reference of the cube

(as in slide 36). Print the corresponding 3 Euclidean coordinates of the camera center in the frame of

reference of the cube.

6. Consider the 3×3 matrix M’ composed of the rst 3 columns of matrix M. Rescale the elements of this

matrix so that its element m33 becomes equal to 1. Print matrix M’ . Now let the rotation matrices

be as dened in slide 38 where the axes e1, e2, e3 are the x, y, z axes respectively. Therefore M’ can

be written as M0 = KRT

z RT

y RT

x

7. We will perform the RQ factorization of M’ in several steps. First, nd a rotation matrix Rx that sets

the term at position (3,2) to zero when Rx is multiplied to M’. The cosine and sine used in this matrix

are of the form

cos(θx) = m33/sqrt(m2

33 + m2

32)

sin(θx) = −m32/sqrt(m2

33 + m2

32)

Note that the term at position (3, 2) would also be set to zero if the signs of cos(θx) and sin(θx) were

reversed, but this would lead to nding a negative focal length for the camera. So we should choose the

signs that leads to a positive focal length. Compute the angle θx of this rotation in degrees. Compute

matrix N = M’ ∗ Rx. Print Rx, θx and N.

8. The element n31 of N is small enough so that there is no need for a rotation Ry. However, element n21

is large and a rotation matrix Rz is needed to set it to zero. Compute the rotation matrix Rz using

cosine and sine of the form

cos(θz) = n22/sqrt(n

2

21 + n

2

22)

sin(θz) = −n21/sqrt(n

2

21 + n

2

22)

Compute the rotation angle θz in degrees. This angle is actually very small.

9. Since we factorized out Rz we can directly compute the calibration matrix K, how? Compute K and

rescale so that its element K33 is set to 1. Print K. What are the focal lengths of the camera in pixels?

What are the pixel coordinates of the image center of the camera?

## Part II: Camera Calibration using 2D calibration object (70 points)

In this part we are going to implement camera calibration from multiple images of 2D planes. Additionally

we will learn how to augment images with virtual objects. We will follow the method described in the

book chapter on camera calibration by Zhengyou Zhang which was proposed in his paper Flexible Camera

Calibration by Viewing a Plane from Unknown Orientations – Zhang, ICCV99. Start by carefully reading

Section 2.4 of that Chapter.

## Calibration Grid and images

We know the following about the grid. The grid is 9 squares in width and 7 in height. Each square is 30mm

x 30mm. If we select the bottom left corner of the grid to be the origin of the world coordinate system, and

the grid to be the plane corresponding to Z=0 , then we know the 3D coordinates of each corner in that

grid.

For calibration we will use four images: images2.png, images9.png, images12.png, images20.png

2

## Corner Extraction and Homography computation (10 points)

First we want to extract the four corners of the calibration grid form each image. We will use the grid

corners to estimate the homographies relating two images. We can manually get the four grid corners from

each image. One way to let a user manually select points in matlab is using ginput function.

Once the 4 corners are extracted, compute the homography H that relates the grid 3d coordinates to

the corners. Use the function homography2d which is provided. Repeat this for the four images provided.

Report the computed H for each image [deliverable].

## Computing the Intrinsic and Extrinsic parameters (30 points)

Now given the four homographies, follow the instructions in section 2.4.4 to compute the intrinsic parameters

and extrinsic parameters. We need to linearize the two constraints in Eq 2.19 and 2.20 into two equations in

a homogeneous system as in Eq 2.25.

Then solve for b and estimate the intrinsic parameters as described in

page 21. Print the computed matrix B and the intrinsic parameters. Then compute and print R, t for each

image [deliverable].

Verify that your rotation matrix is in fact a rotation matrix, print RT R, is it an identity as it should be?

We can enforce R to be a rotation matrix by SVD decomposition of R and setting the singular values to

ones, i.e., set the rotation matrix to UV T where R = UΣV

T

.

Print the new R and RT R after enforcing the rotation matrix constraints. [deliverable]

## Improving accuracy (30 points)

Since we used four manually entered points to compute the homography. A small error in one of the points

will directly eect the computed homography. To x this we are going to estimate the homography from all

grid points.

• First given the computed homographies from Section 2, compute the approximate location of each

grid corner in the image. (Hint : This can be done since we know the 3d locations of the grid corners

and the approximate homography. Call these points p_approx. Create a gure with the image and

approximate grid locations. Call this Figure 1 : Projected grid corners [deliverable]

• Second, using the provided Harris function detect Harris corners in the image and display them. Use

the following parameter values for the Harris detection : sigma = 2, thresh = 500, radius = 2.

[cim, r, c, rsubp, csubp] = harris(rgb2gray(im), sigma, thresh, radius, disp);

Here r is the y-coordinate of the Harris corner, c is the x-coordinate of the Harris corner, rsubp is

the y coordinate with subpixel accuracy, csubp is the x coordinate with subpixel accuracy. Use rsubp,

csubp. Create a gure with image and overlayed Harris corners. Call this Figure 2 : Harris corners .

[deliverable]

• Third, compute the closest Harris corner to each approximate grid corner. (You may nd it useful to

use the dist2.m function provided). Let these closest Harris corners be p_correct. Create a gure with

the image and p_correct overlayed. Call this Figure 3 : grid points . [deliverable]

• Finally, compute a new homography from p_correct, print H [deliverable]

• Repeat this for the other three images. Then use the homographies to estimate K and R, t for each

image. Report your K, R’s, and t’s [deliverable]. Save your results, you will need to use them in

Part III

• Using the new computed H, compute the errors between points in p_correct and points you get by

projecting grid corners to the image (Hint there is no need to use R, t for projecting) . Call this

err_reprojection. Report your result. [deliverable]

• Now repeat the process using 4 images. Compare your results to your previous results and those of

part 2 [deliverable].

• Can you suggest a way this can be done automatically (i.e without rst letting the user manually select

the 4 corners) ?

3

## Part III: Augmented Reality 101 (50 points)

## Augmenting an Image (30 points)

Now we would like to use our computed homographies from part II to map a clip art image onto the grid

such that it seems to be part of the grid. The image should be synthesized such that the clip art bottom left

corners is the same as the grids (0,0) corner.

When tting the clip art you should rescale it to t the grid

while keeping the clip art aspect ratio. Using your computed homography nd a way to map your image

on the grid such that you image will have the same projective distortion as the grid. If the clip art have

any white pixels you should make these pixels appears transparent when overlayed over the grid.

For each

image of the four images in Part II, create a gure with the original image and your virtual clip art overlayed

over the grid.

Your image should be one of the images provided in the clipart directory. To nd which clip

art you are supposed to map, take the last 4 digits of your ruid id add them up and use the clip art le

corresponding to the rst digit in the sum, i.e., if the rst 4 digits in your RUID are 5243, use the clip art

4.xxx .

## Augmenting an Object (20 points)

Now we would like to augment our images with 3D objects. For our purposes we are going to use a cube as

a virtual object. We will only render the cube as a wire frame and we would like its base to be locate on

the 3×3 grid of squares in the bottom left corner of our grid.

The cube should be standing up from the grid.

First print the 3D coordinates of the cube. Then, nd a way to use your computed H, K, R, t to synthesize

new images with the virtual cube inserted.

Extra credit (20 points)

Do one of the following :

1. Instead of augmenting a cube, augment a general mesh from a 3D le of your choice.

2. Can you nd a way to estimate the intrinsic and extrinsic parameters from only two images of the

grid. What assumptions on the intrinsic parameters are needed to achieve this. (Hint the answer can

be found in Sec 2.4)

4