Description
1. See period 11.ipynb.
2. Let a 4-by-2 matrix X have SVD X = USV T where U =
1
2
1 1
1 −1
1 −1
1 1
, S =
”
1 0
0 γ
#
, and V = √
1
2
”
1 1
1 −1
#
.
a) Express the solution to the least-squares problem arg minw ||Xw −y||2
2
as a function of U, S,V , and y.
b) Let y =
1
0
0
1
. Find the weights w that minimize ||Xw − y||2
2
as a function of
γ. Calculate ||Xw − y||2
2
and ||w||2
2
as a function of γ for this value of w. What
happens to ||w||2
2
as γ → 0?
c) Now consider a “low-rank” inverse. Instead of writing
(XTX)
−1XT =
X
p
i=1
1
σi
viu
T
i
where p is the number of columns of X (assumed less than the number of rows),
we approximate
(XTX)
−1XT ≈
Xr
i=1
1
σi
viu
T
i
In this approximation we only invert the largest r singular values, and ignore all
of them smaller than σr. If r = 1, use the low-rank inverse to find w, ||y−Xw||2
2
,
and ||w||2
2 when y =
1
0
0
1
as in part b). Compare ||y − Xw||2
2
, and ||w||2
2
to
the results for part b).