Description
1. (a) Why is it more efficient to process data points if they are lower-dimensional
vectors? State one reason.
(b) What is a potential trouble of reducing the dimensionality of input vectors before training a classifier? State one reason.
2. (a) Given a training set D = {x1,…,xN}, show that the reconstruction error of
principal component analysis (PCA) could be written down as
1
N
N
∑
n=1
kxn −xˆnk
2
2 =
d
∑
j=q+1
w
>
i Cwi
,
where wi
is the i-th principal component or the eigenvector of the input covariance
matrix C.
(b) Show that
Σ = W>CW
⇐⇒ σ
2
j = w
>
j Cwj
, for all j = 1,…,d,
where W is the weight matrix of PCA, C is the input covariance matrix, and
Σ = diag(σ
2
1
,…,σ
2
q
) =
σ
2
1
0 ··· 0
0 σ2
2
··· 0
.
.
. 0 ···
.
.
.
.
.
.
.
.
. ···
.
.
.
0 0 ··· σ
2
q
is the covariance matrix of the code vectors.
1
3. (Programming Assignment) Complete the implementation of PCA and NMF using
Python and scikit-learn. The completed notebooks must be submitted together with the
answers to the questions above.
When submitting Jupyter notebooks, make sure to save printed outputs as well.
PCA https://github.com/nyu-dl/Intro_to_ML_Lecture_Note/blob/
master/homeworks/hw6_pca.ipynb
NMF https://github.com/nyu-dl/Intro_to_ML_Lecture_Note/blob/
master/homeworks/hw6_nmf.ipynb
2
