Description
1. Let X denote an n × p matrix with each row an input vector and y denote an
n-dimensional vector of the output in the training set. For fixed q ≥ 1, define
Bridgeλ
(β) = (y − Xβ)
T
(y − Xβ) + λ
X
p
j=1
|βj
|
q
for λ > 0.
Denote the minimal value of the penalty function over the least
squares solution set by
t0 = min
β:XT Xβ=XT y
X
p
j=1
|βj
|
q
.
(a) Show that Bridgeλ
(β) for λ > 0 is a convex function in β, which is strictly
convex for q > 1.
(b) Show that for q > 1 there is a unique minimizer, βˆ(λ), with Pp
j=1 |βˆ
j (λ)|
q ≤
t0.
(c) Show that for q = 1 there exists a minimizer and for all minimizers, βˆ(λ),
the penalty function takes the same value
s(λ) ,
X
p
j=1
|βˆ
j (λ)|
q ≤ t0.
Thus for q ≥ 1, s(λ) is well defined as a function of λ on the interval (0, ∞).
(d) Show that minimizing Bridgeλ
(β) is equivalent to minimizing
(y − Xβ)
T
(y − Xβ) subject to X
p
j=1
|βj
|
q ≤ s(λ).
2. Exercise 1 (Section 6.8) of [ISL] (p. 259).
3. Exercise 3.16 of [ESL] (p. 96).
4. The prostate data described in Chapter 3 of [ESL] have been divided into a
training set of size 67 and a test set of size 30. Carry out the following analyses
on the training set:
(a) Best-subset linear regression with k chosen by 5-fold cross-validation.
(b) Best-subset linear regression with k chosen by BIC.
(c) Lasso regression with λ chosen by 5-fold cross-validation.
(d) Lasso regression with λ chosen by BIC.
(e) Principle component regression with q chosen by 5-fold cross-validation.
For each analysis, compute and plot the cross-validation or BIC estimates of the
prediction error as the model complexity increases as in Figure 3.7. Report the
final estimated model as well as the test error and its standard error over the
test set as in Table 3.3. Briefly discuss your results.
