Description
Part A: Theory (15pts)
(5pts) Linear Regression and Regularization
In class we derived and discussed linear regression in detail. Find the result of minimize the
loss of sum of the squared errors; however, add in a penalty for an L2 penalty on the weights.
More formally,
argmin
w
(
X
i
(w
T
xi − yi)
2 +λkwk
2
2
)
(1)
How does this change the solution to the original linear regression solution? What is the
impact of adding in this penalty?
(5pts) Density Estimation
In k-nearest neighbors (KNN), the classification is achieved by majority vote in the vicinity of
data. Suppose there are two classes of data each of n/2 points overlapped to some extent in
a 2-dimensional space. Describe what happens to the training error (using all available data)
when the neighbor size k varies from n to 1.
(5pts) Feature Selection & Preprocessing
A friend asks you for some help with a feature selection project. Your friend goes out and collects
data, D, for their project. Using D, your friend tries many subsets F ⊂ X by adapting F based
on the error of a classifier. They return F that corresponds to the smallest classification error.
This is the procedure they carry out to validate the impact of the feature selection routine.
This procedure is repeated
• Make a new data set D
0 with F features using the feature selection routine.
• Repeat 50 times
– Split D
0
into randomized training & testing sets (80/20% splits)
– Train a classifier and record its error
• Report the error averaged over 50 trials
Critique and respond to how your friend performed their analysis.
Part B: Practice (25pts)
You are free to use functions already implemented in Matlab, Python or R with the exception of
problem 1. I recommend using Python’s Scikit-learn (http://scikit-learn.org/stable/)
as is implements most of the methods we will be discussing in this course. . . as well as problems
in this homework!
arizona.edu 2 January 29, 2017
Gregory Ditzler Dept. of ECE University of Arizona
(10pts) Logistic Regression on Synthetic and Real-World Data
Write your own implementation of logistic regression and implement your model on either realworld (see Github data sets), or synthetic data. If you simply use Scikit-learn’s (R’s, Matlab’s,
or another builtin procedure) implementation of the logistic regression classifier then you’ll
receive a maximum of 6 out of 15 points. A full 10/10 will be awarded to those that actually
implement logistic regression using the optimization of cross-entropy using stochastic gradient
descent.
(5pts) Dimensionality Reduction + 2 Bonus
Choose 10 data sets of your choice from the ECE523. Implement a comparison between either
two classifiers of your choice, or a classifier with does and does not using a preprocessing step
(i.e., feature selection, PCA, etc.), and report the accuracies of the two models in table for (e.g.,
a 10 × 2 table of classifier accuracies). For two bonus points use a hypothesis testing procedure from homework #1 to determine if there is statistical significance (i.e., do both approaches
perform equally well).
(10pts) Density Estimation in Practice
The ECE523 Github page has code for generating data from a checkerboard data set. Generate
checkerboard data from two classes and use any density estimate technique we discussed to
classify new data using
Pb(Y |X) =
Pb(X|Y )Pb(Y )
Pb(X)
where Pb(Y |X) is your estimate of the posterior given you estimates of Pb(X|Y ) using a density
estimator and Pb(Y ) using a maximum likelihood estimator.
arizona.edu 3 January 29, 2017
