Description
Questions:
1. (20 points) An Alternative Bernoulli Model: Consider the probability mass function for X ∈ {0, 1}
shown below, which is an alternate parameterization of the standard Bernoulli distribution. Use this model
to answer the following questions.
P(X = x) =
1
1 + exp(−λ)
[x=1]
1
1 + exp(λ)
[x=0]
a. (5 pts) Show that this distribution is properly normalized.
b. (10 pts)Suppose we have a data set D containing a one’s and b zero’s. Derive the MLE of λ given D as
1
the data set. You may assume that a and b are both greater than zero. Show your work.
c. (5 pts) What is the mathematical relationship between this parameterization of the Bernoulli distribution
and the standard parameterization of the Bernoulli distribution? Explain your answer.
2. (40 points) Logistic Regression with an Informative Prior: Recall that standard `2 regularization for
logistic regression is equivalent to MAP estimation of the model parameters under a zero-mean, spherical
Gaussian prior. In some situations, including transfer learning, we may have an informative prior for the
model parameters including a non-zero prior mean [w0, b0]. Assuming a spherical Gaussian around this
mean results in the following learning objective function. Use this learning objective to answer the following
questions.
L(D, θ) = X
N
n=1
log(1 + exp(−yn(wxT
n + b))) + λkw − w0k
2
2 + λ(b − b0)
2
a. (5 pts) What is the gradient of L(D, θ) with respect to w and b? Show your work.
b. (20 pts)Starting from the provided template, implement a Scikit-Learn compatible class for this model
including fit, predict, set_params, and get_params, as well as functions for computing the objective function
and gradient. As your answer to this question, describe your approach to learning the model parameters in
your report and submit your commented code for auto grading as described above.
c. (10 pts) Using the provided data and prior mean parameters w0, b0, perform an experiment where the
model is learned using λ = 0 and λ = 10 while varying the number of training cases from 10 to 400 in steps
of 10. Provide one figure containing two line plots showing the test accuracy as a function of the number of
training data cases for both values of λ.
d. (5 pts) Explain why the relative performance of using λ = 0 and λ = 10 changes as the amount of
training data increases.
3. (40 points) Generalized Robust Regression: One important problem with standard least-squares linear
regression is that outliers can significantly corrupt learning. Consider the generalized robust regression loss
function shown below. The loss switches from being a polynomial of degree 2k to a linear function at
the point δ. Both the point δ and the polynomial order parameter k are parameters of the this loss. δ is a
non-negative real number and k is a positive integer (k = 1, 2, 3…).
Lk,δ(y, y0
) = (
1
2k
(y − y
0
)
2k
… if |y − y
0
| ≤ δ
δ
2k−1
(|y − y
0
| − (2k−1)
2k
δ) … otherwise
Using this loss for a linear regression model f(xn, θ) = wxT
n + b results in the learning objective
Lk,δ(D, θ) = X
N
n=1
Lk,δ(yn, f(xn, θ))
Use this model, objective function, and loss to answer the following questions. Assume that x ∈ R
D and
y ∈ R in your derivations and code.
2
a. (5 pts) What is the gradient of Lk,δ(D, θ) with respect to w and b? Show your work.
b. (20 pts)Starting from the provided template, implement a Scikit-Learn compatible class for this model
including fit, predict, set_params, and get_params, as well as functions for computing the objective and
gradient. As your answer to this question, describe your approach to learning the model parameters in your
report and submit your commented code for auto grading as described above.
c. (5 pts) Using the provided data set, apply the model using k = 1 and δ = 1. Also apply ScikitLearn’s standard least-squares linear regression model sklearn.linear_model.LinearRegression. Report the
MSE achieved by both models on the train data set.
d. (5 pts) Provide a single figure that includes a scatter plot of the training data, the regression line found
using the robust model, and the regression line found using the standard least-squares model.
e. (5 pts) Based on this plot, which model do you think would have better generalization performance on
future test data? Explain your answer.
f. (5 pts) Bonus: Explore other values of k and δ. How does changing these values affect the learned
regression line? Supply additional figures to support your answer. How would you choose optimal values
of k and δ?
3