## Description

## Problem 1: Poisson Maximum Likelihood Estimation (20pts)

Consider a nonnegative, integer-valued random variable X that is distributed according to a Poisson

distribution X ∼

λ

xe−λ

x!

for some real-valued parameter λ > 0.

1. Given data samples x

(1), . . . , x(m)

, what is the maximum likelihood estimate for λ?

2. Suppose now that you introduce a prior probability distribution, λ ∼

1

5 max{−λ/10 + 1, 0}.

What is the MAP estimate under this prior probability distribution?

3. Why might you not prefer a prior probability distribution like the above for this estimation

task? What might be a better prior?

## Problem 2: Log-Normal Maximum Likelihood Estimation (20pts)

Consider a positive, real-valued random variable X that is distributed according to a log-normal

distribution: X ∼

1

xσ√

2π

exp(−

(ln x−µ)

2

2σ2 ) for real-valued parameters µ and σ with σ > 0.

Suppose

that you are given M data observations x

(1), . . . , x(M)

.

1. For a given µ and σ, what is the log-likelihood of the data observations?

2. Find the maximum likelihood estimators for µ and σ. Note that the log-likelihood is not a

concave function of µ and σ.

3. Are the maximum likelihood estimators for µ and σ unbiased?

4. Why might a Gaussian distribution not be a good choice for a prior distribution over µ?

## Problem 3: Logistic Regression (30pts)

For this problem, consider the Sonar data sets from Homework 4.

1

1. Fit a logistic regression classifier to training data set. What is the accuracy on the test set?

Explain why in standard logistic regression, without any type of regularization, the weights

may not converge (even though the predicted label for each data point effectively does) if the

input data is linearly separable.

2. Fit a logistic regression classifier with an `2 penalty on the weights to this data set using the

validation set to select a good choice of the regularization constant. Report your selected

constant, the learned weights and bias, and the accuracy on the test set.

3. Fit a logistic regression classifier with an `1 penalty on the weights to this data set using the

validation set to select a good choice of the regularization constant. Report your selected

constant, the learned weights and bias, and the accuracy on the test set.

4. Does `1 or `2 tend to produce sparser weight vectors?

## Problem 4: Gaussian Na¨ıve Bayes (30pts)

For this problem, consider the Sonar data set from Homework 4. Suppose that you want to fit a

Gaussian NB model to this data. That is, assume that the probability distribution is of the form

p(x1, . . . , xn, y) = p(y)

Yn

i=1

p(xi

|y)

where p(xi

|y) is a distinctly parameterized normal distribution, i.e., p(xi

|y) is parameterized by

µi,y and σi,y.

1. Given a data set with m continuous features, what is the log-likelihood of the Gaussian NB

model? Compute the MLE for each of the model parameters.

2. Fit a Gaussian NB model to the training data. What is the accuracy of your trained model

on the test set?

3. What kind of prior might make sense for this model? Explain.

4. Do you think the NB assumption is reasonable here?

2