## Description

Question 1 (10%)

1. Let x ∈ R be a random variable with mean E[x] = µ. Show that the variance of x, E[(x−µ)

2

]

is E[x

2

]− µ

2

.

2. Let x ∈ R

n be an n-dimensional random vector with mean E[x] = µ. Show that the covariance

matrix of x, E[(x− µ)(x− µ)

T

] is E[xxT

]− µµT

.

Question 2 (20%)

Suppose that a real-valued measurement x is generated by one of two equally probable probability density functions (pdfs), both of the parametric form p(x|L = l) ∝ e

−|x−al

|/bl for l ∈ 1,2 and

bl > 0 for both pdfs.

1. Find the normalization factor for this parametric pdf family as a function of the respective a

and b parameters, such that these pdfs integrate to 1 along the real axis.

2. Determine the simplest analytical expression for the log-likelihood ratio between class labels

1 and 2 evaluated at a given x, `(x) = lnp(x|L = 1)−lnp(x|L = 2).

3. Generate a plot of this log-likelihood-ratio function for the case a1 = 0,b1 = 1 and a2 =

1,b2 = 2 using a suitable programming language. Label the axes of the plot properly as `(x)

and x using legible font sizes, and include an intelligible title and caption for the plot. In

all future assignments, this will be expected without explicit instruction. All figures must be

properly labeled, titled, and captioned. If there are multiple curves or data visualizations in

the plot also include a legend to help identify each component.

Question 3 (20%)

Consider a two-class setting where class prior probabilities are P(L = 1) = P(L = 2) = 1/2. We

measure a one dimensional real-valued feature variable x, which takes values from Uni f orm[a,b]

for class 1, and Uni f orm[r,t] for class 2. Here a < r < b < t, so the support of these two uniform

class-conditioned feature pdfs p(x—L=1) and p(x—L=2) overlap. Derive a minimum probability

of error classification rule.

1

Question 4 (30%)

Consider a two-class setting with equal class priors and a real-valued feature with Gaussian

class-conditioned pdfs N (0,1) and N (µ,σ

2

).

1. Derive an expression for the classification/decision rule that achieves minimum probability

of error. This will be a function of x that depends on parameters µ and σ. Please simplify

the expression to a nice form.

2. For the case µ = 1 and σ

2 = 2 generate plots that visualize the class-conditional pdfs p(x|L =

l) for l ∈ 1,2, as well as class posterior probabilities p(L = l|x) for l ∈ 1,2. Demonstrate the

decision boundary you found earlier in these visualizations.

3. Numerically estimate the achieved minimum probability of error for the specific case you

visualized above. Describe your procedure and the numerical result intelligibly.

4. Consider the case where µ = 0 and σ 1 for the second class conditional pdf. Discuss

what happens to the decision boundary and regions. Imagine and briefly describe a practical

situation when this kind of data distribution might arise.

Question 5 (20%)

The pdf of an n-dimensional random vector drawn from N (µ,Σ) is

p(ζ ) = (2π)

−n/2

|Σ|

−1/2

e

−

1

2

(ζ −µ)

TΣ

−1

(ζ −µ)

(1)

1. Determine the pdf of x: Let x = Az + b be an n-dimensional random vector where A ∈

R

n×n

and b ∈ R

n

are fixed and known, z ∼ N (0,I). Assuming that A is full rank, and by

using/citing an appropriate theorem from probability theory, show that x ∼ N (b,AAT

).

2. Determine Linear Transformation: We want to generate a random vector drawn from

N (µ,Σ) using the linear transformation technique introduced and studies above. Determine

A and b to achieve the desired mean vector and covariance matrix. Specifically, determine

A and b in terms of µ and Σ.

3. Implement Code: Write code that takes in N, n, µ, and Σ and produces N samples of independent and identically distributed (iid) n-dimensional random vectors {x1,…,xN} drawn

from N (µ,Σ) using the linear transformation technique applied to samples of z ∼ N (0,I).

2