Description
1)
a) Implement logistic-regression gradient descent, writing your own function
ThetaStar =
GradDescentLogistic(x,y,eta,epsilon,StartingTheta,StopTime)
First 5 parameters should be clear from class. StopTime is time in seconds after which you
decide the program is not converging.
[20 points]
b) Interactively work with Matlab Write to test GradDescentLogistic . Save the commands
you run into a script TestGDL.m that should:
i) Set StopTime to 60
ii) Choose N=100 x values in D=8 dimensions randomly & uniformly within the range (-5,5)
D
iii) Use SimLogistic with =0 to generate N training classification values y, and evaluate
your classifier using M=20 test datapoints generated with the same parameters.
iv) Perform multiple runs on the same input , with the same and different
eta,epsilon,StartingTheta with eta,epsilon that you set as you wish and
StartingTheta chosen randomly in (-10,10)D
.
Perform as many runs as you see fit, till you decide on parameters eta,epsilon you think are
good (call them ). Measure the time it took GradDescentLogistic to converge for each
run. The last run is with .
Report:
TestGDL.m the script that does all this
Inputs.txt D+1 columns, N+M rows detailing x,y (last column is y, last M rows are test)
RealThetas.txt One column vector of D+1 entries (the first one for the free coefficient)
Runs.txt 2D+7 columns with a row for each runs. First two columns report:
eta,epsilon ; next D+1: StartingTheta; next D+1: optimized values;
next 2 columns: final value of the loss on training and test data; last column:
runtime in seconds
[45 points]
c) Rerun with eta
for consecutive (positive and negative) integers k to demonstrate the
problems with too large/small eta. Report EtaRuns.txt, formatted exactly as Runs.txt
above, and TestEta.m the script that produces it.
[15 points]
2) Consider the neural network in the attached NN.A4.2.pdf, with two inputs, denoted by subscripts 1
and 2; two 1st
-level logistic neurons, denoted by subscripts 3 and 4; two 2nd
-level logistic neurons,
denoted by subscripts 5 and 6; and a single 3rd
–level logistic neuron, denoted by the subscript 7. The
intercept for all neurons is modeled as respective coefficients for x0=1 . Write down explicit update
equations for a gradient descent process with logistic loss.
[20 points]
Good luck!
