Description
Task (20 point)
You are a new hire in a mid-size company. You have just completed your
fifth task.
The last months were exhausting. You really need a vacation. You went
to HR and scheduled some days off in May. You were about to return to your
desk, when you noticed your CEO.
“Good to see you! I have another assignment for you. Come with me!”,
she called you. You entered the well familiar meeting room. Your technical
manager and the developer were already there.
The CEO said: “You did a splendid work. We are all happy with your
performance. We should soon have a conversation about your promotion.
We will get a few new summer interns, you might take one or two and train
them. And for this, you should have an upgraded title – what would you
think about Lead Data Scientist? We can talk about the details of your
promotion later, maybe end of the month.
In the meantime, we have another task for you. Los Angeles Department
Of Water and Power is developing a new weather predictive system. They
wish to predict the temperature for the next 24 hours.
You will get historical records from our city, about 5 years. Your task is
to propose a model that is capable of predicting temperature based on the
recent history. You should report how good your model is – and how long in
the future you can predict with a reasonable accuracy (how many hours or
how many days).
I would like to see a preliminary report in two weeks. We will meet again
on Thursday, April 15, at 10 am.”
Hint: This task is called a time series forecasting. As a help and inspiration, read
“Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow”, 2nd Edition,
Chapter 15 as well as the official TensorFlow tutorial, ht tp s: // ww w. te ns or fl ow .o
rg /t ut or ia ls /s tr uc tu re d _d at a/ ti me _s er ie s . You can also look for other materials (just always cite the sources!).
Remember, we do not grade on a curve. You don’t have to be better then your collegues
to get 100% from that task. Your model can have a moderate performance – but as long
as your training and testing methodologies are good, and as long as everything is well described, you can get 100% for this problem set. Remember, it is always important to know
limits of you model. In other words, if your model doesn’t work well – but you can detect
it, and maybe explain the reason why it fails – such a solution will not be treated as wrong
or not-completed.
Your Technical Manager said: “It is up to you, how you will approach
this problem. However, my suggestion is that you should start from constructing some baselines, that exploit the seasonality of the temperature
(yearly and daily). Then, you can train some (more advance) models that
predict either a single timestep or several timesteps at once. The most natural choice is to use recurrent neural networks, but you can also start from
some simpler architectures (not recurrent).
It can be useful to have some
simple models first – you will be able to use the performance of those simpler models as your reference point – so when you train your recurrent neural
network, you will be able to easily tell, if you make progress or not. By the
way, you do not have to test all approaches that are mentioned in the literature – just pick a few that you believe are the most suitable and compare
them.
I also noticed, that the data are quite rich. You have information not only
about the temperature but also about humidity, pressure, general weather
conditions, etc. Your baseline models do not have to use all those information.
Finally, if you can, try to measure and report how important those different features are. It could be useful to know how useful the information about
the humidity or pressure, or wind are for your predictive task (predicting the
temperature).”
Hint: There are different ways how to measure the feature importance. One (easy)
method is to use the permutation importance technique, see ht tp s: // sc ik it -l ea rn
.o rg /s ta bl e/ mo du le s/ pe rm ut at io n _i mp or ta nc e. ht ml . You can also train two
models, one with only temperature, another with some selected features – and you can
compare the precision of your predictions. If you run out of the time, you don’t have to
measure all combinations – just some, so we know that you know how to do it.
The senior developer took you aside and said: “My task will be to
maintain your code. Please, write comments and try to explain any nontrivial
part of your code!”
2
Hint: If you use Jupyter Notebooks, remember that you can also add special “markdown
cells”. You can use it to split your notebook into a few logical parts – see also post @87 on
Piazza.
Learning Objective: You will be able to work with time series and you will be able to
predict future events.
Data
There are two data files.The first file, ps6_trainvalid.csv, contains labeled
training and validation data. You can use the second file, ps6_test.csv, to
test the final versions of your models.
Report
To help you, I prepared a template for you, see https://www.overleaf
.com/read/qcbcsdscvcdb. You are encouraged to use the template, but if
you prefer to use another editor – or to made some modifications – this is
acceptable. Just remember to always export your report to the pdf format
at the end.
Report Submission
Submit your report (a pdf file) using Blackboard (check the “Assignments”
section on Blackboard page of our class).
Grading Rules
In order to grade your work, we will role-play the following situation. We
will assume, that you are a new-hire in our company. You were asked to
provide a comprehensive technical report, that illustrates your findings. We
will evaluate it from the perspective of three people.
• Your CEO (she would like to hear the high-level stuff; She will probably
read only the abstract and the conclusions, maybe she will also look
at the main figure or the main plot). (4 points for the report)
• Your manager (he would like to see a detailed report; he might also
look at some parts of the code). (6 points for the report and 2 points
for the code)
• A senior developer (they would like to see your code; they might not
read the report at all). (8 points for the code)
3
Your final score is the sum of scores given by each person. Cumulatively,
there are 10 points for the report and 10 points for the code.
Don’t Panic
Don’t panic. We understand, that this is a large, open-ended task. We also
understand, that this might be only your sixth technical report that you
were tasked to write. We are dedicated to help you do the best work – and
while we keep high standards for you (and for us) – in the same time, we
acknowledge, that you have limited time and limited resources to complete
your task. This report doesn’t have to be perfect to give you 100% score.
If you don’t know where to start, read https://www.tensorflow.org/t
utorials or optionally, the Chapter 16 from “Hands-On Machine Learning
with Scikit-Learn, Keras, and TensorFlow”, 2nd Edition by Aurélien Géron.
If something is not clear, ask questions on Piazza.
4
