Description
Assignment-2 has two tracks: competition track and reseach track.
Students should attend one of the two tracks. 2 or 3 students are
suggested to form a group to attend one of these two tracks.
1. Competition track description [100 Marks]:
In this assignment, you are to solve the multi-label classification task. Each
sample in this dataset includes:
• an image,
• one or more (up 20) labels,
• a short caption that summarizes the image.
Your goal is to implement an image classifier that predicts the labels of image
data sample. You may optionally include the caption in the input of your
classifier — it’s up to you!
Please submit your submission file via Kaggle
https://www.kaggle.com/c/2020s1comp5329assignment2/overview/descripti
on.
Remember: the ranking contributes to 20% of your assignment mark.
Please make sure you name your team in the following format.
{unikey1}_{unikey2}_{unikey3}
The evaluation metric for this assignment is Mean F1-Score. The F1 score,
commonly used in information retrieval, measures accuracy using the
statistics precision p and recall r. Precision is the ratio of true positives (tp) to
all predicted positives (tp + fp). Recall is the ratio of true positives to all actual
positives (tp + fn). The F1 score is given by:
The F1 metric weights recall and precision equally, and a good retrieval
algorithm will maximize both precision and recall simultaneously. Thus,
moderately good performance on both will be favored over extremely good
performance on one and poor performance on the other.
Submission Format
For every image in the dataset, submission files should contain two columns:
image id and labels. Labels should be a space-delimited list.
For example
ImageID,Labels
1.jpg,1
8.jpg,8
9.jpg,9 10
10.jpg,10 9
etc.
You can use any methods in deep learning to accomplish the classification
task. You must guarantee that the submitted codes are self-complete, and can
be successfully run in common python3 and PyTorch environment.
Instructions to hand in the assignment
1.1 Go to Canvas and upload the following files/folders compressed
together as a zip file
a) Report (a pdf file)
The report should include each member’s details (student ID and name)
b) Code (a folder)
i. Algorithm (a sub-folder)
Your code (could be multiple files or a project)
ii. Input (a sub-folder)
Empty. Please do NOT include the dataset in the zip file as they
are too large.
iii. Output (a sub-folder)
“Predicted_labels.txt” – This file contains the predicted labels of
test exampels. You may want to submit the prediction that
achieves the best performance on kaggle.
If you work as a group, only one student needs to submit the zip file
which must be named as student ID numbers of all group members
separated by underscores. E.g. “xxxxxxxx_xxxxxxxx_xxxxxxxx .zip”
1.2 Your submission should include the report and the code. A plagiarism
checker will be used. Clearly provide instructions on how to run your
code in the appendix of the report.
1.3 The report must clearly show (i) details of your modules, (ii) the
predicted results from your classifier on test examples, (iii) run-time,
and (iv) hardware and software specifications of the computer that you
used for performance evaluations.
1.4 There is no special format to follow for the report but please make it
as clear as possible and similar to a research paper.
1.5 Remember, the due date to submit them on Canvas is 29-May-2020,
18:00
Late submission:
Suppose you hand in work after the deadline:
If you have not been granted special consideration or arrangements
– A penalty of 5% of the maximum marks will be taken per day (or part)
late. After ten days, you will be awarded a mark of zero.
– e.g. If an assignment is worth 40% of the final mark and you are one
hour late submitting, then the maximum marks possible would be 38%.
– e.g. If an assignment is worth 40% of the final mark and you are 28
hours late submitting, then the maximum marks possible marks would be
36%.
– Warning: submission sites get very slow near deadlines – Submit early;
you can resubmit if there is time before the deadline.
Marking scheme
Category Criterion Marks Comments
Report [70]
Introduction [10]
– What’s the aim of the study?
– Why is the study important?
– The general introduction of your
used method in the assignment and
your motivation for such a solution.
Related works [10]
– Existing related methods in the
literature.
Techniques [20]
– The principle of your method used
in this assignment.
– Justify the reasonability of the
method.
– Any advantage or novelty of the
proposed method.
Experiments and results [20]
– Accuracy/efficiency (Figures or
Tables)
– Extensive analysis (ablation
studies, comparison methods, hyper
parameter analysis)
Conclusions and Discussion [5]
– Meaningful conclusion and
discussion.
Other [5]
– At the discretion of the marker: for
impressing the marker, excelling
expectation, etc. Examples include
fast code, using LATEX, etc.
Code [10] Code runs within a feasible
time [5]
Size of resulting deep models
for the prediction < 100MB [5]
Classification
performance [20]
Groups in top 10% [20]
Groups in top 10%-30% [15]
Groups in top 40%-60% [10]
Groups in top 60%-80% [5]
Others [3]
Penalties [-]
Badly written code: [-20]
Not including instructions on
how to run your code: [-30]
Late submission
2. Research track description [100 Marks]:
This research track calls for brave new ideas on deep learning. In this track,
you are encouraged to propose and investigate new algorithms or problems in
deep learning.
You must not use the project (e.g. your capstone or SSP project) that you have
aleady done or are currently doing in other units to participate in this track.
You are encouraged to contact our teaching team, if you want to have some
discussion on your proposed research problems.
Marking scheme
Category Criterion Marks Comments
Report [70]
Introduction [10]
– What’s the aim of the study?
– Why is the study important?
– The general introduction of your used
method in the assignment and your
motivation for such a solution.
Related works [10]
– Existing related methods in the
literature.
Techniques [20]
– The principle of your method used in this
assignment.
– Justify the reasonability of the method.
– Any advantage or novelty of the
proposed method.
Experiments and results [20]
– Accuracy/efficiency (Figures or
Tables)
– Extensive analysis (ablation studies,
comparison methods, hyper parameter
analysis)
Conclusions and Discussion [5]
– Meaningful conclusion and discussion.
Other [5]
– At the discretion of the marker: for
impressing the marker, excelling
expectation, etc. Examples include fast
code, using LATEX, etc.
Novelty
[20]
The novelty of the proposed project
and its solution.
Presentation
[10]
A presention on zoom (week 13).
Penalties [-] Late submission
Submite the report, source codes and slides on Canvas before the due date 29-
May-2020, 18:00
If you have any question about the assignment, please contact:
Shumin Kong skon2020@uni.sydney.edu.au
