RMIT Classification: Trusted
COSC2673 MACHINE LEARNING 2024
Assignment 2
Machine Learning Project
Weight: 50% of the final course mark
Due Date: 5:00 pm 15th of May 2024
Learning Outcomes: This assignment contributes to CLOs: 1, 2, 3, 4, 6
In this assignment you will design and create an end-to-end machine learning system for a real-world problem.
This assignment is designed for you to apply and practice skills of critical analysis and evaluation to
circumstances similar to those found in real-world problems. This is a group project (max 2 students).
In this assignment you will:
• Design and Create an end-to-end machine learning system
• Apply multiple algorithms to a real-world machine learning problem
• Analyse and Evaluate the output of the algorithms
• Research into extending techniques that are taught in class
• Provide an ultimate judgement of the final trained model(s) that you would use in a real-world setting
This assignment has the following deliverables:
1. A report (of no more than 5 pages, plus up to 2 pages for appendices) critically analysing your approach
and ultimate judgement.
2. An independent evaluation of your model and ultimate judgement (to be included in the report).
3. Your Python scripts, Jupyter notebooks, and software used to build your learning system and produce
the models and results.
To complete this assignment, you will require skills and knowledge from lecture and lab material for Weeks
1 to 12 (inclusive). You may find that you will be unable to complete some of the activities until you have
completed the relevant lab work. However, you will be able to commence work on some sections. Thus, do the
work you can initially, and continue to build in new features as you learn the relevant skills. A machine
learning learning model cannot be developed within a day or two. Therefore, start
early.
2
RMIT Classification: Trusted
This assignment contributes to the following course CLOs:
• CLO 1: Understand the fundamental concepts and algorithms of machine learning and applications.
• CLO 2: Understand a range of machine learning methods and the kinds of problem to which they are
suited.
• CLO 3: Set up a machine learning configuration, including processing data and performing feature
engineering, for a range of applications.
• CLO 4: Apply machine learning software and toolkits for diverse applications.
• CLO 6: Understand the ethical considerations involved in the application of machine learning.
You may optionally conduct this assignment in pairs (groups of 2). This is because many of the interesting
problems can be far more thoroughly investigated by working in pairs. The marking rubric does not evaluate
group work or differentiate between those working individually or in groups.
If you work in a group, both members will receive the same mark. Groups must be formed by no later than
Friday 19th of April Week 6 and registered on Canvas.If there are issues with group work, you must inform the
course coordinator by no later than 5.00pm Monday of Week 10 (13th of May), otherwise both group members
will receive the same grade, with no exceptions.
The mark each member receives depends on their contribution to the project. Peer evaluation and self
evaluation forms will be released and discussed on Week 6. These forms will be filled by each group member
and affect your marks.
Academic integrity is about honest presentation of your academic work. It means acknowledging the work of
others while developing your own insights, knowledge and ideas. You should take extreme care that you have:
• Acknowledged words, data, diagrams, models, frameworks and/or ideas of others you have quoted (i.e.
directly copied), summarised, paraphrased, discussed or mentioned in your assessment through the appropriate referencing methods
• Provided a reference list of the publication details so your reader can locate the source if necessary. This
includes material taken from Internet sites. If you do not acknowledge the sources of your material, you
may be accused of plagiarism because you have passed off the work and ideas of another person without
appropriate referencing, as if they were your own.
RMIT University treats plagiarism as a very serious offence constituting misconduct. Plagiarism covers a variety
of inappropriate behaviours, including:
• Failure to properly document a source
• Copyright material from the internet or databases
• Collusion between students
For further information on our policies and procedures, please refer to the following: https://www.rmit.edu.au/
students/student-essentials/rights-and-responsibilities/academic-integrity.
3
RMIT Classification: Trusted
Using machine learning in real-world settings involves a more than just running a data set through a particular
algorithm. In this assignment, you will design, analyse and evaluate a complete machine learning system.
For this assignment you have a choice of your project. You may select this project from the list in Section 3, or
you may negotiate a project with the course co-ordinator.
Regardless of the problem you choose, you must conduct the following tasks:
1. You need to come up with an approach, where each element of the system is justi
ed using data analysis, performance analysis and/or knowledge from relevant literature.
2. Investigate various Machine Learning solutions to the problem
3. Make an ultimate judgement
4. Evaluate your ultimate judgement against independent testing data
5. Produce a report of your design, investigation, evaluation and findings
Your investigation will require you to design, use, analyse and evaluate an end-to-end machine learning system.
You should consider a variety of techniques that have been discussed in class, and techniques you have researched.
Your end-to-end system may consist of elements such as:
• A well justified evaluation framework.
• Pre-processing the data set to make it suitable for providing to various machine learning algorithms.
• Carefully selected and justified baseline model(s).
• Hyper-parameter setting and tuning to refine the model.
• Evaluating the trained models, analysing and interpreting the results.
Each project features many of these above aspects. Each project also has unique aspects which cover a sample of
issues from across machine learning. Additionally, each project has unique mandatory requirement(s), detailed
for each project. The details of each project are listed in Section 3.
You must make an ultimate judgement of the "best" model that you would use and recommend for your
particular project. It is up to you to determine the criteria by which you evaluate your model and determine
what is means to be "the best model".
You need to conduct an independent evaluation of your ultimate judgement, using data collected completly
outside of the scope of your original training and evaluation. This evaluation simulates how your ultimate
judgement would perform if it were deployed in a real-world setting,where you are unable to re-train and adjust
the model.
The key aspect of this assignment is the design, analysis, and evaluation of your methodology, investigation, and results. This assignmentfocuses on both the accuracy of your model, and your understanding
of your approach and model.
The details in this spec are the minimum requirements. A thorough investigation must consider more
that the minimum to receive high grades.
4
RMIT Classification: Trusted
You must compile a report analysing the approach you have taken in your investigation. Your report:
• Must be no longer that 5 pages of text
• May contain an additional 2 pages for appendices
• Use a single-column layout with no less than size 11pt font
• The appendices may only contain citations, figures, diagrams, or data tables that provide evidence to
support the statements in your report.
• Include the name(s) and student id's of the student(s) who wrote the report.
In this report you should analyse elements such as:
• Machine learning algorithms that you considered
• Why you selected these approaches
• Evaluations of the performance of trained model(s)
• Your ultimate judgement with supporting analysis and evidence
This will allow us to understand your rationale. We encourage you to explore this problem and not just focus
on maximising a single performance metric. By the end of your report, we should be convinced that of your
ultimate judgement and that you have considered all reasonable aspects in investigating your chosen problem.
Remember that good analysis provides factual statements, evidence and justifications for conclusions that you
draw. A statements such as:
"I did <xyz> because I elt that it was good"
is not analysis. This is an unjustified opinion. Instead, you should aim for statements such as:
"I did <xyz> because it is more efficient. It is more efficient because . . . "
For your project:
1. You may choose from one of the below 3 suggested projects. Each project has unique aspects and will
allow you to explore different aspects of the ML field.
2. You may negotiate your own project. Note the special requirements and timeline fornegotiating a project.
Each project has different requirements, so ensure you are aware of these differences.
Project 1: Classify Images of Road Traffic Signs
This project is to train a model to classify images of European road traffic signs. You will be using a modified
version of the Belgium Traffic Sign Classification Benchmark. These are images of road traffic signs taken from
real-world vehicles. Note, this dataset, along with it's sister German TSC dataset, appear in many different
forms on various research and ML online resources. The data set for you to use in this assignment has been
specifically prepared for you, and is provided on Canvas.
The dataset consists of 28x28 gray-scale images and you are expected to use the dataset to perform two tasks:
• Classify images according to sign-shape, such as diamond, hex, rectangle, round, triangle.
• Classify images according to sign-type, such as stop, speed, warning, parking, etc.
Any over length content, or content outside of these requirements will not be marked. For example, if
you report is too long, ONLY the first 5 pages pages of text will be read and marked.
The key aspect of this assignment isn't your code or model, but the thought process behind your work.
5
RMIT Classification: Trusted
The correct classification of the images is given by the image sub-directories. Images are first sub-divided by
their shape, and then by their sign type. You should also note that some sign types have different individual
signs. For example, the speed-sign type has examples of signs of speeds from 10 - 70 mph. You are not required
to further sub-divide these signs, but consider all of the different signs as being of the sample type. Your tasks
is to investigate classifying the signs using both categories.
REQUIREMENTS
• You must investigate at least one supervised machine learning algorithms or each of the two categories
(Tasks). That is, you must build at least one model capable of classified the shape of the sign, and at least
one model capable of classifying the type of the sign.
• You are not required to use separate type(s) of machine learning algorithms, however, a thorough instigations should consider different types of algorithms.
• You are required to ully train your own algorithms. You may not use pre-trained systems.
• You may NOT augment this data set with additional data.
• Your final report must conduct an analysis and comparison between classifying the two categories.
INDEPENDENT EVALUATION
• Your independent evaluation should consist of classifying images of traffic signs that you have collected.
You will need to either take your own digital photographs of traffic signs, and/or source suitable signs
from internet resources. You will need to process these images so they may be used with your trained
algorithms.
• As part of your evaluation, you should discuss challenges you face in combining this independent data and
your models.
Project 2: Predict Energy Use
This project is to predict the energy use of lights and appliances in a low-energy building using the UCI data
set collected in 2017. The dataset for this project is available on Canvas. You should note, the target variable
is the total energy use.
REQUIREMENTS
• You must investigate, at a minimum, two completely separate types of supervised machine learning algorithms, of which at least one method must be a non neural network based algorithm.
• You are required to ully train your own algorithms. You may not use pre-trained systems.
• Unlike other projects, this data set contains time-series information. To improve your model you may find
it necessary to construct additional features to represent the time series information.
• You may use feature selection and ensemble learning method, however, these do not count towards the
above requirement.
INDEPENDENT EVALUATION
• As you don't have access a low-energy house, the independent evaluation of this project is different to the
other projects. A fundamental feature of the scientific process is reproducing existing work, and comparing
new results against exist work.
• The original data set was published publicly as part of the following paper:
Luis M. Candanedo, Veronique Feldheim, Dominique Deramaix, Data driven prediction models
o energy use o appliances in a low-energy house, Energy and Buildings, Volume 140, 1 April 2017,
Pages 81-97, ISSN 0378-7788
Since publication, a number of papers, and online resources, have been published that use this data set.
• Your independent evaluation is to research a number of these published works. Then you must evaluate,
compare and contrast your results to these other works.
• As part of your evaluation, you should discuss challenges you face in your independent comparison.
Project 3: Learning to Switch Traffic Lights
This project is to use reinforcement learning to improve the throughput of traffic at an intersection with traffic
lights. Most traffic lights operate on a predetermined phasing (that is, a timing when the lights change), which
does not take into account the number of vehicles that are waiting at, or approaching, the intersection. This
leads to a sub-optimal throughput of vehicles. For example, some vehicles may be waiting at a red light while
the intersection is otherwise clear. A more intelligent policy, can increase the throughput.
This project is to use reinforcement learning to find a policy to control the switching of traffic lights at a
simulated intersection, to improve the traffic throughput. You will need to demonstrate and compare your
6
RMIT Classification: Trusted
learnt policy to a more traditional static phasing policy.
REQUIREMENTS
• This project will require you to build a simple simulation to conduct the reinforcement learning. It is
advisable to build a simulation system that has configurable parameters for the flow of traffic.
• The simulated intersection must include at least one right turning lane. An example intersection would
be the intersection at Victoria Street - Lygon Street.
• Due to above work, you only need to use a single form of reinforcement learning to train an optimal policy.
• However, you will need to also test the throughput of a static policy, and devise a suitable static policy
that might be employed at a real-world intersection.
INDEPENDENT EVALUATION
• For your independent evaluation you must model a real-world intersection.
• You will need to observe the intersection and collect basic information, such as the average throughput,
and the rate at which traffic arrives at the intersection.
• You will need to approximate the static control policy of the lights.
• You will need to simulate both the static control policy, and learn a suitable policy for this intersection.
• We suggest choosing a simple intersection, and observing it over a short window of around 10-15 minutes.
You may also wish to simplify the problem by excluding right-turn signals.
• In case the social distancing measures do not allow you to observe a simple intersection, some statistics
or a test intersection will be provided to you.
You may propose and negotiate a project and machine learning problem to investigate, with the course coordinator. This project must meet a number of constraints:
• The project must be of a suitable complexity and challenge that is similar to the suggested projects. As
part of the negotiation, the scope and deliverables of the project will be set.
• If the project is using an existing data set, the problem should be phrased in a manner that can be solved
by multiple machine learning methods, of which at least two methods will be investigated.
• If the project requires a data set to be generated, devised, or collected, this collection should require
sufficient effort. In these cases, especially for reinforcement learning tasks, only one machine learning
method may need to be investigated.
• The proposed project must be independent of previously or concurrently assessed work.
In general, negotiations will take place via email, during consultation hours, or by appointment. Please note,
that the course co-ordinator is not available outside of business hours.
All negotiated projects must be finalised by no later than 5pm, Friday 9 May . This is the absolute deadline.
If you wish to conduct a negotiated project, begin the negotiation process early . A negotiated project may
be denied be ore the deadline if there is insufficient time for the negotiation process.
Most questions should be asked onCanvas, however, please do not post any code. There is aFAQ, and anything
in the FAQ will override what is specified in this specifications, if there is ambiguity.
Your lecturer is happy to discuss questions and your results with you. Please feel free to come talk to us during
consultation, or even a quick question, during lecture break.
You may not conduct a project if you have already been assessed on the work, or are concurrently being
assessed on the work.
7
RMIT Classification: Trusted
A detailed rubric is attached on canvas. In summary:
• Approach 60%;
• Ultimate Judgment & Analysis (Independent Evaluation) 20%;
• Report Presentation 20%.
You must submit all the relevant material as listed below via Canvas.
1. A report (of no more than 5 pages, plus up to 2 pages for appendices) critically analysing your approach
and ultimate judgement
2. An independent evaluation of your model and ultimate judgement (included in the report).
3. Your Python scripts, Jupyter notebooks, and software used to build your learning system and produce
the models and results.
The submission portal on canvas consists of two sub-pages. First page for report submission, the second page for
code submission. More information is provided on Canvas. Include only source code in a zip file containing your
name. We strongly recommend you to attach a README file with instructions on how to run your application.
Make sure that your assignment can run only with the code included in your zip file!
A penalty of 10% of the maximum mark per day (including weekends) will apply to late assignments up to a
maximum of five days. Assignments will not be marked after this time.

请加QQ：99515681  邮箱：99515681@qq.com   WX：codinghelp