OMP9727: Recommender Systems

Assignment: Content-Based Movie Recommendation

Due Date:Week 4, Friday, June 21, 5:00 p.m.

Value:30%

This assignment is inspired by a typical application of recommender systems. The task is to

build a content-based “movie recommender” such as might be used by a streaming service (such

as Netflix) or review site (such as IMDb) to give users a personalizedlist of movies that match

their interests. The main learning objective for the assignment is togive a concrete example of

the issues that must be faced when building and evaluating a recommender system in a realistic

context. Note that, while movie recommender systems commonly make use of user ratings, our

scenario is not unrealistic as often all that a movie recommender system has are basic summaries

of the movies and the watch histories of the users.

For this assignment, you will be given a collection of 2000 movies that have been labelled as one

of 8 main genres (topics):animation,comedy,drama,family,horror,romance,sci-fiandthriller.

The movies of each genre are in a separate.tsvfile named for the genre (such asanimation.tsv)

with 7 fields:title,year,genre,director,cast,summaryandcountry.

The assignment is in three parts, corresponding to the components of a content-based recommender

system. The focus throughout is onexplanationof choices andevaluationof the various methods

and models, which involves choosing and justifying appropriate metrics. The whole assignment

will be prepared (and submitted) as a Jupyter notebook, similar to those being used in tutorials,

that contains a mixture of running code and tutorial-style explanation.

Part 1 of the assignment is to examine various supervised machine learning methods using a variety

of features and settings to determine what methods work best for topic (genre) classification in

this domain/dataset. For this purpose, simply concatenate all theinformation for one movie into

a single “document”. You will use Bernoulli Naive Bayes from the tutorial, Multinomial Naive

Bayes from the lecture, and one other machine learning method of your choice from scikit-learn

or another machine learning library, and NLTK for auxiliary functionsif needed.

Part 2 of the assignment is to test a potential recommender system that uses the method for

topic classification chosen in Part 1 by “simulating” a recommender system with a variety of

hypothetical users. This involves evaluating a number of techniques for “matching” user profiles

with movies using the similarity measures mentioned in the lecture. As we do not have real users,

for this part of the assignment, we will simply “invent” some (hopefully typical) users and evaluate

how well the recommender system would work for them, using appropriate metrics. Again you

will need to justify the choice of these metrics and explain how you arrived at your conclusions.

Part 3 of the assignment is to run a very small “user study” which means here findingoneperson,

preferably not someone in the class, to try out your recommendation method and give some

informal comments on the performance of your system from the user point of view. This does

not require any user interface to be built, the user can simply be shown the output (or use) the

Jupyter notebook from Parts 1 and 2. However, you will have to decide how many movies to show

the user at any one time, and how to get feedback from them on which movies they would click on

and which movies match their interests. A simple “talk aloud” protocol is a good idea here (this

is where you ask the user to use your system and say out loud what they are thinking/doing at

the same time – however please do not record the user’s voice – for that we need ethics approval).

Note that standard UNSW late penalties apply.

Assignment

Below are a series of questions to guide you through this assignment. Your answer to each question

should be in a separate clearly labelled section of the Jupyter notebook you submit. Each answer

should contain a mixture of explanation and code. Use comments in the code to explain any code

that you think readers will find unclear. The “readers” here are students similar to yourselves

who know something about machine learning and text classification but who may not be familiar

with the details of the methods.

Part 1. Topic (Genre) Classification

1. (2 marks) There are a few simplifications in the Jupyter notebookin the tutorial: (i) the regex

might remove too many special characters, and (ii) the evaluation isbased on only one training-

test split rather than using cross-validation. Explain how you are going to fix these mistakes and

then highlight any changes to the code in the answers to the next questions.

2. (2 marks) Develop a Multinomial Naive Bayes (MNB) model similar to the Bernoulli Naive

Bayes (BNB) model. Now consider all the steps in text preprocessing used prior to classification

with both BNB and MNB. The aim here is to find preprocessing steps that maximize overall ac-

curacy (under the default settings of the classifiers and usingCountVectorizerwith the standard

settings). Consider the special characters to be removed (and how and when they are removed),

the definition of a “word”, the stopword list (from either NLTK or scikit-learn), lowercasing and

stemming/lemmatization. Summarize the preprocessing steps thatyou think work “best” overall

and do not change this for the rest of the assignment.

3. (2 marks) Compare BNB and MNB models by evaluating them using the full dataset with

cross-validation. Choose appropriate metrics from those in the lecture that focus on the overall

accuracy of classification (i.e. not top-N metrics). Briefly discuss the tradeoffs between the various

metrics and then justify your choice of the main metrics for evaluation, taking into account whether

this dataset is balanced or imbalanced. On this basis, conclude whether either of BNB or MNB is

superior. Justify this conclusion with plots/tables.

4. (2 marks) Consider varying the number of features (words) used by BNB and MNB in the

classification, using thesklearnsetting which limits the number to the top N most frequent

words in the Vectorizer. Compare classification results for variousvalues for N and justify, based

on experimental results, one value for N that works well overall and use this value for the rest

of the assignment. Show plots or tables that support your decision. The emphasis is on clear

presentation of the results so do not print out large tables or too many tables that are difficult to

understand.

5. (5 marks) Choose one other machine learning method, perhaps one mentioned in the lecture.

Summarize this method in a single tutorial-style paragraph and explainwhy you think it is suitable

for topic classification for this dataset (for example, maybe otherpeople have used this method

for a similar problem). Use the implementation of this method from a standard machine learning

library such assklearn(notother people’s code from the Internet) to implement this method on

the news dataset using the same text preprocessing as for BNB and MNB. If the method has any

hyperparameters for tuning, explain how you will select those settings (or use the default settings),

and present a concrete hypothesis for how this method will compare to BNB and MNB.

Conduct experiments (and show the code for these experiments)using cross-validation and com-

ment on whether you confirmed (or not) your hypothesis. Finally, compare this method to BNB

and MNB on the metrics you used in Step 3 and choose one overall “best” method and settings

for topic classification.

Part 2. Recommendation Methods

1. (6 marks) The aim is to use the information retrieval algorithms for “matching” user profiles

to “documents” described in the lecture as a recommendation method. The overall idea is that

the classifier from Part 1 will assign a new movie to one of the 8 genres, and this movie will be

recommended to the user if the tf-idf vector for the movie is similar to the tf-idf vector for the

profile of the user in the predicted genre. The user profile for eachgenre will consist of the words,

or top M words, representing the interests of the user in that genre, computed as a tf-idf vector

across all movies predicted in that genre of interest to the user.

To get started, assume there is “training data” for the user profiles and “test data” for the

recommender defined as follows. There are 250 movies in each file. Suppose that the order in the

file is the time ordering of the movies, and suppose these movies camefrom a series of weeks, with

50 movies from each week. Assume Weeks 1–3 (movies 1–150) form the training data and Week 4

(movies 151–200) are the test data. UseTfidfVectorizeron all documents in the training data

to create a tf-idf matrix that defines a vector for each document(movie) in the training set.

Use these tf-idf values to define auser profile, which consists of a vector for each of the 8 genres.

To do this, for each genre, combine the movies from the training setpredicted to be in that genre

that the user “likes” into one (larger) document, so there will be 8 documents, one for each genre,

and use the vectorizer defined above to define a tf-idf vector foreach such document (genre).

Unfortunately we do not have any real users for our recommender system (because it has not yet

been built!), but we want some idea of how well it would perform. We invent two hypothetical

users, and simulate their use of the system. We specify the interests of each user with a set of

keywords for each genre. These user profiles can be found in the filesuser1.tsvanduser2.tsv

where each line in the file is a genre and (followed by a tab) a list of keywords. All the words are

case insensitive.Important: Although we

know the pairing of the genres and keywords,

all the recommender system “knows” is what movies the user liked in each genre.

Develop user profiles for User 1 and User 2 from the simulated training data (notthe keywords

used to define their interests) by supposing they liked all the moviesfrom Weeks 1–3 that matched

their interests and were predicted to be in the right category, i.e. assume the true genre is not

known, but instead the topic classifier is used to predict the movie genre, and the movie is shown

to the user listed under that genre. Print the top 20 words in their profiles for each of the genres.

Comment if these words seem reasonable.

Define another hypothetical “user” (User 3) by choosing different keywords across a range of

genres (perhaps those that match your interests or those of someone you know), and print the

top 20 keywords in their profile for each of their topics of interest.Comment if these words seem

reasonable.

2. (6 marks) Suppose a user sees N recommended movies and “likes”some of them. Choose and

justify appropriate metrics to evaluate the performance of the recommendation method. Also

choose an appropriate value for N based on how you think the movieswill be presented. Pay

attention to the large variety of movies and the need to obtain useful feedback from the user (i.e.

they must likesomemovies shown to them).

Evaluate the performance of the recommendation method by testing how well the top N movies

that the recommender suggests for Week 4, based on the user profiles, match the interests of each

user. That is, assume that each user likes all and only those movies inthe top N recommendations

that matched their profile for the predicted (not true) genre (where N is your chosen value). State

clearly whether you are showing N movies in total or N movies per genre. As part of the analysis,

consider various values for M, the number of words in the user profile for each genre, compared to

using all words.

Show the metrics for some of the matching algorithms to see which performs better for Users 1,

2 and 3. Explain any differences between the users. On the basis of these results, choose one

algorithm for matching user profiles and movies and explain your decision.

Part 3. User Evaluation

1. (5 marks) Conduct a “user study” of a hypothetical recommender system based on the method

chosen in Part 2. Your evaluation in Part 2 will have included a choice ofthe number N of movies

to show the user at any one time. For simplicity, suppose the user uses your system once per

week. Simulate running the recommender system for 3 weeks and training the model at the end

of Week 3 using interaction data obtained from the user, and testing the recommendations that

would be provided to that user in Week 4.

Choose one friendly “subject” and ask them to view (successively over a period of 4 simulated

weeks) N movies chosen at random for each “week”, for Weeks 1, 2and 3, and then (after training

the model) the recommended movies from Week 4. The subject couldbe someone else from the

course, but preferably is someone without knowledge of recommendation algorithms who will give

useful and unbiased feedback.

To be more precise, the user is shown 3 randomly chosen batches ofN movies, one batch from

Week 1 (N movies from 1–50), one batch from Week 2 (N movies from 51–100), and one batch

from Week 3 (N movies from 101–150), and says which of these they“like”. This gives training

data from which you can then train a recommendation model using the method in Part 2. The

user is then shown a batch ofrecommendedmovies from Week 4 (N movies from 151–200) in rank

order, and metrics are calculated based on which ofthesemovies the user likes. Show all these

metrics in a suitable form (plots or tables).

Ask the subject to talk aloud but make sure you find out which moviesthey are interested in.

Calculate and show the various metrics for the Week 4 recommendedmovies that you would show

using the model developed in Part 2. Explain any differences betweenmetrics calculated in Part 2

and the metrics obtained from the real user. Finally, mention any general user feedback concerning

the quality of the recommendations.

Submission and Assessment

?Please include your name and zid at the start of the notebook.

?Submit your notebook files using the following command:

give cs9727 asst .ipynb

You can check that your submission has been received using the command:

9727 classrun -check asst

?Assessment criteria include the correctness and thoroughness of code and experimental anal-

ysis, clarity and succinctness of explanations, and presentation quality.

Plagiarism

Remember that ALL work submitted for this assignment must be your own work and no sharing

or copying of code or answers is allowed. You may discuss the assignment with other students but

must not collaborate on developing answers to the questions. You may use code from the Internet

only with suitable attribution of the source. You may not use ChatGPT or any similar software to

generate any part of your explanations, evaluations or code. Do not use public code repositories

on sites such as github or file sharing sites such as Google Drive to save any part of your work –

make sure your code repository or cloud storage is private and do not share any links. This also

applies after you have finished the course, as we do not want next year’s students accessing your

solution, and plagiarism penalties can still apply after the course hasfinished.

All submitted assignments will be run through plagiarism detection software to detect similarities

to other submissions, including from past years. You shouldcarefullyread the UNSW policy on

academic integrity and plagiarism (linked from the course web page),noting, in particular, that

collusion(working together on an assignment, or sharing parts of assignment solutions) is a form

of plagiarism.

Finally, do not use any contract cheating “academies” or online “tutoring” services. This counts

as serious misconduct with heavy penalties up to automatic failure ofthe course with 0 marks,

and expulsion from the university for repeat offenders.

请加QQ：99515681  邮箱：99515681@qq.com   WX：codinghelp