Description
In this homework, you’ll write a MapReduce algorithm to analyze sample twitter dataset containing approximately 3.8 million tweets.
- Install Hadoop to your own server or use cs433.cse.unr.edu.
- You need to use jump host to access cs433.cse.unr.edu from outside of UNR campus. So, you can first login to nxlogin.engr.unr.edu and from there to cs433.cse.unr.edu
- Download ZIP file in here. Its size is around 405 MB. The files are already uploaded to HDFS in cs433.cse.unr.edu under “/” directory. Check by running “Hadoop dfs -ls /homework1/”
- Unzip the file and upload “training_set_tweets.txt” (tweets) and “training_set_users.txt” (users) files to HDFS
Once your Hadoop cluster is up and running do the following tasks:
- Show HDFS daemons (hint: search for processes called namenode, datanode) (5 pts)
- Show how many blocks created in HDFS for “tweets” file, either through command line or namenode web ui (5 pts)
- Show how many map tasks are created when you try to process “tweets” file in HDFS (10pts)
- Set the number of reduce tasks to 3 and show that Hadoop created 3 reduce tasks (10 pts)
- Write a MapReduce code to count the number of hash tags occurrences and find the most repeated 10 hashtags. (20 pts)
- Write a MapReduce code find the most tweeted 10 days. (Tweets are associated with time stamps so you need to count all the tweets posted in same days) (20 pts)
- Write a MapReduce code to find the most tweeted 10 cities along with the number of tweets (“training_set_users.txt” file has user_id city relation to extract city information) (30 pts)
Important Notes
- It is NOT allowed to use global variables in Q5 and Q6 as they are easy to implement with single MR job.
- Although it is not an ideal solution, you can use a global variable in Q7 to keep the solution simple. However, I offer 10pt bonus points if you implement without using a global variable. You’ll need to write multiple jobs in one application and use reduce-side join to implement this way.
What to deliver
Create following files/folders and compress them in a single zip file with name <LASTNAME>_<NAME>_HW1.zip and submit on WebCampus
- Take screenshots for Question 1-4 to a file answers1-4.pdf
- Copy the most repeated 30 hashtags along with number of occurrences to a file called “popular_tweets.txt” file
- Copy the most tweeted 20 days along with number of tweets to a file called “most_tweeted_days.txt” file
- Copy the most tweeted 10 cities along with number of tweets to a file called “most_tweeted_citites.txt” file
- Create three directories Q5, Q6, and Q7 and copy your source code for question 5, 6, and 7 into those directories.
- [Important] Create README file that shows how to run compile and run your code
- [Important] Do not include input files in your final submission
Statement on Academic Dishonesty (from syllabus):
“Cheating, plagiarism or otherwise obtaining grades under false pretenses constitute academic dishonesty according to the code of this university. Academic dishonesty will not be tolerated and penalties can include filing a final grade of “F”; reducing the student’s final course grade one or two full grade points; awarding a failing mark on the coursework in question; or requiring the student to retake or resubmit the coursework. For more details, see the University of Nevada, Reno General Catalog.”
