GooglePlay Web Crawler

What is Hadoop Ecosystem?

• The core compositions of Hadoop are HDFS, Yarn, and other engines and App, like Mapreduce, Tez, Nutch, Pig, Hive, Spark, etc.
• HDFS is composed of NameNode and DataNode for data storage.
• Yarn is composed of Resource Manager and node Manager for resource assignment.
• APPs like Pig, Hive are higher level language processor. They can conduct mapreduce job much easier.

How does web crawler work?

• Use a customized Nutch to crawl apps metadata in GooglePlay

• Inject seed to nutchDB
• Generate urls to crawl from nutchDB
• Fetch app meatadata from html pages
• parse extracted metadata and outlinks
• update nutchDB with new outlinks
• Pig Loadfunc transforms nutchDB to readable text file form
• Create table and manage data by Hive

Command Line

• git clone

git clone https://github.com/apache/nutch
git checkout release-1.12

• customize nutch

patch -p1 < /googleplaycrawler/googleplaycrawler.patch

• run googleplaycrawler on single nutch cluster

echo "https://play.google.com/store/apps/details?id=com.facebook.orca" > seed

hadoop fs -put seed 

hadoop jar build/apache-nutch-1.12.job org.apache.nutch.googleplay.GooglePlayCrawler seed -numFetchers 10

• check output

hadoop fs -text file:///xxxxxx/nutchdb/segments/xxxxx/parse_data/part-00000/data

• fix the skew data job

patch -p1 < fixskew.patch

• uplode seeds file and run web scrawler in AWS EMR

• Aws S3

register target/nutchdbloader-0.0.1-SNAPSHOT.jar
register /home/hadoop/hadoop-2.7.3/share/hadoop/tools/lib/hadoop-aws-2.7.3.jar register nutch-1.12.jar
loaded = load 's3n://test/nutchdb/segments/*/parse_data/part-*/data' using com.example.NutchParsedDataLoader();
filtered = filter loaded by $0 is not null;
store filtered into 'output';

text reults