MapReduce_PySpark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qXDmaFo2s7ux"
   },
   "source": [
    "# MapReduce Implementation in PySpark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GEBWY9KctRZ3"
   },
   "source": [
    "In this work, we will perform basic data processing task using PySpark."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X87F2RAikK7G"
   },
   "source": [
    "The **Amazon_Responded_Oct05.csv** contatins information of 400K tweets. The following 3 columns will be used for this implementation:\n",
    "\n",
    "\n",
    "1.   **user_id_str**: user ID\n",
    "2.   **user_followers_count**: the number of followers\n",
    "3.   **text_**: the text of tweets\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JJDoSYA5lcNG"
   },
   "source": [
    "**Tasks:** \n",
    "1. Find out popular users whose followers are more than 5000, and \n",
    "2. Get Top 10 most popular words from the tweets posted by these popular users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ilB0y7C6l3OI"
   },
   "source": [
    "Specifically, we need to do the following steps:\n",
    "\n",
    "\n",
    "\n",
    "1.   **Read/load data**\n",
    "2.   **Extract the columns** (user_id_str and user_followers_count and text_) \n",
    "3.   **Remove the duplicated user id**: some users have different number of followers in different rows. In this case, we will just keep the maximum number of followers for a particular user.\n",
    "\n",
    "4.   **Find popular users**: create a filter to find popular users who have more than 5000 followers using the new pairs in step 3.\n",
    "\n",
    "5.   **Count words frequency**: count words frequency of of the tweets posted by the popular users we get from step 4, and get the Top 10 most popular words and their words frequency."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bDPzJW4N_8tJ"
   },
   "source": [
    "# Installing Spark on Google Colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "-h4AR98zkJ-M",
    "outputId": "04512ab2-ce22-464d-b9a1-35051173d2b3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample_data\n"
     ]
    }
   ],
   "source": [
    "# Checking folder\n",
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OpJvmZNJs6Vt"
   },
   "outputs": [],
   "source": [
    "# Installing JDK\n",
    "!apt-get install openjdk-8-jdk-headless -qq > /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "duE_alkZs65_"
   },
   "outputs": [],
   "source": [
    "# Getting Spark installer (check the path on spark.apache.org)\n",
    "!wget -q http://apache.mirrors.pair.com/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "C_beCouFs7VG",
    "outputId": "392d3fab-316f-4d8f-e794-481ebe34b712"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample_data  spark-2.4.4-bin-hadoop2.7.tgz\n"
     ]
    }
   ],
   "source": [
    "# Checking if the file is copied\n",
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "n7qwYpF_s7xl",
    "outputId": "1e3e93ed-a8a6-4d66-b01c-bc3fd46f8ad4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spark-2.4.4-bin-hadoop2.7/\n",
      "spark-2.4.4-bin-hadoop2.7/R/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/sparkr.zip\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/INDEX\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/html/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/html/R.css\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/html/00Index.html\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/help/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/help/aliases.rds\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/help/AnIndex\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/help/SparkR.rdx\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/help/SparkR.rdb\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/help/paths.rds\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/worker/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/worker/worker.R\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/worker/daemon.R\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/tests/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/tests/testthat/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/tests/testthat/test_basic.R\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/profile/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/profile/shell.R\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/profile/general.R\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/R/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/R/SparkR.rdx\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/R/SparkR.rdb\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/R/SparkR\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/Meta/\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/Meta/nsInfo.rds\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/Meta/links.rds\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/Meta/hsearch.rds\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/Meta/Rd.rds\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/Meta/features.rds\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/Meta/package.rds\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/DESCRIPTION\n",
      "spark-2.4.4-bin-hadoop2.7/R/lib/SparkR/NAMESPACE\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-shuffle-service.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-thriftserver.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-slave.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-shuffle-service.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-mesos-shuffle-service.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-master.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-history-server.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/spark-config.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-thriftserver.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-slaves.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-slave.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-mesos-shuffle-service.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-mesos-dispatcher.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-master.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-history-server.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/stop-all.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-slaves.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-mesos-dispatcher.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/start-all.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/spark-daemons.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/spark-daemon.sh\n",
      "spark-2.4.4-bin-hadoop2.7/sbin/slaves.sh\n",
      "spark-2.4.4-bin-hadoop2.7/python/\n",
      "spark-2.4.4-bin-hadoop2.7/python/dist/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark.egg-info/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark.egg-info/SOURCES.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark.egg-info/dependency_links.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark.egg-info/top_level.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark.egg-info/PKG-INFO\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark.egg-info/requires.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/README.md\n",
      "spark-2.4.4-bin-hadoop2.7/python/MANIFEST.in\n",
      "spark-2.4.4-bin-hadoop2.7/python/setup.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/run-tests.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/run-tests-with-coverage\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/userlibrary.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/userlib-0.1.zip\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/text-test.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/streaming/text-test.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/people_array_utf16le.json\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/people_array.json\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/people1.json\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/people.json\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=9/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/.part-r-00004.gz.parquet.crc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/.part-r-00002.gz.parquet.crc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/month=9/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/.part-r-00008.gz.parquet.crc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/_metadata\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/_common_metadata\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/parquet_partitioned/_SUCCESS\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=1/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=1/c=1/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=1/c=1/part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=1/c=1/.part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc.crc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=0/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=0/c=0/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=0/c=0/part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/b=0/c=0/.part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc.crc\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/orc_partitioned/_SUCCESS\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/ages_newlines.csv\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/sql/ages.csv\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/hello/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/hello/sub_hello/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/hello/sub_hello/sub_hello.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/hello/hello.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_support/SimpleHTTPServer.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_coverage/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_coverage/sitecustomize.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_coverage/coverage_daemon.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_coverage/conf/\n",
      "spark-2.4.4-bin-hadoop2.7/python/test_coverage/conf/spark-defaults.conf\n",
      "spark-2.4.4-bin-hadoop2.7/python/setup.cfg\n",
      "spark-2.4.4-bin-hadoop2.7/python/run-tests\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/python/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/python/pyspark/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/python/pyspark/shell.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/shuffle.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/serializers.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/rdd.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/profiler.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/java_gateway.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/files.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/daemon.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/context.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/conf.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/cloudpickle.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/broadcast.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/accumulators.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/worker.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/version.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/util.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/tests.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/test_serializers.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/test_broadcast.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/taskcontext.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/storagelevel.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/traceback_utils.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/tests.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/kinesis.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/kafka.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/flume.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/dstream.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/context.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/util.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/listener.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/streaming/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/status.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/statcounter.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/tests.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/streaming.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/session.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/readwriter.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/functions.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/dataframe.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/context.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/window.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/utils.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/udf.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/types.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/group.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/conf.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/column.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/catalog.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/sql/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/shell.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/resultiterable.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/rddsampler.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/util.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/tree.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/tests.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/regression.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/recommendation.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/random.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/fpm.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/feature.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/evaluation.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/clustering.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/classification.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/stat/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/stat/_statistics.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/stat/test.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/stat/distribution.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/stat/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/stat/KernelDensity.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/linalg/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/linalg/distributed.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/linalg/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/common.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/mllib/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/wrapper.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/util.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/tuning.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/tests.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/regression.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/recommendation.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/image.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/fpm.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/feature.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/evaluation.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/clustering.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/classification.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/stat.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/pipeline.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/param/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/param/shared.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/param/_shared_params_code_gen.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/param/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/linalg/\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/linalg/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/common.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/base.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/ml/__init__.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/join.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/heapq3.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/find_spark_home.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pyspark/_globals.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/pylintrc\n",
      "spark-2.4.4-bin-hadoop2.7/python/lib/\n",
      "spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip\n",
      "spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip\n",
      "spark-2.4.4-bin-hadoop2.7/python/lib/PY4J_LICENSE.txt\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/pyspark.streaming.rst\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/pyspark.sql.rst\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/epytext.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/conf.py\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/Makefile\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/pyspark.rst\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/pyspark.mllib.rst\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/pyspark.ml.rst\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/make2.bat\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/make.bat\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/index.rst\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/_templates/\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/_templates/layout.html\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/_static/\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/_static/pyspark.js\n",
      "spark-2.4.4-bin-hadoop2.7/python/docs/_static/pyspark.css\n",
      "spark-2.4.4-bin-hadoop2.7/python/.gitignore\n",
      "spark-2.4.4-bin-hadoop2.7/python/.coveragerc\n",
      "spark-2.4.4-bin-hadoop2.7/bin/\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-class\n",
      "spark-2.4.4-bin-hadoop2.7/bin/pyspark2.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/pyspark\n",
      "spark-2.4.4-bin-hadoop2.7/bin/load-spark-env.sh\n",
      "spark-2.4.4-bin-hadoop2.7/bin/load-spark-env.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/docker-image-tool.sh\n",
      "spark-2.4.4-bin-hadoop2.7/bin/sparkR2.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/sparkR.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/sparkR\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-submit2.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-submit.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-submit\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-sql2.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-sql.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-sql\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-shell2.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-shell.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-shell\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-class2.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/spark-class.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/run-example.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/run-example\n",
      "spark-2.4.4-bin-hadoop2.7/bin/pyspark.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/find-spark-home.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/find-spark-home\n",
      "spark-2.4.4-bin-hadoop2.7/bin/beeline.cmd\n",
      "spark-2.4.4-bin-hadoop2.7/bin/beeline\n",
      "spark-2.4.4-bin-hadoop2.7/README.md\n",
      "spark-2.4.4-bin-hadoop2.7/conf/\n",
      "spark-2.4.4-bin-hadoop2.7/conf/spark-env.sh.template\n",
      "spark-2.4.4-bin-hadoop2.7/conf/spark-defaults.conf.template\n",
      "spark-2.4.4-bin-hadoop2.7/conf/slaves.template\n",
      "spark-2.4.4-bin-hadoop2.7/conf/metrics.properties.template\n",
      "spark-2.4.4-bin-hadoop2.7/conf/log4j.properties.template\n",
      "spark-2.4.4-bin-hadoop2.7/conf/fairscheduler.xml.template\n",
      "spark-2.4.4-bin-hadoop2.7/conf/docker.properties.template\n",
      "spark-2.4.4-bin-hadoop2.7/data/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/gmm_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/als/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/als/test.data\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/als/sample_movielens_ratings.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/streaming_kmeans_data_test.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_svm_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_multiclass_classification_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_movielens_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_linear_regression_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_libsvm_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_lda_libsvm_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_lda_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_kmeans_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_isotonic_regression_libsvm_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_fpgrowth.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/sample_binary_classification_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/ridge-data/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/ridge-data/lpsa.data\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/pic_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/pagerank_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/kmeans_data.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/iris_libsvm.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-02/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-01/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA.png\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-02/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-02/54893.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-01/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-01/not-image.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/multi-channel/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/multi-channel/grayscale.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/multi-channel/chr30.4.184.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/multi-channel/BGRA_alpha_60.png\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/multi-channel/BGRA.png\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/license.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/kittens/\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/kittens/not-image.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/kittens/DP802813.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/kittens/DP153539.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/kittens/54893.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg\n",
      "spark-2.4.4-bin-hadoop2.7/data/mllib/images/license.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/graphx/\n",
      "spark-2.4.4-bin-hadoop2.7/data/graphx/users.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/graphx/followers.txt\n",
      "spark-2.4.4-bin-hadoop2.7/data/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/data/streaming/AFINN-111.txt\n",
      "spark-2.4.4-bin-hadoop2.7/NOTICE\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-jtransforms.html\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-json-formatter.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-jquery.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-join.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-jodd.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-jline.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-javolution.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-javassist.html\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-janino.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-heapq.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-graphlib-dot.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-f2j.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-datatables.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-dagre-d3.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-d3.min.js.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-cloudpickle.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-bootstrap.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-automaton.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-arpack.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-antlr.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-CC0.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-AnchorJS.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-zstd.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-zstd-jni.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-xmlenc.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-vis.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-spire.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-sorttable.js.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-slf4j.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-scopt.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-scala.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-sbt-launch-lib.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-respond.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-reflectasm.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-pyrolite.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-py4j.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-protobuf.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-pmml-model.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-paranamer.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-netlib.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-mustache.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-modernizr.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-minlog.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-matchMedia-polyfill.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-machinist.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-leveldbjni.txt\n",
      "spark-2.4.4-bin-hadoop2.7/licenses/LICENSE-kryo.txt\n",
      "spark-2.4.4-bin-hadoop2.7/LICENSE\n",
      "spark-2.4.4-bin-hadoop2.7/examples/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/ElementwiseProductExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/CosineSimilarity.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/ChiSqSelectorExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/BisectingKMeansExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/AssociationRulesExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/AbstractParams.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/Word2VecExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/TallSkinnySVD.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/TallSkinnyPCA.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/TFIDFExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SummaryStatisticsExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingTestExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingLogisticRegression.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingLinearRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingKMeansExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/StandardScalerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SimpleFPGrowth.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SampledRDDs.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SVMWithSGDExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/SVDExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RegressionMetricsExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RecommendationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RandomRDDGeneration.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PrefixSpanExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PowerIterationClusteringExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnSourceVectorExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnRowMatrixExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/PCAExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/NormalizerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/NaiveBayesExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/MultivariateSummarizer.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/MulticlassMetricsExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/MultiLabelMetricsExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LogisticRegressionWithLBFGSExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LinearRegressionWithSGDExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LatentDirichletAllocationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LBFGSExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/KernelDensityEstimationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/KMeansExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/IsotonicRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingKolmogorovSmirnovTestExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingClassificationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/GaussianMixtureExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/FPGrowthExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/RankingMetricsExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/MovieLensALS.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/LinearRegression.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/Word2VecExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/VectorSlicerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/VectorSizeHintExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/VectorIndexerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/VectorAssemblerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/UnaryTransformerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/TokenizerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/TfIdfExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/SummarizerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/StringIndexerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/StopWordsRemoverExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/StandardScalerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/SQLTransformerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestRegressorExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RFormulaExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/QuantileDiscretizerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PolynomialExpansionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PipelineExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PCAExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/OneVsRestExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/NormalizerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/NaiveBayesExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/NGramExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MultilayerPerceptronClassifierExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MulticlassLogisticRegressionWithElasticNetExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ModelSelectionViaTrainValidationSplitExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ModelSelectionViaCrossValidationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MinMaxScalerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MinHashLSHExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/MaxAbsScalerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionWithElasticNetExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionSummaryExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LinearSVCExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionWithElasticNetExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/LDAExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/KMeansExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/IsotonicRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/InteractionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/IndexToStringExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ImputerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeRegressorExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GeneralizedLinearRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GaussianMixtureExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/FeatureHasherExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/EstimatorTransformerParamExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ElementwiseProductExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DeveloperApiExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DataFrameExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DCTExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/CountVectorizerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/CorrelationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ChiSquareTestExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ChiSqSelectorExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/BucketizerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/BucketedRandomProjectionLSHExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/BinarizerExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/AFTSurvivalRegressionExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestClassifierExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/PrefixSpanExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/OneHotEncoderEstimatorExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeClassifierExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeClassificationExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ml/BisectingKMeansExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/TriangleCountingExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/SynthBenchmark.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/SSSPExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/PageRankExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/ConnectedComponentsExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/ComprehensiveExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/Analytics.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/graphx/AggregateMessagesExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkTC.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkRemoteFileTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkPi.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkPageRank.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkLR.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkKMeans.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkHdfsLR.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SparkALS.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SkewedGroupByTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/SimpleSkewedGroupByTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/MultiBroadcastTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalPi.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalLR.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalKMeans.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalFileLR.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LocalALS.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/GroupByTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/ExceptionHandlingTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/BroadcastTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/PageViewStream.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/PageViewGenerator.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/StreamingExamples.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/RecoverableNetworkWordCount.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/RawNetworkGrep.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/QueueStream.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/HdfsWordCount.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/streaming/CustomReceiver.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredSessionization.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredNetworkWordCountWindowed.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredNetworkWordCount.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKafkaWordCount.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/hive/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/hive/SparkHiveExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedUntypedAggregation.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedTypedAggregation.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/RDDRelation.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/SparkSQLExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/pythonconverters/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/pythonconverters/AvroConverters.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/LogQuery.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/HdfsTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/users.parquet\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/users.orc\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/users.avro\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/user.avsc\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/people.txt\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/people.json\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/people.csv\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/kv1.txt\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/full_user.avsc\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/resources/employees.json\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/streaming/structured_network_wordcount.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/svmLinear.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/survreg.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/randomForest.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/naiveBayes.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/mlp.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/ml.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/logit.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/lda.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/kstest.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/kmeans.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/isoreg.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/glm.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/gbt.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/gaussianMixture.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/fpm.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/decisionTree.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/bisectingKmeans.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/ml/als.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/dataframe.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/data-manipulation.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/r/RSparkSQLExample.R\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/transitive_closure.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/stateful_network_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/sql_network_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/queue_stream.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/network_wordjoinsentiments.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/network_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/hdfs_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/recoverable_network_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/kafka_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/flume_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/streaming/direct_kafka_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/status_api_demo.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/streaming/structured_network_wordcount_windowed.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/streaming/structured_network_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/streaming/structured_kafka_wordcount.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/basic.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/hive.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/datasource.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sql/arrow.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/sort.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/pi.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/parquet_inputformat.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/pagerank.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/word2vec_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/word2vec.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/tf_idf_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/svm_with_sgd_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/svd_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/summary_statistics_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/streaming_linear_regression_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/streaming_k_means_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/stratified_sampling_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/sampled_rdds.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/regression_metrics_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/recommendation_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/random_rdd_generation.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/random_forest_regression_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/random_forest_classification_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/power_iteration_clustering_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/pca_rowmatrix_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/normalizer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/naive_bayes_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/multi_label_metrics_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/logistic_regression.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/linear_regression_with_sgd_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/latent_dirichlet_allocation_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/kmeans.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/kernel_density_estimation_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/k_means_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/hypothesis_testing_kolmogorov_smirnov_test_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/hypothesis_testing_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/gradient_boosting_regression_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/gradient_boosting_classification_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/gaussian_mixture_model.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/gaussian_mixture_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/fpgrowth_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/elementwise_product_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/decision_tree_regression_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/decision_tree_classification_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/correlations_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/correlations.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/binary_classification_metrics_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/standard_scaler_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/ranking_metrics_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/multi_class_metrics_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/isotonic_regression_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/mllib/bisecting_k_means_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/word2vec_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/vector_slicer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/vector_size_hint_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/vector_indexer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/vector_assembler_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/train_validation_split.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/tokenizer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/tf_idf_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/summarizer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/string_indexer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/stopwords_remover_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/standard_scaler_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/sql_transformer.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/rformula_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/random_forest_regressor_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/random_forest_classifier_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/quantile_discretizer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/prefixspan_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/polynomial_expansion_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/pipeline_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/pca_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/one_vs_rest_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/normalizer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/naive_bayes_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/n_gram_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/multilayer_perceptron_classification.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/multiclass_logistic_regression_with_elastic_net.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/min_max_scaler_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/min_hash_lsh_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/max_abs_scaler_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/logistic_regression_with_elastic_net.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/logistic_regression_summary_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/linearsvc.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/linear_regression_with_elastic_net.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/lda_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/kmeans_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/isotonic_regression_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/index_to_string_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/imputer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/gradient_boosted_tree_regressor_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/gradient_boosted_tree_classifier_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/generalized_linear_regression_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/gaussian_mixture_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/fpgrowth_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/feature_hasher_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/estimator_transformer_param_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/elementwise_product_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/decision_tree_regression_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/decision_tree_classification_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/dct_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/dataframe_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/cross_validator.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/count_vectorizer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/correlation_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/chisq_selector_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/chi_square_test_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/bucketizer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/bucketed_random_projection_lsh_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/binarizer_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/als_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/aft_survival_regression.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/onehot_encoder_estimator_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/ml/bisecting_k_means_example.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/logistic_regression.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/kmeans.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/avro_inputformat.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/python/als.py\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaStatefulNetworkWordCount.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaSqlNetworkWordCount.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecord.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaQueueStream.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/streaming/JavaCustomReceiver.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCountWindowed.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCount.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredSessionization.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/hive/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/hive/JavaSparkHiveExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedUntypedAggregation.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedTypedAggregation.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQLExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaSummaryStatisticsExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaStreamingTestExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaSimpleFPGrowth.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaSVMWithSGDExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaSVDExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRegressionMetricsExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRecommendationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestRegressionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaPrefixSpanExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaPowerIterationClusteringExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaPCAExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaNaiveBayesExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaMulticlassClassificationMetricsExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaMultiLabelClassificationMetricsExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaLogisticRegressionWithLBFGSExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaLinearRegressionWithSGDExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaLatentDirichletAllocationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaLBFGSExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaKernelDensityEstimationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaKMeansExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaIsotonicRegressionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingKolmogorovSmirnovTestExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaElementwiseProductExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeRegressionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaChiSqSelectorExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaBisectingKMeansExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaBinaryClassificationMetricsExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaAssociationRulesExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaALS.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRankingMetricsExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestClassificationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/mllib/JavaCorrelationsExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaWord2VecExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorSlicerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorSizeHintExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorIndexerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorAssemblerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaTokenizerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaTfIdfExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaSummarizerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaStringIndexerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaStopWordsRemoverExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaStandardScalerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaSQLTransformerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaRFormulaExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPowerIterationClusteringExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialExpansionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPipelineExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPCAExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaOneVsRestExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaNormalizerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaNaiveBayesExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaNGramExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMultilayerPerceptronClassifierExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMulticlassLogisticRegressionWithElasticNetExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaTrainValidationSplitExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMinMaxScalerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMinHashLSHExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaMaxAbsScalerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionWithElasticNetExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionSummaryExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLinearSVCExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLinearRegressionWithElasticNetExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLabeledDocument.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaLDAExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaKMeansExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaIsotonicRegressionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaInteractionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaImputerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaGeneralizedLinearRegressionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaGaussianMixtureExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaFeatureHasherExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaFPGrowthExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaEstimatorTransformerParamExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaElementwiseProductExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaDocument.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeRegressionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaDCTExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaCountVectorizerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaCorrelationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaChiSquareTestExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaChiSqSelectorExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaAFTSurvivalRegressionExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestClassifierExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaPrefixSpanExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderEstimatorExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaBisectingKMeansExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/ml/JavaALSExample.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaWordCount.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaStatusTrackerDemo.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaSparkPi.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaPageRank.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaLogQuery.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaHdfsLR.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/src/main/java/org/apache/spark/examples/JavaTC.java\n",
      "spark-2.4.4-bin-hadoop2.7/examples/jars/\n",
      "spark-2.4.4-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/examples/jars/scopt_2.11-3.7.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/tests/\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/tests/pyfiles.py\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/tests/worker_memory_check.py\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/tests/py_container_checks.py\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/spark/\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/spark/entrypoint.sh\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/spark/Dockerfile\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/python/\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/python/Dockerfile\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/R/\n",
      "spark-2.4.4-bin-hadoop2.7/kubernetes/dockerfiles/spark/bindings/R/Dockerfile\n",
      "spark-2.4.4-bin-hadoop2.7/yarn/\n",
      "spark-2.4.4-bin-hadoop2.7/yarn/spark-2.4.4-yarn-shuffle.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-mapper-asl-1.9.13.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-jaxrs-1.9.13.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-dataformat-yaml-2.6.7.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-databind-2.6.7.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-core-asl-1.9.13.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-core-2.6.7.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-annotations-2.6.7.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/ivy-2.4.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/httpcore-4.4.10.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/httpclient-4.5.6.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/htrace-core-3.1.0-incubating.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hppc-0.7.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hk2-utils-2.4.0-b34.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hk2-locator-2.4.0-b34.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hk2-api-2.4.0-b34.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hive-metastore-1.2.1.spark2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hive-jdbc-1.2.1.spark2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hive-exec-1.2.1.spark2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hive-cli-1.2.1.spark2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hive-beeline-1.2.1.spark2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-yarn-server-web-proxy-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-yarn-server-common-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-yarn-common-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-yarn-client-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-yarn-api-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-mapreduce-client-shuffle-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-mapreduce-client-jobclient-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-mapreduce-client-core-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-mapreduce-client-common-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-mapreduce-client-app-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-hdfs-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-common-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-client-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-auth-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/hadoop-annotations-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/guice-servlet-3.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/guice-3.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/guava-14.0.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/gson-2.2.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/generex-1.0.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/flatbuffers-1.2.0-3f79e055.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/eigenbase-properties-1.1.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/derby-10.12.1.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/datanucleus-rdbms-3.2.9.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/datanucleus-core-3.2.10.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/datanucleus-api-jdo-3.2.6.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/curator-recipes-2.7.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/curator-framework-2.7.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/curator-client-2.7.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/core-1.1.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/compress-lzf-1.0.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-pool-1.5.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-net-3.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-math3-3.4.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-logging-1.1.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-lang3-3.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-lang-2.6.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-io-2.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-httpclient-3.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-digester-1.8.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-dbcp-1.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-crypto-1.0.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-configuration-1.6.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-compress-1.8.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-compiler-3.0.9.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-collections-3.2.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-codec-1.10.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-cli-1.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/commons-beanutils-1.9.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/chill_2.11-0.9.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/chill-java-0.9.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/calcite-linq4j-1.2.0-incubating.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/calcite-core-1.2.0-incubating.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/calcite-avatica-1.2.0-incubating.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/breeze_2.11-0.13.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/breeze-macros_2.11-0.13.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/bonecp-0.8.0.RELEASE.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/avro-mapred-1.8.2-hadoop2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/avro-ipc-1.8.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/avro-1.8.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/automaton-1.11-8.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/arrow-vector-0.10.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/arrow-memory-0.10.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/arrow-format-0.10.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/arpack_combined_all-0.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/api-util-1.0.0-M20.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/api-asn1-api-1.0.0-M20.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/apacheds-kerberos-codec-2.0.0-M15.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/apacheds-i18n-2.0.0-M15.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/apache-log4j-extras-1.2.17.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/aopalliance-repackaged-2.4.0-b34.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/aopalliance-1.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/antlr4-runtime-4.7.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/antlr-runtime-3.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/antlr-2.7.7.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/aircompressor-0.10.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/activation-1.1.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/ST4-4.0.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/RoaringBitmap-0.7.45.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/JavaEWAH-0.3.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/zstd-jni-1.3.2-2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/zookeeper-3.4.6.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/zjsonpatch-0.3.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/xz-1.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/xmlenc-0.52.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/xercesImpl-2.9.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/xbean-asm6-shaded-4.8.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/validation-api-1.1.0.Final.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/univocity-parsers-2.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/super-csv-2.2.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/stringtemplate-3.2.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/stream-2.7.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/stax-api-1.0.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/stax-api-1.0-2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spire_2.11-0.13.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spire-macros_2.11-0.13.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-yarn_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-unsafe_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-tags_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-tags_2.11-2.4.4-tests.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-streaming_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-sql_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-sketch_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-repl_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-network-shuffle_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-network-common_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-mllib_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-mllib-local_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-mesos_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-launcher_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-kvstore_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-kubernetes_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-hive_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-hive-thriftserver_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-graphx_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-core_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/spark-catalyst_2.11-2.4.4.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/snappy-java-1.1.7.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/snappy-0.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/snakeyaml-1.15.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/slf4j-api-1.7.16.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/shims-0.7.45.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/shapeless_2.11-2.3.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/scala-xml_2.11-1.0.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/scala-reflect-2.11.12.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/scala-parser-combinators_2.11-1.1.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/scala-library-2.11.12.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/scala-compiler-2.11.12.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/pyrolite-4.13.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/py4j-0.10.7.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/protobuf-java-2.5.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/parquet-jackson-1.10.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/parquet-hadoop-bundle-1.6.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/parquet-hadoop-1.10.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/parquet-format-2.4.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/parquet-encoding-1.10.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/parquet-common-1.10.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/parquet-column-1.10.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/paranamer-2.8.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/osgi-resource-locator-1.0.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/oro-2.0.8.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/orc-shims-1.5.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/orc-mapreduce-1.5.5-nohive.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/orc-core-1.5.5-nohive.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/opencsv-2.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/okio-1.13.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/okhttp-3.8.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/objenesis-2.5.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/netty-all-4.1.17.Final.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/netty-3.9.9.Final.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/minlog-1.3.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/metrics-jvm-3.1.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/metrics-json-3.1.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/metrics-graphite-3.1.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/metrics-core-3.1.5.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/mesos-1.4.0-shaded-protobuf.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/macro-compat_2.11-1.1.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/machinist_2.11-0.6.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/lz4-java-1.4.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/logging-interceptor-3.12.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/log4j-1.2.17.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/libthrift-0.9.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/libfb303-0.9.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/leveldbjni-all-1.8.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/kubernetes-model-common-4.1.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/kubernetes-model-4.1.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/kubernetes-client-4.1.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/kryo-shaded-4.0.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jul-to-slf4j-1.7.16.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jtransforms-2.4.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jta-1.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jsr305-1.3.9.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jsp-api-2.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/json4s-scalap_2.11-3.5.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/json4s-jackson_2.11-3.5.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/json4s-core_2.11-3.5.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/json4s-ast_2.11-3.5.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jpam-1.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jodd-core-3.5.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/joda-time-2.9.3.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jline-2.14.6.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jetty-util-6.1.26.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jetty-6.1.26.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jersey-server-2.22.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jersey-media-jaxb-2.22.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jersey-guava-2.22.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jersey-container-servlet-core-2.22.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jersey-container-servlet-2.22.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jersey-common-2.22.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jersey-client-2.22.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jdo-api-3.0.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jcl-over-slf4j-1.7.16.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jaxb-api-2.2.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/javolution-5.5.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/javax.ws.rs-api-2.0.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/javax.servlet-api-3.1.0.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/javax.inject-2.4.0-b34.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/javax.inject-1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/javax.annotation-api-1.2.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/javassist-3.18.1-GA.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/janino-3.0.9.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-xc-1.9.13.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-module-scala_2.11-2.6.7.1.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-module-paranamer-2.7.9.jar\n",
      "spark-2.4.4-bin-hadoop2.7/jars/jackson-module-jaxb-annotations-2.6.7.jar\n",
      "spark-2.4.4-bin-hadoop2.7/RELEASE\n"
     ]
    }
   ],
   "source": [
    "# Untar the Spark installer\n",
    "!tar -xvf spark-2.4.4-bin-hadoop2.7.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "LnvmACKatjUz",
    "outputId": "56337726-bb38-4d5e-a3ef-53c2fee7df4f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample_data  spark-2.4.4-bin-hadoop2.7\tspark-2.4.4-bin-hadoop2.7.tgz\n"
     ]
    }
   ],
   "source": [
    "# Checking the Spark folder after untar\n",
    "!ls "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wajhDzs4tlmy"
   },
   "outputs": [],
   "source": [
    "# Installing findspark - a python library to find Spark\n",
    "!pip install -q findspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zYr6_U6ztl5d"
   },
   "outputs": [],
   "source": [
    "# Setting environment variables: Setting Java and Spark home based on the location where they are stored\n",
    "import os\n",
    "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
    "os.environ[\"SPARK_HOME\"] = \"/content/spark-2.4.4-bin-hadoop2.7\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5aG5NJRXu-PD"
   },
   "source": [
    "# Step 1: Loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MU4d9fRYtmib"
   },
   "outputs": [],
   "source": [
    "# Creating a local Spark session\n",
    "import findspark\n",
    "findspark.init()\n",
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.master(\"local[*]\").getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "RQqcfrPudic8",
    "outputId": "6cf6895c-9c3b-49d6-f397-266a583b69d6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/gdrive\n"
     ]
    }
   ],
   "source": [
    "# Mounting google drive\n",
    "from google.colab import drive\n",
    "drive.mount('/content/gdrive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 435
    },
    "colab_type": "code",
    "id": "PmNG2OzJyLIo",
    "outputId": "59c40613-c157-4b2f-dd13-69bfce4a1ee3"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_str</th>\n",
       "      <th>tweet_created_at</th>\n",
       "      <th>user_screen_name</th>\n",
       "      <th>user_id_str</th>\n",
       "      <th>user_statuses_count</th>\n",
       "      <th>user_favourites_count</th>\n",
       "      <th>user_protected</th>\n",
       "      <th>user_listed_count</th>\n",
       "      <th>user_following</th>\n",
       "      <th>user_description</th>\n",
       "      <th>user_location</th>\n",
       "      <th>user_verified</th>\n",
       "      <th>user_followers_count</th>\n",
       "      <th>user_friends_count</th>\n",
       "      <th>user_created_at</th>\n",
       "      <th>tweet_language</th>\n",
       "      <th>text_</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>favorited</th>\n",
       "      <th>in_reply_to_screen_name</th>\n",
       "      <th>in_reply_to_status_id_str</th>\n",
       "      <th>in_reply_to_user_id_str</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>retweeted</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>'793270689780203520'</td>\n",
       "      <td>Tue Nov 01 01:57:25 +0000 2016</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>51287.0</td>\n",
       "      <td>4079.0</td>\n",
       "      <td>False</td>\n",
       "      <td>74.0</td>\n",
       "      <td>False</td>\n",
       "      <td>Content marketer; Polyglot; Beard aficionado; ...</td>\n",
       "      <td>غریب الوطن</td>\n",
       "      <td>False</td>\n",
       "      <td>1503.0</td>\n",
       "      <td>850.0</td>\n",
       "      <td>Thu May 13 17:43:52 +0000 2010</td>\n",
       "      <td>en</td>\n",
       "      <td>@AmazonHelp Can you please DM me? A product I ...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>'793281386912354304'</td>\n",
       "      <td>Tue Nov 01 02:39:55 +0000 2016</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>2225450.0</td>\n",
       "      <td>11366.0</td>\n",
       "      <td>False</td>\n",
       "      <td>796.0</td>\n",
       "      <td>False</td>\n",
       "      <td>We answer Amazon support questions 7 days a we...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>149569.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>Wed Oct 28 04:17:54 +0000 2009</td>\n",
       "      <td>en</td>\n",
       "      <td>@SeanEPanjab I'm sorry, we're unable to DM you...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>7.930000e+17</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>'793501578766319616'</td>\n",
       "      <td>Tue Nov 01 17:14:53 +0000 2016</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>51287.0</td>\n",
       "      <td>4079.0</td>\n",
       "      <td>False</td>\n",
       "      <td>74.0</td>\n",
       "      <td>False</td>\n",
       "      <td>Content marketer; Polyglot; Beard aficionado; ...</td>\n",
       "      <td>غریب الوطن</td>\n",
       "      <td>False</td>\n",
       "      <td>1503.0</td>\n",
       "      <td>850.0</td>\n",
       "      <td>Thu May 13 17:43:52 +0000 2010</td>\n",
       "      <td>en</td>\n",
       "      <td>@AmazonHelp It was purchased on https://t.co/g...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>7.930000e+17</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>@AmazonHelp It was purchased on https://t.co/g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>'793501657346682880'</td>\n",
       "      <td>Tue Nov 01 17:15:12 +0000 2016</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>51287.0</td>\n",
       "      <td>4079.0</td>\n",
       "      <td>False</td>\n",
       "      <td>74.0</td>\n",
       "      <td>False</td>\n",
       "      <td>Content marketer; Polyglot; Beard aficionado; ...</td>\n",
       "      <td>غریب الوطن</td>\n",
       "      <td>False</td>\n",
       "      <td>1503.0</td>\n",
       "      <td>850.0</td>\n",
       "      <td>Thu May 13 17:43:52 +0000 2010</td>\n",
       "      <td>en</td>\n",
       "      <td>@AmazonHelp I am following you now, if it help...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>7.930000e+17</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id_str  ...                                               text\n",
       "0                   NaN  ...                                                NaN\n",
       "1  '793270689780203520'  ...                                                NaN\n",
       "2  '793281386912354304'  ...                                                NaN\n",
       "3  '793501578766319616'  ...  @AmazonHelp It was purchased on https://t.co/g...\n",
       "4  '793501657346682880'  ...                                                NaN\n",
       "\n",
       "[5 rows x 25 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Loading data in pandas dataframe\n",
    "import pandas as pd\n",
    "amazon_df = pd.read_csv(\"/content/gdrive/Amazon_Responded_Oct05.csv\") \n",
    "amazon_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "nxZBPqrrEoFb",
    "outputId": "2437256c-d524-493b-a9a1-23d40de6ce3b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(462029, 25)"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "3aiETYNLGM-A",
    "outputId": "116227b5-93ef-4324-caa3-c633725e59f8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(378134, 25)"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dropping all null rows\n",
    "amazon_df.dropna(how=\"all\", inplace=True)\n",
    "amazon_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 487
    },
    "colab_type": "code",
    "id": "sO9nH9162q7s",
    "outputId": "4fac5ca7-3b8e-48b5-9f29-c07f064a5adb"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_str</th>\n",
       "      <th>tweet_created_at</th>\n",
       "      <th>user_screen_name</th>\n",
       "      <th>user_id_str</th>\n",
       "      <th>user_statuses_count</th>\n",
       "      <th>user_favourites_count</th>\n",
       "      <th>user_protected</th>\n",
       "      <th>user_listed_count</th>\n",
       "      <th>user_following</th>\n",
       "      <th>user_description</th>\n",
       "      <th>user_location</th>\n",
       "      <th>user_verified</th>\n",
       "      <th>user_followers_count</th>\n",
       "      <th>user_friends_count</th>\n",
       "      <th>user_created_at</th>\n",
       "      <th>tweet_language</th>\n",
       "      <th>text_</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>favorited</th>\n",
       "      <th>in_reply_to_screen_name</th>\n",
       "      <th>in_reply_to_status_id_str</th>\n",
       "      <th>in_reply_to_user_id_str</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>retweeted</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>'793270689780203520'</td>\n",
       "      <td>Tue Nov 01 01:57:25 +0000 2016</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>51287.0</td>\n",
       "      <td>4079.0</td>\n",
       "      <td>False</td>\n",
       "      <td>74.0</td>\n",
       "      <td>False</td>\n",
       "      <td>Content marketer; Polyglot; Beard aficionado; ...</td>\n",
       "      <td>غریب الوطن</td>\n",
       "      <td>False</td>\n",
       "      <td>1503.0</td>\n",
       "      <td>850.0</td>\n",
       "      <td>Thu May 13 17:43:52 +0000 2010</td>\n",
       "      <td>en</td>\n",
       "      <td>@AmazonHelp Can you please DM me? A product I ...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>NaN</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>'793281386912354304'</td>\n",
       "      <td>Tue Nov 01 02:39:55 +0000 2016</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>2225450.0</td>\n",
       "      <td>11366.0</td>\n",
       "      <td>False</td>\n",
       "      <td>796.0</td>\n",
       "      <td>False</td>\n",
       "      <td>We answer Amazon support questions 7 days a we...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>149569.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>Wed Oct 28 04:17:54 +0000 2009</td>\n",
       "      <td>en</td>\n",
       "      <td>@SeanEPanjab I'm sorry, we're unable to DM you...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>7.930000e+17</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>'793501578766319616'</td>\n",
       "      <td>Tue Nov 01 17:14:53 +0000 2016</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>51287.0</td>\n",
       "      <td>4079.0</td>\n",
       "      <td>False</td>\n",
       "      <td>74.0</td>\n",
       "      <td>False</td>\n",
       "      <td>Content marketer; Polyglot; Beard aficionado; ...</td>\n",
       "      <td>غریب الوطن</td>\n",
       "      <td>False</td>\n",
       "      <td>1503.0</td>\n",
       "      <td>850.0</td>\n",
       "      <td>Thu May 13 17:43:52 +0000 2010</td>\n",
       "      <td>en</td>\n",
       "      <td>@AmazonHelp It was purchased on https://t.co/g...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>7.930000e+17</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>@AmazonHelp It was purchased on https://t.co/g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>'793501657346682880'</td>\n",
       "      <td>Tue Nov 01 17:15:12 +0000 2016</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>51287.0</td>\n",
       "      <td>4079.0</td>\n",
       "      <td>False</td>\n",
       "      <td>74.0</td>\n",
       "      <td>False</td>\n",
       "      <td>Content marketer; Polyglot; Beard aficionado; ...</td>\n",
       "      <td>غریب الوطن</td>\n",
       "      <td>False</td>\n",
       "      <td>1503.0</td>\n",
       "      <td>850.0</td>\n",
       "      <td>Thu May 13 17:43:52 +0000 2010</td>\n",
       "      <td>en</td>\n",
       "      <td>@AmazonHelp I am following you now, if it help...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>7.930000e+17</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>'793502854459879424'</td>\n",
       "      <td>Tue Nov 01 17:19:57 +0000 2016</td>\n",
       "      <td>AmazonHelp</td>\n",
       "      <td>85741735.0</td>\n",
       "      <td>2225450.0</td>\n",
       "      <td>11366.0</td>\n",
       "      <td>False</td>\n",
       "      <td>796.0</td>\n",
       "      <td>False</td>\n",
       "      <td>We answer Amazon support questions 7 days a we...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>149569.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>Wed Oct 28 04:17:54 +0000 2009</td>\n",
       "      <td>en</td>\n",
       "      <td>@SeanEPanjab Please give us a call/chat so we ...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>SeanEPanjab</td>\n",
       "      <td>7.940000e+17</td>\n",
       "      <td>143515471.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "      <td>@SeanEPanjab Please give us a call/chat so we ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id_str  ...                                               text\n",
       "1  '793270689780203520'  ...                                                NaN\n",
       "2  '793281386912354304'  ...                                                NaN\n",
       "3  '793501578766319616'  ...  @AmazonHelp It was purchased on https://t.co/g...\n",
       "4  '793501657346682880'  ...                                                NaN\n",
       "5  '793502854459879424'  ...  @SeanEPanjab Please give us a call/chat so we ...\n",
       "\n",
       "[5 rows x 25 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Replacing carriage return and new line characters with a space\n",
    "amazon_df = amazon_df.replace({r'\\r\\n': ' '}, regex=True)\n",
    "amazon_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 333
    },
    "colab_type": "code",
    "id": "ZOPaeQoc-quq",
    "outputId": "d98e431c-e652-4ec4-dd9a-d098fe6de779"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+------------------------------+----------------+-----------+-------------------+---------------------+--------------+-----------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------------+--------------------+------------------+------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------+--------------+---------+-----------------------+-------------------------+-----------------------+-------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|id_str              |tweet_created_at              |user_screen_name|user_id_str|user_statuses_count|user_favourites_count|user_protected|user_listed_count|user_following|user_description                                                                                                                              |user_location|user_verified|user_followers_count|user_friends_count|user_created_at               |tweet_language|text_                                                                                                                                     |favorite_count|favorited|in_reply_to_screen_name|in_reply_to_status_id_str|in_reply_to_user_id_str|retweet_count|retweeted|text                                                                                                                                     |\n",
      "+--------------------+------------------------------+----------------+-----------+-------------------+---------------------+--------------+-----------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------------+--------------------+------------------+------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------+--------------+---------+-----------------------+-------------------------+-----------------------+-------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|'793270689780203520'|Tue Nov 01 01:57:25 +0000 2016|SeanEPanjab     |143515471.0|51287.0            |4079.0               |False         |74.0             |False         |Content marketer; Polyglot; Beard aficionado; Sikh. Persian, Catalan, French, Spanish. You'll find lol in the interstitial lulls of my tweets.|غریب الوطن   |False        |1503.0              |850.0             |Thu May 13 17:43:52 +0000 2010|en            |@AmazonHelp Can you please DM me? A product I ordered last year never arrived.                                                            |0.0           |False    |AmazonHelp             |nan                      |85741735.0             |0.0          |False    |nan                                                                                                                                      |\n",
      "|'793281386912354304'|Tue Nov 01 02:39:55 +0000 2016|AmazonHelp      |85741735.0 |2225450.0          |11366.0              |False         |796.0            |False         |We answer Amazon support questions 7 days a week. Support available in English / Deutsch / Español / Português / Français / Italiano / 日本語 |nan          |True         |149569.0            |53.0              |Wed Oct 28 04:17:54 +0000 2009|en            |@SeanEPanjab I'm sorry, we're unable to DM you. Was this order purchased on https://t.co/nUUp5MLhYl, or one of our other sites? ^CL       |0.0           |False    |SeanEPanjab            |7.93e+17                 |143515471.0            |0.0          |False    |nan                                                                                                                                      |\n",
      "|'793501578766319616'|Tue Nov 01 17:14:53 +0000 2016|SeanEPanjab     |143515471.0|51287.0            |4079.0               |False         |74.0             |False         |Content marketer; Polyglot; Beard aficionado; Sikh. Persian, Catalan, French, Spanish. You'll find lol in the interstitial lulls of my tweets.|غریب الوطن   |False        |1503.0              |850.0             |Thu May 13 17:43:52 +0000 2010|en            |@AmazonHelp It was purchased on https://t.co/gPWnuSkbwC.                                                                                  |0.0           |False    |AmazonHelp             |7.93e+17                 |85741735.0             |0.0          |False    |@AmazonHelp It was purchased on https://t.co/gPWnuSkbwC.                                                                                 |\n",
      "|'793501657346682880'|Tue Nov 01 17:15:12 +0000 2016|SeanEPanjab     |143515471.0|51287.0            |4079.0               |False         |74.0             |False         |Content marketer; Polyglot; Beard aficionado; Sikh. Persian, Catalan, French, Spanish. You'll find lol in the interstitial lulls of my tweets.|غریب الوطن   |False        |1503.0              |850.0             |Thu May 13 17:43:52 +0000 2010|en            |@AmazonHelp I am following you now, if it helps to DM.                                                                                    |0.0           |False    |AmazonHelp             |7.93e+17                 |85741735.0             |0.0          |False    |nan                                                                                                                                      |\n",
      "|'793502854459879424'|Tue Nov 01 17:19:57 +0000 2016|AmazonHelp      |85741735.0 |2225450.0          |11366.0              |False         |796.0            |False         |We answer Amazon support questions 7 days a week. Support available in English / Deutsch / Español / Português / Français / Italiano / 日本語 |nan          |True         |149569.0            |53.0              |Wed Oct 28 04:17:54 +0000 2009|en            |@SeanEPanjab Please give us a call/chat so we can look into this order for you: https://t.co/hApLpMlfHN. ^HB                              |0.0           |False    |SeanEPanjab            |7.94e+17                 |143515471.0            |0.0          |False    |@SeanEPanjab Please give us a call/chat so we can look into this order for you: https://t.co/hApLpMlfHN. ^HB                             |\n",
      "|'793504235400884224'|Tue Nov 01 17:25:26 +0000 2016|AmazonHelp      |85741735.0 |2225450.0          |11366.0              |False         |796.0            |False         |We answer Amazon support questions 7 days a week. Support available in English / Deutsch / Español / Português / Français / Italiano / 日本語 |nan          |True         |149569.0            |53.0              |Wed Oct 28 04:17:54 +0000 2009|en            |@SeanEPanjab Without providing any sensitive information, do the order details indicate that the item was ever shipped? ^SW               |0.0           |False    |SeanEPanjab            |7.94e+17                 |143515471.0            |0.0          |False    |nan                                                                                                                                      |\n",
      "|'793511847899070465'|Tue Nov 01 17:55:41 +0000 2016|SeanEPanjab     |143515471.0|51287.0            |4079.0               |False         |74.0             |False         |Content marketer; Polyglot; Beard aficionado; Sikh. Persian, Catalan, French, Spanish. You'll find lol in the interstitial lulls of my tweets.|غریب الوطن   |False        |1503.0              |850.0             |Thu May 13 17:43:52 +0000 2010|en            |@AmazonHelp It was shipped and possibly delivered to my old place of employment. But when I asked for it, no one had it. And so I wonder +|0.0           |False    |AmazonHelp             |7.94e+17                 |85741735.0             |0.0          |False    |nan                                                                                                                                      |\n",
      "|'793511899279208449'|Tue Nov 01 17:55:54 +0000 2016|SeanEPanjab     |143515471.0|51287.0            |4079.0               |False         |74.0             |False         |Content marketer; Polyglot; Beard aficionado; Sikh. Persian, Catalan, French, Spanish. You'll find lol in the interstitial lulls of my tweets.|غریب الوطن   |False        |1503.0              |850.0             |Thu May 13 17:43:52 +0000 2010|en            |@AmazonHelp if it was lost. I have the order #.                                                                                           |0.0           |False    |AmazonHelp             |7.94e+17                 |85741735.0             |0.0          |False    |nan                                                                                                                                      |\n",
      "|'793513446633533440'|Tue Nov 01 18:02:03 +0000 2016|AmazonHelp      |85741735.0 |2225450.0          |11366.0              |False         |796.0            |False         |We answer Amazon support questions 7 days a week. Support available in English / Deutsch / Español / Português / Français / Italiano / 日本語 |nan          |True         |149569.0            |53.0              |Wed Oct 28 04:17:54 +0000 2009|en            |@SeanEPanjab I'm not able to access account info here. Please reach out by Phone/Chat so we can look at this: https://t.co/EKXRLsnxJu ^GL |0.0           |False    |SeanEPanjab            |7.94e+17                 |143515471.0            |0.0          |False    |@SeanEPanjab I'm not able to access account info here. Please reach out by Phone/Chat so we can look at this: https://t.co/EKXRLsnxJu ^GL|\n",
      "|'793299404975247360'|Tue Nov 01 03:51:31 +0000 2016|aakashwangnoo   |71457972.0 |592.0              |471.0                |False         |11.0             |False         |Don't try to be GOD in your lives...Let HIM work in your lives...and see the magic that he weaves...                                          |New Delhi    |False        |234.0               |57.0              |Fri Sep 04 04:55:01 +0000 2009|en            |@JeffBezos @amazonIN @AmazonHelp Tring...Tring...Tring Who's There? Your Suffering Customers from India Mr. Bezos...Get Up and Help!      |0.0           |False    |JeffBezos              |nan                      |15506669.0             |0.0          |False    |nan                                                                                                                                      |\n",
      "+--------------------+------------------------------+----------------+-----------+-------------------+---------------------+--------------+-----------------+--------------+----------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------------+--------------------+------------------+------------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------+--------------+---------+-----------------------+-------------------------+-----------------------+-------------+---------+-----------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Converting Pandas Dataframe into Spark Dataframe\n",
    "amazon_df = amazon_df.astype(str) # Converting pandas df to string first\n",
    "amazon_sdf = spark.createDataFrame(amazon_df)\n",
    "amazon_sdf.show(10, False) # False allows us to show entire content of the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 479
    },
    "colab_type": "code",
    "id": "QdBArdGqiFK7",
    "outputId": "06e8cde5-0268-47f8-b56d-2f102130e402"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['id_str',\n",
       " 'tweet_created_at',\n",
       " 'user_screen_name',\n",
       " 'user_id_str',\n",
       " 'user_statuses_count',\n",
       " 'user_favourites_count',\n",
       " 'user_protected',\n",
       " 'user_listed_count',\n",
       " 'user_following',\n",
       " 'user_description',\n",
       " 'user_location',\n",
       " 'user_verified',\n",
       " 'user_followers_count',\n",
       " 'user_friends_count',\n",
       " 'user_created_at',\n",
       " 'tweet_language',\n",
       " 'text_',\n",
       " 'favorite_count',\n",
       " 'favorited',\n",
       " 'in_reply_to_screen_name',\n",
       " 'in_reply_to_status_id_str',\n",
       " 'in_reply_to_user_id_str',\n",
       " 'retweet_count',\n",
       " 'retweeted',\n",
       " 'text']"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Columns in df\n",
    "amazon_sdf.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 516
    },
    "colab_type": "code",
    "id": "TF9WcoG9iFVO",
    "outputId": "e61e76fb-f431-4acd-8422-27a2797575d6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- id_str: string (nullable = true)\n",
      " |-- tweet_created_at: string (nullable = true)\n",
      " |-- user_screen_name: string (nullable = true)\n",
      " |-- user_id_str: string (nullable = true)\n",
      " |-- user_statuses_count: string (nullable = true)\n",
      " |-- user_favourites_count: string (nullable = true)\n",
      " |-- user_protected: string (nullable = true)\n",
      " |-- user_listed_count: string (nullable = true)\n",
      " |-- user_following: string (nullable = true)\n",
      " |-- user_description: string (nullable = true)\n",
      " |-- user_location: string (nullable = true)\n",
      " |-- user_verified: string (nullable = true)\n",
      " |-- user_followers_count: string (nullable = true)\n",
      " |-- user_friends_count: string (nullable = true)\n",
      " |-- user_created_at: string (nullable = true)\n",
      " |-- tweet_language: string (nullable = true)\n",
      " |-- text_: string (nullable = true)\n",
      " |-- favorite_count: string (nullable = true)\n",
      " |-- favorited: string (nullable = true)\n",
      " |-- in_reply_to_screen_name: string (nullable = true)\n",
      " |-- in_reply_to_status_id_str: string (nullable = true)\n",
      " |-- in_reply_to_user_id_str: string (nullable = true)\n",
      " |-- retweet_count: string (nullable = true)\n",
      " |-- retweeted: string (nullable = true)\n",
      " |-- text: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Schema: Datatypes associated with columns\n",
    "amazon_sdf.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "gAxJtm97jMxa",
    "outputId": "e1ae789b-b56c-42d4-d973-782cbfa8161c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "378134"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Total number of rows\n",
    "amazon_sdf.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TeB9Kq0IvM1u"
   },
   "source": [
    "# Step 2: Extracting the three columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 518
    },
    "colab_type": "code",
    "id": "Za5KDUXROYgG",
    "outputId": "02bfcfd8-dd55-45fd-8cb7-6d5d4b899113"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|user_id_str|user_followers_count|text_                                                                                                                                                        |\n",
      "+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|143515471.0|1503                |@AmazonHelp Can you please DM me? A product I ordered last year never arrived.                                                                               |\n",
      "|85741735.0 |149569              |@SeanEPanjab I'm sorry, we're unable to DM you. Was this order purchased on https://t.co/nUUp5MLhYl, or one of our other sites? ^CL                          |\n",
      "|143515471.0|1503                |@AmazonHelp It was purchased on https://t.co/gPWnuSkbwC.                                                                                                     |\n",
      "|143515471.0|1503                |@AmazonHelp I am following you now, if it helps to DM.                                                                                                       |\n",
      "|85741735.0 |149569              |@SeanEPanjab Please give us a call/chat so we can look into this order for you: https://t.co/hApLpMlfHN. ^HB                                                 |\n",
      "|85741735.0 |149569              |@SeanEPanjab Without providing any sensitive information, do the order details indicate that the item was ever shipped? ^SW                                  |\n",
      "|143515471.0|1503                |@AmazonHelp It was shipped and possibly delivered to my old place of employment. But when I asked for it, no one had it. And so I wonder +                   |\n",
      "|143515471.0|1503                |@AmazonHelp if it was lost. I have the order #.                                                                                                              |\n",
      "|85741735.0 |149569              |@SeanEPanjab I'm not able to access account info here. Please reach out by Phone/Chat so we can look at this: https://t.co/EKXRLsnxJu ^GL                    |\n",
      "|71457972.0 |234                 |@JeffBezos @amazonIN @AmazonHelp Tring...Tring...Tring Who's There? Your Suffering Customers from India Mr. Bezos...Get Up and Help!                         |\n",
      "|85741735.0 |149568              |@aakashwangnoo Hi there, kindly write back to us with the details requested so we can proceed further with this.  ^KJ                                        |\n",
      "|71457972.0 |234                 |@AmazonHelp How many times do you want to write back to you guys??? Check the complaints through email regarding  Order 171-1338898-5999507.                 |\n",
      "|85741735.0 |149568              |@aakashwangnoo Hi! We have responded to you here: https://t.co/v4YVCa3rff ^SG(1/2)                                                                           |\n",
      "|85741735.0 |149568              |@aakashwangnoo Please don’t provide your order details as we consider them to be personal info. Our page is visible to public. ^SG(2/2)                      |\n",
      "|71457972.0 |234                 |@AmazonHelp @amazonIN I am providing my order details as even after providing that ur CC Team is repeatedly shooting me cut copy paste email                 |\n",
      "|85741735.0 |149568              |@aakashwangnoo Hi, I believe we have already got back to you on this. Kindly check for the correspondence here: https://t.co/NTkrxpsbHJ ^AB                  |\n",
      "|71457972.0 |234                 |@AmazonHelp @amazonIN @JeffBezos @consumeraction @ConsumerAffairs @ConsumersUnion You want to silent my voice and call that as Answers to me                 |\n",
      "|71457972.0 |234                 |@AmazonHelp @amazonIN @JeffBezos @consumeraction @ConsumerAffairs @ConsumersUnion SHAME on you Team Amazon India...SHAME SHAME SHAME!!!                      |\n",
      "|71457972.0 |234                 |@AmazonHelp @amazonIN @JeffBezos @consumeraction @ConsumerAffairs @ConsumersUnion Please see attachments. These are Amazon responses. https://t.co/W9FcqQVtCC|\n",
      "|71457972.0 |234                 |@AmazonHelp @amazonIN @JeffBezos @consumeraction @ConsumerAffairs @ConsumersUnion and these are my responses. https://t.co/vTwhtX5oBt                        |\n",
      "+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Extracting columns 'user_id_str', 'user_followers_count', and 'text_'\n",
    "amazon_sub_df = amazon_sdf.select(amazon_sdf.user_id_str, amazon_sdf.user_followers_count.cast('int').alias('user_followers_count'), amazon_sdf.text_)\n",
    "amazon_sub_df.show(20, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 109
    },
    "colab_type": "code",
    "id": "wXij0qeRPNqT",
    "outputId": "a0fa369d-2912-4f3a-8d0d-2ba06a08a643"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- user_id_str: string (nullable = true)\n",
      " |-- user_followers_count: integer (nullable = true)\n",
      " |-- text_: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Checking datatype of columns\n",
    "amazon_sub_df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 128
    },
    "colab_type": "code",
    "id": "46zqZLHHjb7b",
    "outputId": "c3f6642a-568a-4f2d-e198-8dcab686db8d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------------+\n",
      "|count(DISTINCT user_id_str)|\n",
      "+---------------------------+\n",
      "|                      71417|\n",
      "+---------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Number of distinct users \n",
    "import pyspark.sql.functions as f\n",
    "amazon_sub_df.select(f.countDistinct(\"user_id_str\")).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MJvkKi1zvVMx"
   },
   "source": [
    "# Step 3: Removing the duplicated records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "papk5-yJvrjr",
    "outputId": "8129dd9e-4317-491a-c636-fd20258c0af2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "170691"
      ]
     },
     "execution_count": 22,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking number of rows (tweets) for a particular user\n",
    "amazon_sub_df.filter(amazon_sub_df.user_id_str == '85741735.0').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 498
    },
    "colab_type": "code",
    "id": "Ah872N49avJS",
    "outputId": "fcc265e9-e3f8-468c-d320-e5bba0382f83"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+-----+\n",
      "| user_id_str|  max|\n",
      "+------------+-----+\n",
      "| 182037498.0|   38|\n",
      "|  16669765.0|   57|\n",
      "| 264218502.0|   70|\n",
      "|1873374613.0|  174|\n",
      "|1305569118.0|  128|\n",
      "|3255474476.0|    1|\n",
      "|  29207516.0|  213|\n",
      "| 123058781.0|  380|\n",
      "|1079912580.0|  673|\n",
      "| 277017869.0| 1103|\n",
      "| 553031167.0|  491|\n",
      "|  43597889.0| 1915|\n",
      "|  45659327.0|  338|\n",
      "| 101381642.0|  602|\n",
      "|2811064347.0|11588|\n",
      "|2325986832.0|  141|\n",
      "| 998124780.0|  102|\n",
      "| 220146294.0|  497|\n",
      "| 7.11572e+17|  143|\n",
      "|1362528962.0|  567|\n",
      "+------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Removing duplicate records by keeping just the maximum number of followers for any user\n",
    "# Step 1: First let us get the max followers for every user\n",
    "import pyspark.sql.functions as f\n",
    "maxf = amazon_sub_df.groupBy(\"user_id_str\").agg(f.max(\"user_followers_count\").alias(\"max\")).alias(\"maxf\")\n",
    "maxf.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "d0ebpWb6imFL",
    "outputId": "8bb8749b-341a-4e8b-bc4f-f89fa9320fa8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "71417"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Total number of rows\n",
    "maxf.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SwMY2axhc3yw"
   },
   "outputs": [],
   "source": [
    "# Step 2: Let us now join this with the original df (amazon_sub_df) to get all the rows of users which match their max follower count obtained in 'maxf'\n",
    "from pyspark.sql.functions import col \n",
    "amazon_sub_df = amazon_sub_df.alias(\"amazon_sub_df\") # defining alias for original df\n",
    "amazon_sub_df2 = amazon_sub_df.join(maxf, (col(\"user_followers_count\") == col(\"max\")) & \n",
    "                                    (col(\"amazon_sub_df.user_id_str\") == col(\"maxf.user_id_str\"))).select(\n",
    "                                     col(\"amazon_sub_df.user_id_str\"), col(\"amazon_sub_df.user_followers_count\"), col(\"amazon_sub_df.text_\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 518
    },
    "colab_type": "code",
    "id": "Z9nDZ9kp5ySc",
    "outputId": "ef7fd93c-868e-4100-ea94-0ac9570d5791"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|user_id_str |user_followers_count|text_                                                                                                                                       |\n",
      "+------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|8.32319e+17 |0                   |@Amazon poor service by Amazon...my friend is waiting for the gift since a week still not delivered...really disappointed                   |\n",
      "|8.14921e+17 |1                   |@AmazonHelp My order says its been delivered since dec 30 &amp; I still haven't received it. I called CS &amp; they said to wait 3 + days...|\n",
      "|8.14921e+17 |1                   |@AmazonHelp I'm so confused because I also have amazon prime &amp; now its the 6 &amp; its still not here idk what to do..                  |\n",
      "|8.14921e+17 |1                   |@AmazonHelp I've looked everywhere.. https://t.co/XHCrrmCl9P                                                                                |\n",
      "|8.14921e+17 |1                   |@AmazonHelp I did the rep told me to wait 3 days just in case it was lost or delayed but its been past 3 days now                           |\n",
      "|8.33109e+17 |1                   |@Amazon one of 200$ from Walgreen's and  one for 50$ at Safeway both loaded to my account just fine..                                       |\n",
      "|8.33109e+17 |1                   |@amazon is anyone going to respond?                                                                                                         |\n",
      "|8.33109e+17 |1                   |@AmazonHelp omg the same thing. More generic replys I swear to god I am not going to let anyone else steal from me .over my dead body       |\n",
      "|2600866955.0|2                   |@AmazonUK been with amazon for ever , but you are messing with my loyalty. I have been locked out for 3 days, say you will call, nothing!!  |\n",
      "|2600866955.0|2                   |@AmazonHelp Several emails, they say ring customer services, i do, they say we will call you back, you dont. Annoying eh?                   |\n",
      "|2600866955.0|2                   |@AmazonHelp Have done that, no extra ones. emails just say call customer service, I call and they say that the specialists will call back!! |\n",
      "|2600866955.0|2                   |@AmazonHelp Been locked out for 3 days, you promise to call and nothing. Been with prime for years its no way to treat loyalty.             |\n",
      "|2600866955.0|2                   |@AmazonHelp Guys that's what I have done 3 times, I'm stuck in a Groundhog Day loop. The operator just says the account team will call back |\n",
      "|516012363.0 |2                   |@amazon What's up with the packaging lately? Today, I received a pantry box with an item that twisted open and leaked all over everything.  |\n",
      "|516012363.0 |2                   |@AmazonHelp Yes, I did. The site won't let me to leave the feedback. Second time in a couple weeks now...                                   |\n",
      "|8.45516e+17 |2                   |Just got my new #fidghetspinner at @amazon for 3$ . Get yours now https://t.co/RQVcIh4sGU                                                   |\n",
      "|8.45516e+17 |2                   |@AmazonHelp Thanks a lot !!!!                                                                                                               |\n",
      "|1386896743.0|3                   |@AmazonHelp sir actually it is showing the refund has been done but the amount has not been deposited to my bank account                    |\n",
      "|1386896743.0|3                   |@AmazonHelp this was only sent to my mail https://t.co/uVPMLet204                                                                           |\n",
      "|1386896743.0|3                   |@AmazonHelp sir till now the amount was not been returned to my bank account                                                                |\n",
      "+------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "amazon_sub_df2.show(20, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "_quRmxfMKfpI",
    "outputId": "8143203f-02b8-47f5-9884-c8cab9c096e9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "194305"
      ]
     },
     "execution_count": 27,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Total number of rows\n",
    "amazon_sub_df2.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "xkzmh4v-DogX",
    "outputId": "7323c000-4f95-4cff-90ef-8b83e40f489e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 28,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking number of rows (tweets) for that user again\n",
    "amazon_sub_df2.filter(amazon_sub_df2.user_id_str == '85741735.0').count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wtwt5JIEvdL-"
   },
   "source": [
    "# Step 4: Finding popular users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "DlMXPplxvkFK",
    "outputId": "e60df872-a1ea-407f-bbf1-0123d7d1c034"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5063"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a filter to find popular users who have more than 5000 followers\n",
    "popular_df = amazon_sub_df2.filter(amazon_sub_df2.user_followers_count >= 5000)\n",
    "popular_df.count() # number of rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 518
    },
    "colab_type": "code",
    "id": "tMIHl4KLIM-c",
    "outputId": "3db1426f-503e-4bbb-832e-02f4da3059bd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|user_id_str |user_followers_count|text_                                                                                                                                                         |\n",
      "+------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|3380217501.0|5163                |Looks like my parcel isn't coming today @AmazonHelp 😒 I live 5 mins from a warehouse how can it be late?                                                     |\n",
      "|3380217501.0|5163                |@AmazonHelp It says it was dispatched on the 30th and should arrive today, but nothing. When something I ordered after this arrived on Sat                    |\n",
      "|23069141.0  |6550                |@AmazonUK hi! I didn't have an option in Jan for overnight, just standard, any way to change this without mucking up my preorder? Thx! https://t.co/L4QEBPBbCi|\n",
      "|236555403.0 |10106               |@AmazonHelp hi I've had money go out of my account and says it's for amazon prime which I have never signed up for???                                         |\n",
      "|236555403.0 |10106               |@AmazonHelp it says I can end it but not cancel will this work?                                                                                               |\n",
      "|19454203.0  |11608               |@amazon One of your hard working KY employees lost her ID and other cards while packing my Prime box. How can I return it?                                    |\n",
      "|97610175.0  |15738               |@amazonIN Order# 171-7573811-5583505 has not delivered. Almost 25 days gone. How many months do you need more?                                                |\n",
      "|97610175.0  |15738               |@AmazonHelp What I have to fill could you tell me more clearly?                                                                                               |\n",
      "|97610175.0  |15738               |@amazonIN @AmazonHelp I didn't ask for a gift voucher, I asked you to deliver my product ordered. #ShameonyouAmazon                                           |\n",
      "|97610175.0  |15738               |@AmazonHelp Don't regret, try to deliver the product ordered at the earliest. #ShameonyouAmazon                                                               |\n",
      "|474221855.0 |15847               |Is it true that @amazon no longer sends out City/State info to those who purchase items off wish lists? 🤔 I've bee told it's a new feature?!                 |\n",
      "|268870185.0 |40187               |@amazonprimenow you know it would be great if you had an add-on feature                                                                                       |\n",
      "|426206983.0 |55003               |@AmazonHelp Anyone/anywhere online I can get an update on a parcel paid to be delivered today?                                                                |\n",
      "|1282946089.0|1122296             |@cosanostrakosa It is finally here... Grab your own #OnePlus5, exclusively on Amazon, right here… https://t.co/SlIeWvqUPC                                     |\n",
      "|310361960.0 |5961                |@AmazonHelp Thank you so much for helping me with an author situation I couldn't figure out. Very much appreciated.                                           |\n",
      "|388808885.0 |6392                |@AmazonHelp I cancelled my prime before the free trial, but still got charged, any idea why? Double checked and it's cancelled.                               |\n",
      "|388808885.0 |6392                |@AmazonHelp Nobody else would have access                                                                                                                     |\n",
      "|450442747.0 |6689                |I ordered a book on @amazonIN and this particular poster was used as a packing material for the book . @stc_india https://t.co/DlDaSmgCht                     |\n",
      "|450442747.0 |6689                |@AmazonHelp sadly I can't leave a feedback for the packaging as it was send by third party seller on Amazon .                                                 |\n",
      "|450442747.0 |6689                |@amazonIN thanks for delivering this beauty well in time . Hats off to @GoalSquadShop for making it available in India . #COYG https://t.co/6xQheDF1vy        |\n",
      "+------------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "popular_df.show(20, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 518
    },
    "colab_type": "code",
    "id": "3K8rBpYCNthC",
    "outputId": "fc2b43cf-a70e-4039-8ae4-9f28e88a6f25"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|user_id_str|user_followers_count|text_                                                                                                                                          |\n",
      "+-----------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|26359772.0 |5003                |@AmazonHelp any chance you can tell me what time American Gods is going online on May 1st?                                                     |\n",
      "|213353996.0|5007                |@AmazonHelp Last delivery I ordered was marked as \"handed to resident\" when it was really dropped over fence in rain. https://t.co/8bJgpAiU1m  |\n",
      "|213353996.0|5007                |@AmazonHelp Had two shipments marked as \"delivered\" yesterday that never appeared. Still marked as delivered and I have 0/6 items.             |\n",
      "|314686291.0|5008                |Thanks to @AmazonHelp I’m now uploading my entire photo collection (nearly 2 decades worth!) Safe, cheap and peace of mind storage) Thanks!    |\n",
      "|72697263.0 |5010                |@amazon screwed friend.  Charged a year later after cancelled free trial. #badservice #screwed                                                 |\n",
      "|50457828.0 |5010                |I didn't even realize that I NEEDED a 6 pack of @DuncanToys yo-yos until I saw it. Thanks, @amazon 👍🏻 https://t.co/6M5oSAiKHm                |\n",
      "|258867194.0|5010                |S/O to @Amazon for the Deluxe Edition of #MassEffectAndromeda for the LOW #AmazonPrime https://t.co/SL7MaqgNgn                                 |\n",
      "|72697263.0 |5010                |@AmazonHelp she did but now her bank account is overdraft and fees will be charged to her                                                      |\n",
      "|50457828.0 |5010                |.@DuncanToys @amazon It's already paying off. Nice @IntuitAccts appearance too https://t.co/yn9Smecv0R                                         |\n",
      "|17371268.0 |5012                |@AmazonHelp Hi, I am trying to return something but when I click the box I can't continue https://t.co/qjS2lkHqbt                              |\n",
      "|215375629.0|5015                |This is what happens when #Hermes chucks the @AmazonUK parcel over the garden gate! #smashedplates https://t.co/MQtvXJSPEn                     |\n",
      "|215375629.0|5015                |@AmazonHelp Thanks, I've already raised an exchange request so replacement on its way out tomorrow 👍🏻👍🏻                                    |\n",
      "|8.17876e+17|5023                |@AmazonHelp Worst off, I have two chats confirming items were received!                                                                        |\n",
      "|8.17876e+17|5023                |@AmazonHelp They claimed a case was opened, but I'm not sure if it was. The representative was so rude and accusatory I don't trust him at all.|\n",
      "|20717001.0 |5024                |@AmazonHelp And receive a refund? I tried to cancel last month and was given another \"30 days free\". Then charged a year's fee 🙄              |\n",
      "|20717001.0 |5024                |@AmazonHelp I turned on the reminder to remind me about being charged for prime, nothing. I did get charged for prime though... :(             |\n",
      "|543581602.0|5024                |@AmazonHelp Does this mean in order to watch a movie off Amazon Prime Video, I need to watch the movie on my Kindle Fire?                      |\n",
      "|116509330.0|5025                |@AmazonHelp No, usually if it's Intelcom I know there will be an issue https://t.co/vNkXFg0T2m                                                 |\n",
      "|116509330.0|5025                |@AmazonHelp Arrived this morning, thanks guys.                                                                                                 |\n",
      "|116509330.0|5025                |@AmazonHelp my same day/one day package was supposed to arrive today but so far is not here 😔 https://t.co/SgH44pO0nn                         |\n",
      "+-----------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check: Sorting follower count Ascending\n",
    "popular_df.sort(popular_df.user_followers_count).show(20, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 518
    },
    "colab_type": "code",
    "id": "xFyheoWtRXaS",
    "outputId": "10386574-cd58-40c2-af7c-c32502a56320"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|user_id_str |user_followers_count|text_                                                                                                                                                     |\n",
      "+------------+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|20793816.0  |2776204             |@Liberal_Lisa You're all set. We'll remind you when #PrimeDay is here! https://t.co/kQdsFAczL6                                                            |\n",
      "|83554234.0  |1958810             |@AmazonHelp Over 1k properties in this block, where should I start? 4 orders/ 7 items to be delivered YESTERDAY to me and still... https://t.co/CdIavyHYD0|\n",
      "|83554234.0  |1958810             |@AmazonHelp Can you help?                                                                                                                                 |\n",
      "|83554234.0  |1958810             |@AmazonHelp Up till this point I've received various poor excuses! It couldn't be simpler and amazon are doing there best to complicate things            |\n",
      "|83554234.0  |1958810             |@AmazonHelp Still not resolved...                                                                                                                         |\n",
      "|83554234.0  |1958810             |@AmazonHelp driver marked as delivered but never delivered. customer service refuses to help.                                                             |\n",
      "|83554234.0  |1958810             |@AmazonHelp Hi thank you for all! I called your team earlier and this time I had a lovely gentleman who sorted all. In love with my amazon again!         |\n",
      "|83554234.0  |1958810             |@AmazonHelp Basically there are 4 orders (7 items) and marked as delivered but I can prove via our cctv that wasn't and the customer service won't help   |\n",
      "|83554234.0  |1958810             |@AmazonHelp Thank you. message sent                                                                                                                       |\n",
      "|209004862.0 |1687854             |Love writing? Learn more about The Kindle UK Storyteller Award and the chance to win £20,000: https://t.co/EnlAgaaal2 #Storyteller2017                    |\n",
      "|26136956.0  |1554140             |@AmazonHelp could you please dm me, thanks.                                                                                                               |\n",
      "|151739682.0 |1379453             |@cevalogistics I ordered a desk off @amazon over a month ago. Delivery expected Jan 4th. I emailed u Jan 5, no reply. WHERE IS MY SHIPMENT?!              |\n",
      "|1282946089.0|1122296             |@cosanostrakosa It is finally here... Grab your own #OnePlus5, exclusively on Amazon, right here… https://t.co/SlIeWvqUPC                                 |\n",
      "|15640646.0  |964710              |@AmazonHelp Any progress on resolving this?                                                                                                               |\n",
      "|15640646.0  |964710              |@AmazonHelp My daughter bought me a $50 gift card. I'm still waiting for the email from Amazon. She and I have both contacted support. No resolution 1/2  |\n",
      "|15640646.0  |964710              |Can anyone tell me why @AmazonHelp can't straighten out a simple problem with a gift card? #amazon #amazongiftcard #customerservice                       |\n",
      "|15640646.0  |964710              |@AmazonHelp no resolution yet. She bought it last night about 7pm EDT. I'd like to get my gift card. BTW, my email is jandrewnelson2@gmail.com 2/2        |\n",
      "|15640646.0  |964710              |@AmazonHelp Any word on this?                                                                                                                             |\n",
      "|15640646.0  |964710              |@AmazonHelp Anything you can do to help?                                                                                                                  |\n",
      "|15640646.0  |964710              |@AmazonHelp Yes I did and thank you so much for the help on this. You are amazing and I will be spreading the word. Again, thanks!                        |\n",
      "+------------+--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check: Sorting follower count Descending\n",
    "popular_df.sort((popular_df.user_followers_count).desc()).show(20, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "28Q2qPCvXHbA",
    "outputId": "b468373b-2232-4fa3-8e7d-2a66f193d247"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2130"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of distinct users\n",
    "popular_df.select('user_id_str').distinct().count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nMyyiKbvH2CM"
   },
   "outputs": [],
   "source": [
    "# Finding number of tweets per user using groupBy\n",
    "groupedUsers = popular_df.groupby('user_id_str').count().withColumnRenamed(\"count\",\"tweet_count\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 498
    },
    "colab_type": "code",
    "id": "_jHc7EwmJirm",
    "outputId": "3e35fcbd-bace-4713-da9d-8e5210db25b8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+-----------+\n",
      "| user_id_str|tweet_count|\n",
      "+------------+-----------+\n",
      "|  17535421.0|         23|\n",
      "|2454614098.0|         20|\n",
      "|  16789906.0|         18|\n",
      "|2950894703.0|         18|\n",
      "| 518389533.0|         17|\n",
      "| 568744096.0|         15|\n",
      "|  17225620.0|         14|\n",
      "| 153426866.0|         14|\n",
      "|  22486545.0|         14|\n",
      "|  85741735.0|         13|\n",
      "|   2551781.0|         13|\n",
      "|  27326436.0|         12|\n",
      "|  31697976.0|         11|\n",
      "|  14247313.0|         11|\n",
      "|  94249055.0|         11|\n",
      "| 313202447.0|         11|\n",
      "|  17742722.0|         11|\n",
      "|  74177248.0|         11|\n",
      "|  15658980.0|         11|\n",
      "|  60964525.0|         11|\n",
      "+------------+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Sorting in descending order of count\n",
    "groupedUsers.sort((groupedUsers.tweet_count).desc()).show(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SQ6OObZLvmE8"
   },
   "source": [
    "# Step 5: Finding top words and save the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 129
    },
    "colab_type": "code",
    "id": "QCCYu-9JbI80",
    "outputId": "20036cbc-fd3a-4ead-ab97-c8a93958f986"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Looks like my parcel isn't coming today @AmazonHelp 😒 I live 5 mins from a warehouse how can it be late?\",\n",
       " '@AmazonHelp It says it was dispatched on the 30th and should arrive today, but nothing. When something I ordered after this arrived on Sat',\n",
       " \"@AmazonUK hi! I didn't have an option in Jan for overnight, just standard, any way to change this without mucking up my preorder? Thx! https://t.co/L4QEBPBbCi\",\n",
       " \"@AmazonHelp hi I've had money go out of my account and says it's for amazon prime which I have never signed up for???\",\n",
       " '@AmazonHelp it says I can end it but not cancel will this work?']"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Counting words frequency of the tweets posted by the popular users from step 4\n",
    "# Reading column 'text_' from spark df (popular_df) to a python list, which will then be read into RDD object \n",
    "tweet = popular_df.select(\"text_\").rdd.flatMap(lambda x: x).collect()\n",
    "tweet[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8r8K1y1bRQgC"
   },
   "outputs": [],
   "source": [
    "# Creating Spark Context\n",
    "from pyspark import SparkContext\n",
    "sc = SparkContext.getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 129
    },
    "colab_type": "code",
    "id": "iLyQnupNdQb0",
    "outputId": "2c997539-5189-4eb1-fc6b-dd5f8dd31df7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Looks like my parcel isn't coming today @AmazonHelp 😒 I live 5 mins from a warehouse how can it be late?\",\n",
       " '@AmazonHelp It says it was dispatched on the 30th and should arrive today, but nothing. When something I ordered after this arrived on Sat',\n",
       " \"@AmazonUK hi! I didn't have an option in Jan for overnight, just standard, any way to change this without mucking up my preorder? Thx! https://t.co/L4QEBPBbCi\",\n",
       " \"@AmazonHelp hi I've had money go out of my account and says it's for amazon prime which I have never signed up for???\",\n",
       " '@AmazonHelp it says I can end it but not cancel will this work?']"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reading 'tweet' to a RDD object using spark context\n",
    "tweet_rdd = sc.parallelize(tweet)\n",
    "tweet_rdd.take(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VvpO7MP9-oQ6"
   },
   "outputs": [],
   "source": [
    "# Function for cleaning the text of tweets\n",
    "import string\n",
    "import re\n",
    "\n",
    "def clean_tweet(x):\n",
    "  \n",
    "  # Delete all the URLs in the tweets\n",
    "  text00 = re.sub(r'www\\S+', '', x)\n",
    "  text01 = re.sub(r'http\\S+', '', text00)\n",
    "  \n",
    "  # Delete all the numbers in the tweets\n",
    "  text1 = ''.join([i for i in text01 if not i.isdigit()])\n",
    "  \n",
    "  # Delete all the punctuation marks in the tweets\n",
    "  text2 = text1.translate(str.maketrans('','',string.punctuation))\n",
    "  \n",
    "  # Convert text to LOWERCASE\n",
    "  text3 = text2.lower()\n",
    "\n",
    "  return text3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "id": "IndNUaf2-oHB",
    "outputId": "d8fc7bb3-b805-4440-bc1f-f8bfc434bcea"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['looks like my parcel isnt coming today amazonhelp 😒 i live  mins from a warehouse how can it be late',\n",
       " 'amazonhelp it says it was dispatched on the th and should arrive today but nothing when something i ordered after this arrived on sat',\n",
       " 'amazonuk hi i didnt have an option in jan for overnight just standard any way to change this without mucking up my preorder thx ',\n",
       " 'amazonhelp hi ive had money go out of my account and says its for amazon prime which i have never signed up for',\n",
       " 'amazonhelp it says i can end it but not cancel will this work',\n",
       " 'amazon one of your hard working ky employees lost her id and other cards while packing my prime box how can i return it',\n",
       " 'amazonin order  has not delivered almost  days gone how many months do you need more',\n",
       " 'amazonhelp what i have to fill could you tell me more clearly',\n",
       " 'amazonin amazonhelp i didnt ask for a gift voucher i asked you to deliver my product ordered shameonyouamazon',\n",
       " 'amazonhelp dont regret try to deliver the product ordered at the earliest shameonyouamazon']"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Cleaning the text of tweets: 1. Removing URLs, 2. Removing non-alphabets, 3. Lowercase \n",
    "clean_tweet_rdd = tweet_rdd.map(clean_tweet)\n",
    "clean_tweet_rdd.take(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "uHqfTAdCdedr",
    "outputId": "1f461851-6ef9-41b7-d820-7c39b439a4d2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('looks', 1), ('like', 1), ('my', 1), ('parcel', 1), ('isnt', 1)]"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Building Map function\n",
    "map = clean_tweet_rdd.flatMap(lambda line: line.split(\" \")).map(lambda word: (word, 1))\n",
    "map.take(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "EiuvPiX9d4cK",
    "outputId": "4afedde1-2259-4bde-b368-e32df00521ba"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('looks', 29), ('like', 137), ('😒', 6), ('i', 2896), ('live', 34)]"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Building Reduce function\n",
    "counts = map.reduceByKey(lambda a, b: a + b)\n",
    "counts.take(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "GTlP7QAAd4Vv",
    "outputId": "05d685af-db68-4e10-d372-c1b35df1e8be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6855\n"
     ]
    }
   ],
   "source": [
    "# Total number of distinct words\n",
    "print(len(counts.collect()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "id": "Z0XizxK3iZ7z",
    "outputId": "de13b6b8-691e-42aa-94df-e71210bffcce"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('amazonhelp', 3647),\n",
       " ('i', 2896),\n",
       " ('to', 2491),\n",
       " ('the', 2356),\n",
       " ('a', 1616),\n",
       " ('it', 1551),\n",
       " ('my', 1482),\n",
       " ('amazon', 1434),\n",
       " ('and', 1379),\n",
       " ('you', 1185)]"
      ]
     },
     "execution_count": 27,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting the Top 10 most popular words and their words frequency\n",
    "# Sorting 'counts' in descending order and getting the first 10 elements\n",
    "countsSortedTopTen = counts.takeOrdered(10, lambda a: -a[1] if len(a[0]) > 0 else False) # Conditioned on that number of characters in the string should be at least 1\n",
    "countsSortedTopTen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rf7HnaQQhQbI"
   },
   "outputs": [],
   "source": [
    "# Writing list to a text file\n",
    "with open('/content/gdrive/Output.txt', 'w') as f:\n",
    "    for item in countsSortedTopTen:\n",
    "        f.write(item[0] + ', ' + str(item[1]) + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tNPvXBinkgmO"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "TPU",
  "colab": {
   "collapsed_sections": [],
   "name": "rshah204_HW2_final_mobile.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}