kaggle_introduction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "kaggle_introduction.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mlites/mlites2019/blob/master/kaggle_introduction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "vSp9f2X0ggwa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Introduction to Kaggle Datasets\n",
        "\n",
        "In this exercise, we'll learn how to get data into Colab\n",
        "\n",
        "First, we'll load up Google Drive, where we'll store our work\n",
        "\n",
        "Second, we'll search for data stored on Kaggle's servers\n",
        "\n",
        "Finally, we'll download some datasets we'll use in this course"
      ]
    },
    {
      "metadata": {
        "id": "XfESbjzePbMT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# FYI If you ever need to kill the VM and start fresh, uncomment and run this:\n",
        "#!kill -9 -1\n",
        "\n",
        "# the ! indicates a command is to be run on the system's command line"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8j_ZYrtGhBX0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Mounting Google Drive\n",
        "...is easy"
      ]
    },
    {
      "metadata": {
        "id": "0JalKi1VB8mj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from google.colab import drive # from PACKAGE import MODULE\n",
        "drive.mount('/content/gdrive') # mount your Google Drive to the /content/gdrive folder in your VM"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xpAlTBYU9nA3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_jaa4bkm8K_8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now we'll make a new directory in our Google Drive to work out of. This will make things easier later because the Colab Virtual Machines get recycled every once in a while, so work you've done isn't necessarily persistent across sessions.\n",
        "\n",
        "We'll call the directory 'mlites' and add a link to it in the default /content/ location in Colab"
      ]
    },
    {
      "metadata": {
        "id": "24zB-_ir72xG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!mkdir \"/content/gdrive/My Drive/mlites\"\n",
        "!ln -s \"/content/gdrive/My Drive/mlites\" \"/content/mlites\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Kyyun7qbqZq6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setting up Kaggle\n",
        "\n",
        "...is a bit more complicated\n",
        "\n",
        "[Setting up Kaggle in Colab](https://towardsdatascience.com/setting-up-kaggle-in-google-colab-ebb281b61463)\n",
        "\n",
        "1. Sign up for Kaggle if you're not already a member\n",
        "2. Go to _My Account_\n",
        "3. Go to _Create New API Token_\n",
        "4. That will download a file called **kaggle.json**\n",
        "5. Click *Files* -> *Upload* on the left and upload **kaggle.json**\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "kcu2pIGhpxQb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# set up key authentication\n",
        "# FYI in the Colab VM (Virtual Machine) your user name is 'root'\n",
        "\n",
        "#!mkdir is the system command to make a new directory\n",
        "!mkdir /content/mlites/kaggle\n",
        "\n",
        "#move the authentication file into mlites folder so we don't have to re-upload it in the future\n",
        "!mv /content/kaggle.json /content/mlites/kaggle/\n",
        "\n",
        "#the . makes it hidden, this is where our credentials will be stored\n",
        "!mkdir /root/.kaggle\n",
        "\n",
        "#copy the file to a place that kaggle expects to find it\n",
        "!cp /content/mlites/kaggle/kaggle.json /root/.kaggle/kaggle.json \n",
        "\n",
        "#change the permissions to avoid leaking your credentials\n",
        "!chmod 600 /root/.kaggle/kaggle.json \n",
        "\n",
        "\n",
        "# install the kaggle API package\n",
        "\n",
        "# pip is a python program for installing new packages\n",
        "!pip install kaggle\n",
        "\n",
        "#setup kaggle to use the /content/mlites/kaggle directory we made earlier\n",
        "!kaggle config set -n path -v/content/mlites/kaggle \n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UpCVmBMkt8dV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Find Kaggle datasets of interest\n",
        "\n",
        "Kaggle package API details\n",
        "\n",
        "https://github.com/Kaggle/kaggle-api#datasets\n",
        "\n",
        "only the first 20 results are shown, additional pages can be shown with the --page flag"
      ]
    },
    {
      "metadata": {
        "id": "T_7sVX75sYnh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!kaggle datasets list --tags oceans #find datasets tagged with 'oceans'\n",
        "!kaggle datasets list --user noaa #find datasets by user 'NOAA'\n",
        "!kaggle datasets list --search environment --page 2 #find page 2 of datasets using search term 'environment'\n",
        "!kaggle datasets list --search alaska"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Tn5iOpLEv9H8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Iditarod dataset\n",
        "\n",
        "Now find the specific files you want and download them\n",
        "\n",
        "Let's see what files are available for the 2017 Iditarod dataset"
      ]
    },
    {
      "metadata": {
        "id": "vCWpgYW8wACG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!kaggle datasets files iditarod/iditarod-race"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8oFjm_yzwonh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's download all the files, they'll show up in /content/mlites/kaggle/datasets"
      ]
    },
    {
      "metadata": {
        "id": "ktpZKadxwqg9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!kaggle datasets download iditarod/iditarod-race"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZMIlCZctyq7A",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#unzip the files into the current working directory (\"/content/\")\n",
        "!unzip /content/mlites/kaggle/datasets/iditarod/iditarod-race/iditarod-race.zip -d ./"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "syUDb3mdzmPp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# let's rename it to something more useful\n",
        "!mv report.csv iditarod.csv\n",
        "\n",
        "# and take a quick look\n",
        "import pandas as pd # pandas is library providing high-performance, easy-to-use data structures and data analysis tools\n",
        "iditarod = pd.read_csv('iditarod.csv') # reads the CSV file into a Panda dataframe\n",
        "iditarod.shape # the shape is the number of rows, columns in the dataframe\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "J3dNlrGI2BdC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "iditarod.head(10) # head() is a command to show the top few lines of the file, in this case 10\n",
        "iditarod.tail(10) # tail() shows the bottom\n",
        "\n",
        "#note that the jupyter notebook only displays the last command printed here, as a nicely formatted table"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jxqykLVjSqEi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## BIOL342 Dataset\n",
        "\n",
        "Now we'll download the data that we'll use for examples in this course"
      ]
    },
    {
      "metadata": {
        "id": "X01qpm1kSpkB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!kaggle datasets list --user rec3141\n",
        "!kaggle datasets files rec3141/biol342-genome-data\n",
        "!kaggle datasets download rec3141/biol342-genome-data\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H4OBXQYvTLWG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!unzip /content/mlites/kaggle/datasets/rec3141/biol342-genome-data/biol342-genome-data.zip -d ./"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xKJMxWcrnaAa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "covlengc = pd.read_csv('biol342_cov_len_gc.tsv',sep='\\t') #read in a TAB separated file\n",
        "covlengc.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OWLU-vXdn4QN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "covlengc.head(10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DWJPplOjTd2b",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "covlengc.tail(10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ihos-YLzoBVO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## The End\n",
        "\n",
        "Nice job!\n",
        "\n",
        "In this lesson we learned:\n",
        "\n",
        "1. How to mount our Google Drive into our Jupyter notebook\n",
        "2. How to setup the Kaggle package in our Jupyter notebook\n",
        "3. How to search for and download Kaggle datasets using **!kaggle datasets**\n",
        "4. How to perform system commands like cp, mv, mkdir, unzip, chmod, and pip on the command line using **!**\n",
        "5. How to import a text file into a pandas dataframe\n",
        "6. How to preview and find some basic information about the data using **pd.head()**, **pd.tail()**, and the **shape** attribute"
      ]
    }
  ]
}