Intro_to_data_science_tools-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's Get Started\n",
    "\n",
    "Learning Assumptions and Lab Objective: \n",
    "\n",
    "This first page that is part of the intro to data science tools lab will:\n",
    "\n",
    "* Allow you to make sure you have installed the Docker environment\n",
    "* Give you an overview of data science tools\n",
    "\n",
    "We will be covering this page as a class during the lab, so the most important task prior to class is to make sure Docker is set up and working prior to the lab. I will also discuss the set up of your environment prior to our lab.\n",
    "\n",
    "The intent of these labs is *not* to make you a competent R data analyst or expert data scientist. The objective *is* to help you gain a conceptual understanding of the process and technologies that are used in data science and \"big data\" via hands-on activities. Additionally, these labs will help increase your literacy in data science tools and techniques. Many of the activities in the lab are common tasks for data scientists. This high-level perspective will help you understand how data science tools fit into the larger context of business strategies that seek to leverage data science methods and tools. It will also hopefully motivate you to move beyond the legacy analytic tools of business like Microsoft Excel to see how powerful tools like R and Python are more accessible than ever before to non-engineers and non-scientists.\n",
    "\n",
    "All of us are responsible for creating a productive learning environment for the entire class, so please contribute to making the environment positive for everyone. We will follow the Python code of conduct- https://www.python.org/psf/codeofconduct/.\n",
    "\n",
    "You are expected to work in groups. If your technical and analytical expertise is beyond the basic level of these exercises, take on the role of adviser and help bring others up to speed and extend your work to add more sophistication or functionality. Many groups choose to adopt a [pair programming](https://en.wikipedia.org/wiki/Pair_programming) approach to working on these labs where one or two people act as the drivers (entering code and running analyses) and the others act as navigators (helping decide what to do next, researching things on the web, etc.).\n",
    "\n",
    "**This lab covers these topics:**\n",
    "\n",
    "* Data Science and Analytics Tools\n",
    "* What is a Desktop Environment?\n",
    "* What is Docker?\n",
    "* Using Jupyter Notebooks\n",
    "* Where to get help\n",
    "* How do you load data from a csv? \n",
    "* What is a dataframe?\n",
    "* What is Dplyr and how can it be used to focus and analyze data?\n",
    "* How to install R and RStudio on your desktop\n",
    "* Go through a simple version of a complete analysis lifecycle using weather data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science and Analytics Tools\n",
    "\n",
    "Data scientist has been categorized as [\"the sexiest job of the 21st sentury\"](https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century), [second in growth among the top 20 emerging jobs in 2017](https://economicgraph.linkedin.com/research/LinkedIns-2017-US-Emerging-Jobs-Report), and [in the top 10 most promising jobs of 2018](https://blog.linkedin.com/2018/january/11/linkedin-data-reveals-the-most-promising-jobs-and-in-demand-skills-2018). While there is still some debate about exactly what a data scientist is and what skills they should possess, we can make some conclusions based on our experience related to data science tools. \n",
    "\n",
    "While there are literally hundreds of tools a data scientist might use, perhaps the most foundation tools are what they use for desktop data analysis and exploratory data analysis. Until somewhat recently, anyone doing statistical analysis would typically learn one of the commercial statistical analysis packages like [SPSS](https://www.ibm.com/analytics/data-science/predictive-analytics/spss-statistical-software), [SAS](https://www.sas.com/en_us/software/stat.html), or [Stata](https://www.stata.com/). Today, while commercial products are still available and used by some, open source languages like Python and R are the top tools mentioned in new job postings especially by technology companies. Given many data scientists and analytics teams are coming from computer science rather than statistics, it is not surprising to see more general purpose programming languages like Python become popular data science tools. The open source ecosystem has also allowed these tools to have thousands of developers worldwide contributing to making them better and better while also being free to use which is a compelling value proposition. \n",
    "\n",
    "IMO, there is no one best tool. However, having skills in the popular or newer tools is generally a good thing for your career. \n",
    "\n",
    "Technically, general-purpose programming languages like Python and R are more flexible than proprietary or specialized software like [Tableau](https://www.tableau.com) and [Excel](https://products.office.com/en-us/excel). Both Python and R can complete any programming task as programmers have complete control over all the code. Tableau, Excel, and other more specialized analytical software have their place given they aim to make analysis and data exploration easier for those that don't want to learn to code.\n",
    "\n",
    ">\"You can't do data science in a UI.\" - Hadley Wickham, Chief Scientist- RStudio.\n",
    "\n",
    "Most data scientists today use either Python or R as their primary desktop analysis tool. Beyond being free to use and having thousands of developers contributing to their success, statistical programming languages also allow data scientists to document every step of an analysis to share with others and to use to create analytical products. To understand an analysis from a peer, rather than just looking at the output of an analysis, you have to go through the complete process they used by reading their code. Tools like Excel and Tableau are not scientific tools given any analysis is difficult to replicate as the logic used to create it lives in UI interactions or cell formulas. Finally, both Python and R can be used to move from a desktop analysis to a production, big data cloud environment like [Apache Spark](https://spark.apache.org/) realtively easily.\n",
    "\n",
    "\n",
    "For this workshop, we are going to primarily use the R programming language. While both Python and R are popular, some believe R is easier for non-programmers to learn. Additionally, R has a superior plotting library called GGplot. We will use Python for the last notebook using neural networks given its interface is easier to understand."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is an Environment?\n",
    "\n",
    "Data scientists use a variety of tools in their day to day work. A desktop data science environment includes all the software tools needed to explore data and conduct analyses using their personal computer. These tools can also be used to collect data from \"big data\" or cloud services like [Hadoop](http://hadoop.apache.org/), [Hive](http://hive.apache.org/), and [Spark](http://hadoop.apache.org/). Examples of some of these desktop environment tools include:\n",
    "\n",
    "* **A container software framework**- We are using [Docker](https://docker.com) as a way to have standardized collections of software for our labs. \n",
    "* **A statistical programming language**- examples include [Python](https://www.python.org/), [R](https://www.r-project.org/), or commercial packages like [SPSS](https://www.ibm.com/products/spss-statistics) or [SAS](https://www.sas.com/en_us/software/stat.html).\n",
    "* **Data science specific libraries**- Most statistical programming languages have add-on libraries useful for data science. For Python there is [SciPy](https://www.scipy.org/) and R has the [tidyverse](https://www.tidyverse.org/). There are also a variety of machine learning libraries like [Tensorflow](https://www.tensorflow.org/api_docs/), [MXNet](https://mxnet.apache.org/), [SciKitLearn for Python](http://scikit-learn.org/stable/) and [Caret for R](http://topepo.github.io/caret/index.html).\n",
    "* **Data notebooks**- When sharing results, interactive notebooks are a popular tool. The most common one for Python and increasingly for other languages like R is [Jupyter](http://jupyter.org/). There are several others including [Zepplin](https://zeppelin.apache.org/) from the Apache Software Foundation.\n",
    "* **Visualization tools or libraries**- Python and R have their specific libraries [Matplotlib](https://matplotlib.org/) and [Ggplot2](http://ggplot2.org/) which are included in SciPy and Tidyverse. There are many commercial tools for visualization like [Tableau](https://www.tableau.com/). There are also hybrid open-source/commercial products like [Plotly](https://plot.ly/).\n",
    "* **Software Integrated Development Environments (IDEs)**- IDEs help organize and streamline your software. I like [PyCharm](https://www.jetbrains.com/pycharm/) for Python. I use [Rstudio](https://www.rstudio.com/products/rstudio/) for R.\n",
    "* **Version control software**- As you work on more sophisticated projects and collaborate with others, version control becomes important. [Git](https://git-scm.com/) is the underlying version control software most used, but [Github](https://github.com/) is a centralized repository often used for some of the most important data science tools like https://github.com/apache/spark and https://github.com/tensorflow/tensorflow\n",
    "* **Database clients**- Connecting to traditional databases is a common task despite all the newer big data technologies. There are free clients like [SQL Workbench](http://www.sql-workbench.net/) and commercial products like [Datagrip](https://www.jetbrains.com/datagrip/). While most of the tools I use are open-source, in this case I really like Datagrip and don't mind paying for it.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Docker\n",
    "\n",
    "We are using Docker to manage the environments we are using for the labs in CIS 503 both for the desktop and when required in the cloud. Docker is a technology that is increasingly being used by data scientists to make working together using the same tools and technologies easier. Data scientists use many different technologies. In our class labs, for example, we will use the statistical programming language R, but we will also use over a dozen different R packages and libraries including those for plotting and machine learning. Some of these libraries require very specific set up procedures depending on the operating system. So, when working separately on the same analysis, setting up the same technical environment can be very challenging. \n",
    "\n",
    "Docker and its associated open source technology allow data scientists and developers to use a common platform to run applications and analyses on desktop computers as well as in cloud services like Google Cloud Platform, Amazon Web Services, and Microsoft Azure. Docker acts as a container for different applications. Docker is also the foundation for scaling up analyses using frameworks like Kubernetes and Pachyderm. Some of the benefits of using Docker for data science are detailed here- https://www.docker.com/solutions/data-analytics.\n",
    "\n",
    "The following video describes how GlaxoSmithKline (GSK) uses Docker in its data science practice.\n",
    "\n",
    "https://embed.vidyard.com/share/6yuKzv5Mr8UenJpwyEwbzc\n",
    "\n",
    "The following figure represents how Docker is used to deploy software containers.\n",
    "\n",
    "<img src=\"images/container-what-is-container.png\" width=\"50%\" height=\"50%\"/>\n",
    "\n",
    "\n",
    "# Jupyter Notebooks \n",
    "\n",
    "While many data scientists and developers write code in statistical programming languages like R or Python, raw code can be intimidating for students looking to learn these languages.  For example, raw R and Python code does not support inline plotting, output, or images. [Project Jupyter](http://jupyter.org/) has built a web-based platform for interacting with many languages include R and Python which is much more intuitive and more interactive. \n",
    "\n",
    "This web page was built using a Jupyter Notebook. Jupyter Notebooks are a way to share documents that have both live code in them as well as descriptions (like this code block), plots, images, and the output of any code that is run. Jupyter is a good way to surround your working code with descriptions and contextual information all while having real, running code you can change and experiment with. \n",
    "\n",
    "Jupyter started as a Python project, but given its popularity in data science now supports many other languages including R. Note that you can install R and Jupyter on your local computer without using Docker too if you want. I  also use a hosted version of Jupyter called [JupyterHub](https://github.com/jupyterhub/jupyterhub) for labs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#  Jupyter Notebooks\n",
    "\n",
    "If you are looking at this page on Github at the URL https://github.com/azbones/cis503_data_science_tools/blob/master/Intro_to_data_science_tools-1.ipynb you are seeing a static version of a Jupyter notebook. \n",
    "\n",
    "If you are looking at this page inside the live Jupyter environment, then this is a live, interactive web page that is connected to R.  \n",
    "\n",
    "## Types of Cells in Jupyter Notebooks\n",
    "\n",
    "Jupyter Notebooks are made up of cells which are blocks of code or text. This cell is a text block that uses [markdown](https://daringfireball.net/projects/markdown/) formatting which is sort of a simpler version of HTML.  Markdown is used in many technical systems for enhanced text formatting. In Jupyter Notebooks, it allows us make cells like this with basic formatting and embedded images. \n",
    "\n",
    "You can also include equations by using [MathJax](https://www.mathjax.org/).  For example,\n",
    "\n",
    "\\begin{equation*}\n",
    "\\hat{y} = a + \\beta x \n",
    "\\end{equation*}\n",
    "\n",
    "If you are viewing this notebook on Jupyter Hub or on a locally running Docker Jupyter Notebook, then cells like this one are editable. Click (or double-click) on this cell to see the markdown source. Press the run button to change the source back into the formatted text.\n",
    "\n",
    "Code blocks are the other most used type of cells. Code blocks are connected to a programming language kernel.  Jupyter supports many different languages, but we are working with R. You should see the specific kernel in the upper right had corner of the notebook page like this. This image indicates that the R programming kernel is connected to the notebook.  \n",
    "\n",
    "<img src=\"images/jupyter-kernel.png\">\n",
    "\n",
    "\n",
    "Cells that are code blocks will be preceded by a ```In []:``` which stands for \"input\". When the kernel is busy (running code), you will see a ```[*]``` by the code block. When the code has been run, the blank ```In []``` will be replaced by a number like ``In [1]``.  The output of the code will appear below the code block. \n",
    "\n",
    "In the blank code block below, try entering the following R code and then running it by pressing control-return or Cell -> Run Cells.\n",
    "\n",
    "```\n",
    "print(\"Hello, World!\")\n",
    "```\n",
    "\n",
    "Note that comments, or text that will not be executed when the code block is run are preceded by ```#```.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enter your code to run here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see an R plot run the next cell.  ```cars``` is a sample dataset inlcuded in base R."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(cars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Please continue in the next notebook page by clicking here](Intro_to_data_science_tools-2.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}