getting-started/collect-n-explore.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Collection and Exploration with the Iguazio Data Science Platform\n",
    "\n",
    "Learn how to quickly start using the Iguazio Data Science Platform to collect, ingest, and explore data.\n",
    "\n",
    "- [Overview](#gs-overview)\n",
    "  - [Platform Data Containers](#platform-data-containers)\n",
    "- [Collecting and Ingesting Data](#gs-data-collection-and-ingestion)\n",
    "  - [Ingesting Data From an External Database to a NoSQL Table Using V3IO Frames](#ingest-from-external-db-to-no-sql-using-frames)\n",
    "  - [Ingesting Files from Amazon S3](#ingest-from-amazon-s3)\n",
    "  - [Streaming Data From an External Streaming Engine Using Nuclio](#streaming-data-from-an-external-streaming-engine-using-nuclio)\n",
    "- [Exploring and Processing Data](#gs-data-exploration-and-processing)\n",
    "  - [Exploring Data Using Spark DataFrames](#data-exploration-spark)\n",
    "  - [Exploring Data Using V3IO Frames and pandas DataFrames](#data-exploration-v3io-frames-n-pandas)\n",
    "  - [Exploring Data Using SQL](#data-exploration-sql)\n",
    "- [Data Collection and Exploration Getting-Started Example](#getting-started-example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-overview\"></a>\n",
    "## Overview\n",
    "\n",
    "This tutorial explains and demonstrates how to collect, ingest, and explore data with the Iguazio Data Science Platform (**\"the platform\"**).<br>\n",
    "For an overview of the platform and how it can be used to implement a full data science workflow, see the [**welcome**](../welcome.ipynb) tutorial notebook.<br>\n",
    "For full end-to-end platform use-case application demos, see [**demos**](../demos/README.ipynb) tutorial notebooks directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"platform-data-containers\"></a>\n",
    "### Platform Data Containers\n",
    "\n",
    "Data is stored within data containers in the platform's distributed file system (DFS).\n",
    "All platform clusters have two predefined containers:\n",
    "\n",
    "- <a id=\"default-container\"></a> The default **\"bigdata\"** container.\n",
    "- <a id=\"users-container\"></a>The **\"users\"** container, which is designed to contain **&lt;username&gt;** directories that provide individual development environments for storing user-specific data.\n",
    "  The platform's Jupyter Notebook, Zeppelin, and web-based shell \"command-line services\" automatically create such a directory for the running user of the service and set it as the home directory of the service environment.\n",
    "  You can leverage the following environment variables, which are predefined in the platform's command-line services, to access this running-user directory from your code:\n",
    "\n",
    "  - `V3IO_USERNAME` &mdash; set to the username of the running user of the Jupyter Notebook service.\n",
    "  - `V3IO_HOME` &mdash; set to the running-user directory in the \"users\" container &mdash; **users/&lt;running user&gt;**.\n",
    "  - `V3IO_HOME_URL` &mdash; set to the fully qualified `v3io` path to the running-user directory &mdash; `v3io://users/<running user>`.\n",
    "\n",
    "The data containers and their contents are referenced differently depending on the programming interface.\n",
    "For example:\n",
    "\n",
    "- In local file-system (FS) commands you use the predefined `v3io` root data mount &mdash; `/v3io/<container name>[/<data path>]`.\n",
    "  There's also a predefined local-FS `User` mount to the **users/&lt;running user&gt;** directory, and you can use the aforementioned environment variables when setting data paths.\n",
    "  For example, `/v3io/users/$V3IO_USERNAME`, `/v3io/$V3IO_HOME`, and `/User` are all valid ways of referencing the **users/&lt;running user&gt;** directory from a local FS command.\n",
    "- In Hadoop FS or Spark DataFrame commands you use a fully qualified path of the format `v3io://<container name>/<data path>`.\n",
    "  You can also use environment variables with these interfaces.\n",
    "\n",
    "For detailed information and examples on how to set the data path for each interface, see [Setting Data Paths](https://www.iguazio.com/docs/tutorials/latest-release/getting-started/fundamentals/#data-paths) and the examples in the platform's tutorial Jupyter notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-data-collection-and-ingestion\"></a>\n",
    "## Collecting and Ingesting Data\n",
    "\n",
    "The platform supports various alternative methods for collecting and ingesting data into its data containers (i.e., its data store).\n",
    "For more information, see the [**welcome**](../welcome.ipynb#data-collection-and-ingestion) platform tutorial Jupyter notebook\n",
    "The data collection and ingestion can be done as a one-time operation, using different platform APIs &mdash; which can be run from your preferred programming interface, such as an interactive web-based Jupyter or Zeppelin notebook &mdash; or as an ongoing ingestion stream, using Nuclio serverless functions.\n",
    "This section explains and demonstrates how to collect and ingest (import) data into the platform using code that's run from a Jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ingest-from-external-db-to-no-sql-using-frames\"></a>\n",
    "### Ingesting Data From an External Database to a NoSQL Table Using V3IO Frames\n",
    "\n",
    "For an example of how to collect data from an external database &mdash; such as MySQL, Oracle, and PostgreSQL &mdash; and ingest (write) it into a NoSQL table in the platform, using the V3IO Frames API, see the [read-external-db](read-external-db.ipynb) getting-started tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ingest-from-amazon-s3\"></a>\n",
    "### Ingesting Files from Amazon S3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ingest-from-amazon-s3-using-curl\"></a>\n",
    "#### Ingesting Files from Amazon S3 to the Platform File System Using curl\n",
    "\n",
    "You can use a simple [curl](https://curl.haxx.se/) command to ingest a file (object) from an external web data source, such as an Amazon S3 bucket, to the platform's distributed file system (i.e., into the platform's data store).\n",
    "This is demonstrated in the following code example and in the [getting-started example](#getting-started-example) in this notebook.\n",
    "The [spark-sql-analytics](spark-sql-analytics.ipynb) getting-started tutorial notebook demonstrates a similar ingestion using [Botocore](https://github.com/boto/botocore).\n",
    "\n",
    "The example in the following cells uses curl to read a CSV file from the [Iguazio sample data-sets](http://iguazio-sample-data.s3.amazonaws.com/) public Amazon S3 bucket and save it to an **examples** directory in the running-user directory of the predefined \"users\" data container (`/v3io/users/$V3IO_USERNAME` = `v3io/$V3IO_HOME` = `/User`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p /User/examples # <=> /v3io/${V3IO_HOME}/examples or /v3io/users/${V3IO_USERNAME}/examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  861k  100  861k    0     0   567k      0  0:00:01  0:00:01 --:--:--  567k\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "CSV_PATH=\"/User/examples/stocks.csv\"\n",
    "curl -L \"iguazio-sample-data.s3.amazonaws.com/2018-03-26_BINS_XETR08.csv\" > ${CSV_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ingest-from-amazon-s3-to-nosql-table-using-v3io-frames-n-pandas\"></a>\n",
    "#### Ingesting Data from Amazon S3 to a NoSQL Table Using V3IO Frames and pandas\n",
    "\n",
    "For an example of how to import data from Amazon S3 and save it into a NoSQL table in the platform's data store by using V3IO Frames and pandas DataFrames, see the [frames](frames.ipynb) getting-started tutorial notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"streaming-data-from-an-external-streaming-engine-using-nuclio\"></a>\n",
    "### Streaming Data From an External Streaming Engine Using Nuclio\n",
    "\n",
    "To read data from an external streaming engine &mdash; such as Kafka, Kinesis, or RabbitMQ &mdash; create a Nuclio function that listens on the stream, and write the stream data to a NoSQL or time-series database (TSDB) table:\n",
    "\n",
    "1. In the dashboard's side navigation menu, select **Functions** to display the Nuclio serverless functions dashboard.\n",
    "2. Create a new Nuclio project or select an existing project.\n",
    "3. In the action toolbar, select **Create Function**.\n",
    "4. Enter a function name, select an appropriate template, such as **kafka-to-tsdb**, configure the required template parameters, and apply your changes.\n",
    "5. Select **Deploy** from the action toolbar to deploy your function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-data-exploration-and-processing\"></a>\n",
    "## Exploring and Processing Data\n",
    "\n",
    "After you have ingested data into the platform's data containers, you can use various alternative methods and tools to explore and analyze the data.\n",
    "Data scientists typically use Jupyter Notebook to run the exploration phase.\n",
    "As outlined in the [**welcome**](../welcome.ipynb#data-exploration-and-processing) tutorial notebook, the platform's Jupyter Notebook service has a wide range of pre-deployed popular data science tools (such as Spark and Presto) and allows installation of additional tools and packages, enabling you to use different APIs to access the same data from a single Jupyter notebook.\n",
    "This section explains and demonstrates how to explore data in the platform from a Jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"data-exploration-spark\"></a>\n",
    "### Exploring Data using Spark DataFrames\n",
    "\n",
    "Spark is a distributed computing framework for data analytics.\n",
    "You can easily run distributed Spark jobs on you platform cluster that use Spark DataFrames to access data files (objects), tables, or streams in the platform's data store.\n",
    "For more information and examples, see the [spark-sql-analytics](spark-sql-analytics.ipynb) getting-started tutorial notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"data-exploration-v3io-frames-n-pandas\"></a>\n",
    "### Exploring Data Using V3IO Frames and pandas DataFrames\n",
    "\n",
    "Iguazio's V3IO Frames open-source data-access library <font color=\"#00BCF2\">\\[Tech Preview\\]</font> provides a unified high-performance DataFrames API for accessing NoSQL, stream, and time-series data in the platform's data store.\n",
    "These DataFrames can also be used to analyze the data with pandas. \n",
    "For details and examples, see the [frames](frames.ipynb) getting-started tutorial notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"data-exploration-sql\"></a>\n",
    "### Exploring Data Using SQL\n",
    "\n",
    "You can run SQL statements (`SELECT` only) on top of NoSQL tables in the platform's data store.\n",
    "To do this, you need to use the Jupyter `%sql` or `%%sql` IPython Jupyter magic followed by an SQL statement.\n",
    "The platform supports standard ANSI SQL semantics.\n",
    "Under the hood, the SQL statements are executed via [Presto](https://prestodb.github.io/), which is a distributed SQL engine designed from the ground up for fast analytics queries.\n",
    "\n",
    "In the example in the following cell, as a preparation for the SQL query, the **stocks.csv** file that was ingested to the **users/&lt;running user&gt;/examples** platform data-container directory in the previous [Ingesting Files from Amazon S3 to the Platform](#ingest-from-amazon-s3) example is written to a **stocks_example_tab** NoSQL table in the same directory.\n",
    "Then, an SQL `SELECT` query is run on this table.\n",
    "You can also find a similar example in the [getting-started example](#getting-started-example) in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use V3IO Frames to convert the CSV file that was ingested in the AWS S3 data-collection example to a NoSQL table.\n",
    "# NOTE: Make sure to first create a V3IO Frames service from the \"Services\" page of the platform dashboard, and run the\n",
    "# \"Ingesting Files from Amazon S3 to the Platform File System Using curl\" example to create users/$V3IO_USERNAME/examples/stocks.csv.\n",
    "import pandas as pd\n",
    "import v3io_frames as v3f\n",
    "import os\n",
    "\n",
    "# Create a V3IO Frames client for the \"users\" data container\n",
    "client = v3f.Client(\"framesd:8081\", container=\"users\")\n",
    "\n",
    "# Full CSV file path\n",
    "csv_path = \"/User/examples/stocks.csv\"\n",
    "# Relative NoSQL table path within the \"users\" container\n",
    "rel_nosql_table_path = os.getenv('V3IO_USERNAME') + \"/examples/stocks_example_tab\"\n",
    "\n",
    "# Read the CSV file into a Pandas DataFrame\n",
    "df = pd.read_csv(csv_path, header=\"infer\")\n",
    "\n",
    "# Convert the CSV file to a NoSQL table\n",
    "client.write(\"kv\", rel_nosql_table_path, df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <th>securitydesc</th>\n",
       "        <th>securitytype</th>\n",
       "        <th>time</th>\n",
       "        <th>isin</th>\n",
       "        <th>minprice</th>\n",
       "        <th>date</th>\n",
       "        <th>endprice</th>\n",
       "        <th>numberoftrades</th>\n",
       "        <th>mnemonic</th>\n",
       "        <th>currency</th>\n",
       "        <th>securityid</th>\n",
       "        <th>maxprice</th>\n",
       "        <th>tradedvolume</th>\n",
       "        <th>startprice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ISHARES NIKKEI 225 U.ETF</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:51</td>\n",
       "        <td>DE000A0H08D2</td>\n",
       "        <td>15.818</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>15.818</td>\n",
       "        <td>1</td>\n",
       "        <td>EXX7</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504322</td>\n",
       "        <td>15.818</td>\n",
       "        <td>3217</td>\n",
       "        <td>15.818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>AXA S.A. INH.     EO 2,29</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:54</td>\n",
       "        <td>FR0000120628</td>\n",
       "        <td>21.19</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>21.19</td>\n",
       "        <td>1</td>\n",
       "        <td>AXA</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505172</td>\n",
       "        <td>21.19</td>\n",
       "        <td>25</td>\n",
       "        <td>21.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>BK OF IRELD GRP      EO 1</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:19</td>\n",
       "        <td>IE00BD1RP616</td>\n",
       "        <td>6.935</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>6.935</td>\n",
       "        <td>1</td>\n",
       "        <td>BIRG</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2537059</td>\n",
       "        <td>6.935</td>\n",
       "        <td>200</td>\n",
       "        <td>6.935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>IS MSCI BRAZ.U.ETF USD D</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:52</td>\n",
       "        <td>IE00B0M63516</td>\n",
       "        <td>29.5</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>29.5</td>\n",
       "        <td>1</td>\n",
       "        <td>IQQB</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505576</td>\n",
       "        <td>29.5</td>\n",
       "        <td>100</td>\n",
       "        <td>29.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ISH.S.EU.600 TEL.U.ETF A.</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:52</td>\n",
       "        <td>DE000A0H08R2</td>\n",
       "        <td>22.39</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>22.39</td>\n",
       "        <td>1</td>\n",
       "        <td>EXV2</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504334</td>\n",
       "        <td>22.39</td>\n",
       "        <td>28</td>\n",
       "        <td>22.39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>CARL ZEISS MEDITEC AG</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:55</td>\n",
       "        <td>DE0005313704</td>\n",
       "        <td>50.25</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>50.35</td>\n",
       "        <td>12</td>\n",
       "        <td>AFX</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504922</td>\n",
       "        <td>50.35</td>\n",
       "        <td>1210</td>\n",
       "        <td>50.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ISHSIII-EO C.BD 1-5YR EOD</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:35</td>\n",
       "        <td>IE00B4L60045</td>\n",
       "        <td>110.215</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>110.215</td>\n",
       "        <td>1</td>\n",
       "        <td>EUNT</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505659</td>\n",
       "        <td>110.215</td>\n",
       "        <td>250</td>\n",
       "        <td>110.215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>TOTAL S.A.      EO 2,50</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:53</td>\n",
       "        <td>FR0000120271</td>\n",
       "        <td>45.955</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>45.955</td>\n",
       "        <td>1</td>\n",
       "        <td>TOTB</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505166</td>\n",
       "        <td>45.955</td>\n",
       "        <td>352</td>\n",
       "        <td>45.955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>AURUBIS AG</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:59</td>\n",
       "        <td>DE0006766504</td>\n",
       "        <td>66.8</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>66.8</td>\n",
       "        <td>1</td>\n",
       "        <td>NDA</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505060</td>\n",
       "        <td>66.8</td>\n",
       "        <td>59</td>\n",
       "        <td>66.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>COMDIRECT BANK AG</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:36</td>\n",
       "        <td>DE0005428007</td>\n",
       "        <td>11.6</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>11.6</td>\n",
       "        <td>3</td>\n",
       "        <td>COM</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504931</td>\n",
       "        <td>11.6</td>\n",
       "        <td>389</td>\n",
       "        <td>11.6</td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "[('ISHARES NIKKEI 225 U.ETF', 'ETF', '08:51', 'DE000A0H08D2', 15.818, '2018-03-26', 15.818, 1, 'EXX7', 'EUR', 2504322, 15.818, 3217, 15.818),\n",
       " ('AXA S.A. INH.     EO 2,29', 'Common stock', '08:54', 'FR0000120628', 21.19, '2018-03-26', 21.19, 1, 'AXA', 'EUR', 2505172, 21.19, 25, 21.19),\n",
       " ('BK OF IRELD GRP      EO 1', 'Common stock', '08:19', 'IE00BD1RP616', 6.935, '2018-03-26', 6.935, 1, 'BIRG', 'EUR', 2537059, 6.935, 200, 6.935),\n",
       " ('IS MSCI BRAZ.U.ETF USD D', 'ETF', '08:52', 'IE00B0M63516', 29.5, '2018-03-26', 29.5, 1, 'IQQB', 'EUR', 2505576, 29.5, 100, 29.5),\n",
       " ('ISH.S.EU.600 TEL.U.ETF A.', 'ETF', '08:52', 'DE000A0H08R2', 22.39, '2018-03-26', 22.39, 1, 'EXV2', 'EUR', 2504334, 22.39, 28, 22.39),\n",
       " ('CARL ZEISS MEDITEC AG', 'Common stock', '08:55', 'DE0005313704', 50.25, '2018-03-26', 50.35, 12, 'AFX', 'EUR', 2504922, 50.35, 1210, 50.3),\n",
       " ('ISHSIII-EO C.BD 1-5YR EOD', 'ETF', '08:35', 'IE00B4L60045', 110.215, '2018-03-26', 110.215, 1, 'EUNT', 'EUR', 2505659, 110.215, 250, 110.215),\n",
       " ('TOTAL S.A.      EO 2,50', 'Common stock', '08:53', 'FR0000120271', 45.955, '2018-03-26', 45.955, 1, 'TOTB', 'EUR', 2505166, 45.955, 352, 45.955),\n",
       " ('AURUBIS AG', 'Common stock', '08:59', 'DE0006766504', 66.8, '2018-03-26', 66.8, 1, 'NDA', 'EUR', 2505060, 66.8, 59, 66.8),\n",
       " ('COMDIRECT BANK AG', 'Common stock', '08:36', 'DE0005428007', 11.6, '2018-03-26', 11.6, 3, 'COM', 'EUR', 2504931, 11.6, 389, 11.6)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use Presto to query the NoSQL table that was created in the previous step\n",
    "presto_nosql_table_path = os.path.join('v3io.users.\"' + os.getenv('V3IO_USERNAME') + '/examples/stocks_example_tab\"')\n",
    "%sql select * from $presto_nosql_table_path limit 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example\"></a>\n",
    "## Data Collection and Exploration Getting-Started Example\n",
    "\n",
    "This section demonstrates a data collection, ingestion, and exploration flow.\n",
    "Follow the tutorial by running the code cells in order of appearance:\n",
    "- [Step #1](#getting-started-example-step-ingest-csv) &mdash; a CSV file is read from an Amazon S3 bucket and saved into an examples data-container directory using curl.\n",
    "  The examples directory is first created by using a file-system command.\n",
    "- [Step #2](#getting-started-example-step-convert-csv-to-nosql-table) &mdash; the ingested file is converted into a NoSQL table by using Spark DataFrames.\n",
    "- [Step #3](#getting-started-example-step-run-sql-queries) &mdash; a Presto SQL query is run on the NoSQL table.\n",
    "- [Step #4](#getting-started-example-step-convert-data-to-parquet) &mdash; the ingested CSV file is converted into a Parquet table by using Spark DataFrames.\n",
    "- [Step #5](#getting-started-example-step-browse-the-examples-dir) &mdash; the examples container directory is browsed by using local and Hadoop file-system commands.\n",
    "- At the end of the flow, you can optionally [delete](#getting-started-example-deleting-data) the examples directory using a file-system command.\n",
    "\n",
    "You can find more information about this sample flow in the [Converting a CSV File to a NoSQL Table](https://www.iguazio.com/docs/tutorials/latest-release/getting-started/ingest-n-consume-files/#convert-csv-to-nosql) platform quick-start tutorial.\n",
    "\n",
    "> **Tip:** You can also browse the files and directories that you write to the \"users\" container in this tutorial from the platform dashboard: in the side navigation menu, select **Data**, and then select the **users** container from the table. On the container data page, select the **Browse** tab, and then use the side directory-navigation tree to browse the directories. Selecting a file or directory in the browse table displays its metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-ingest-csv\"></a>\n",
    "### Step 1: Ingest a sample CSV file from Amazon S3\n",
    "\n",
    "Use `curl` to download a sample stocks-data CSV file from the [Iguazio sample data-set](http://iguazio-sample-data.s3.amazonaws.com/) public Amazon S3 bucket.\n",
    "For additional public data sets, check out [Registry of Open Data on AWS](https://registry.opendata.aws/).\n",
    "\n",
    "> **NOTE:** All the platform tutorial notebook examples store the data in an **examples** directory in the running-user directory of the predefined \"users\" container &mdash; **users/&lt;running user&gt;/examples**.\n",
    "> The running-user directory is automatically created by the Jupyter Notebook service.\n",
    "> The `V3IO_HOME` environment variable is used to reference the **users/&lt;running user&gt;** directory.\n",
    "> To save the data to a different root container directory or to a different container, you need to specify the data path in the local file-system commands as `/v3io/<container name>/<data path>`, and in Spark DataFrames or Hadoop FS commands as a fully qualified path of the format `v3io://<container name>/<table path>`.\n",
    "> For more information, see the [v3io-mount](#v3io-mount) and [running-user directory](#running-user-dir) information [Jupyter Notebook Basics](#jupyter-notebook-basics) section of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  861k  100  861k    0     0   126k      0  0:00:06  0:00:06 --:--:--  200k\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "DIR_PATH=\"/User/examples/\" # <=> \"/v3io/${V3IO_HOME}/examples/\" or \"/v3io/users/${V3IO_USERNAME}/examples/\"\n",
    "CSV_PATH=\"${DIR_PATH}stocks.csv\"\n",
    "\n",
    "# Create the examples directory\n",
    "mkdir -p ${DIR_PATH}\n",
    "\n",
    "# Download a sample stocks CSV file from the Iguazio sample data-set Amazon S3 bucket to the examples directory\n",
    "curl -L \"iguazio-sample-data.s3.amazonaws.com/2018-03-26_BINS_XETR08.csv\" > ${CSV_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-convert-csv-to-nosql-table\"></a>\n",
    "### Step 2: Convert the sample CSV file to a NoSQL table\n",
    "\n",
    "Read the sample **stocks.csv** file that you downloaded and ingested in the previous step into a Spark DataFrame, and write the data in NoSQL format to a new \"stocks_tab\" table in the same container directory (**users/&lt;running user&gt;/examples/stocks_tab**). \n",
    "\n",
    "> **Note**\n",
    "> - To use the Iguazio Spark Connector, set the DataFrame's data-source format to `io.iguaz.v3io.spark.sql.kv`.\n",
    "> - The data path in the Spark DataFrame is specified by using the `V3IO_HOME_URL` environment variable, which is set to `v3io://users/<running user>`.\n",
    ">   See the [running-user directory](#running-user-dir) information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------+--------------------+------------+--------+----------+-------------------+-----+----------+--------+--------+--------+------------+--------------+\n",
      "|        ISIN|Mnemonic|        SecurityDesc|SecurityType|Currency|SecurityID|               Date| Time|StartPrice|MaxPrice|MinPrice|EndPrice|TradedVolume|NumberOfTrades|\n",
      "+------------+--------+--------------------+------------+--------+----------+-------------------+-----+----------+--------+--------+--------+------------+--------------+\n",
      "|CH0038389992|    BBZA|BB BIOTECH NAM.  ...|Common stock|     EUR|   2504244|2018-03-26 00:00:00|08:00|      56.4|    56.4|    56.4|    56.4|         320|             4|\n",
      "|CH0038863350|    NESR|NESTLE NAM.      ...|Common stock|     EUR|   2504245|2018-03-26 00:00:00|08:00|     63.04|   63.06|    63.0|   63.06|         314|             3|\n",
      "|LU0378438732|    C001|COMSTAGE-DAX UCIT...|         ETF|     EUR|   2504271|2018-03-26 00:00:00|08:00|    113.42|  113.42|  113.42|  113.42|         100|             1|\n",
      "|LU0411075020|    DBPD|XTR.SHORTDAX X2 D...|         ETF|     EUR|   2504272|2018-03-26 00:00:00|08:00|    4.1335|  4.1335|  4.1295|    4.13|      102993|             8|\n",
      "|LU0838782315|    XDDX|   XTR.DAX INCOME 1D|         ETF|     EUR|   2504277|2018-03-26 00:00:00|08:00|    105.14|   105.2|  105.14|   105.2|         239|             3|\n",
      "|DE000A0DJ6J9|     S92|SMA SOLAR TECHNOL.AG|Common stock|     EUR|   2504287|2018-03-26 00:00:00|08:00|     55.65|   55.65|   55.65|   55.65|         543|             3|\n",
      "|DE000A0D6554|    NDX1|      NORDEX SE O.N.|Common stock|     EUR|   2504290|2018-03-26 00:00:00|08:00|     8.182|    8.21|   8.174|    8.21|       10941|             8|\n",
      "|DE000A0F5UE8|    EXXU|IS.DJ CHINA OFFS....|         ETF|     EUR|   2504302|2018-03-26 00:00:00|08:00|     47.52|   47.52|   47.52|   47.52|         420|             1|\n",
      "|DE000A0HN5C6|    DWNI|DEUTSCHE WOHNEN S...|Common stock|     EUR|   2504314|2018-03-26 00:00:00|08:00|      36.2|   36.24|    36.2|   36.24|         580|             5|\n",
      "|DE000A0LD2U1|     AOX|ALSTRIA OFFICE RE...|Common stock|     EUR|   2504379|2018-03-26 00:00:00|08:00|     12.25|   12.25|   12.25|   12.25|        1728|             3|\n",
      "|DE000A0LR936|     ST5|           STEICO SE|Common stock|     EUR|   2504382|2018-03-26 00:00:00|08:00|     22.35|   22.35|   22.35|   22.35|         334|             1|\n",
      "|DE000A0MZ4B0|     DLX|DELIGNIT AG      ...|Common stock|     EUR|   2504390|2018-03-26 00:00:00|08:00|      10.3|    10.3|    10.3|    10.3|         850|             1|\n",
      "|DE000A0Q8NC8|    ETLX|ETFS DAXGL.G.MIN....|         ETF|     EUR|   2504397|2018-03-26 00:00:00|08:00|    17.844|  17.844|  17.838|  17.838|        3085|             5|\n",
      "|DE000A0V9YU8|    4RT3|ETFS COM.SEC.DZ08...|         ETC|     EUR|   2504421|2018-03-26 00:00:00|08:00|    5.8895|  5.8895|  5.8895|  5.8895|           0|             1|\n",
      "|DE000A0WMPJ6|    AIXA|  AIXTRON SE NA O.N.|Common stock|     EUR|   2504428|2018-03-26 00:00:00|08:00|      16.8|    16.8|   16.75|  16.755|        3329|             8|\n",
      "|DE000A0Z2XN6|     RIB|RIB SOFTWARE SE  ...|Common stock|     EUR|   2504436|2018-03-26 00:00:00|08:00|     24.66|   24.66|   24.52|   24.52|       11741|            29|\n",
      "|DE000A0Z2ZZ5|    FNTN|  FREENET AG NA O.N.|Common stock|     EUR|   2504438|2018-03-26 00:00:00|08:00|     24.41|   24.42|   24.41|   24.42|         695|             6|\n",
      "|DE000A1A6V48|     KSC|      KPS AG NA O.N.|Common stock|     EUR|   2504441|2018-03-26 00:00:00|08:00|      9.15|    9.15|    9.15|    9.15|          73|             1|\n",
      "|DE000A1DAHH0|     BNR| BRENNTAG AG NA O.N.|Common stock|     EUR|   2504453|2018-03-26 00:00:00|08:00|     48.14|   48.14|   48.14|   48.14|         185|             2|\n",
      "|DE000A1EWWW0|     ADS|   ADIDAS AG NA O.N.|Common stock|     EUR|   2504471|2018-03-26 00:00:00|08:00|     196.3|  196.35|   196.3|  196.35|         591|            12|\n",
      "+------------+--------+--------------------+------------+--------+----------+-------------------+-----+----------+--------+--------+--------+------------+--------------+\n",
      "only showing top 20 rows\n",
      "\n",
      "root\n",
      " |-- ISIN: string (nullable = true)\n",
      " |-- Mnemonic: string (nullable = true)\n",
      " |-- SecurityDesc: string (nullable = true)\n",
      " |-- SecurityType: string (nullable = true)\n",
      " |-- Currency: string (nullable = true)\n",
      " |-- SecurityID: integer (nullable = true)\n",
      " |-- Date: timestamp (nullable = true)\n",
      " |-- Time: string (nullable = true)\n",
      " |-- StartPrice: double (nullable = true)\n",
      " |-- MaxPrice: double (nullable = true)\n",
      " |-- MinPrice: double (nullable = true)\n",
      " |-- EndPrice: double (nullable = true)\n",
      " |-- TradedVolume: integer (nullable = true)\n",
      " |-- NumberOfTrades: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "# Example diretory path - a <running user>/examples directory in the \"users\" container\n",
    "dir_path = os.getenv('V3IO_HOME_URL') + '/examples/'\n",
    "# CSV file path\n",
    "csv_path = dir_path + \"stocks.csv\" \n",
    "# NoSQL table path\n",
    "nosql_table_path = dir_path + \"stocks_tab\"\n",
    "\n",
    "# Create a new Spark session\n",
    "spark = SparkSession.builder.appName(\"Iguazio data collection and exploration getting-started example\").getOrCreate()\n",
    "\n",
    "# Read the sample CSV file into a Spark DataFrame, and let Spark infer the schema of the data\n",
    "df = spark.read.option(\"header\", \"true\").csv(csv_path, inferSchema=\"true\")\n",
    "\n",
    "# Show the DataFrame data\n",
    "df.show()\n",
    "\n",
    "# Write the DataFrame data to a NoSQL table in a platform data container.\n",
    "# Define the \"ISIN\" column (attribute) as the table's primary key.\n",
    "df.write.format(\"io.iguaz.v3io.spark.sql.kv\").mode(\"append\").option(\"key\", \"ISIN\").option(\"allow-overwrite-schema\", \"true\").save(nosql_table_path)\n",
    "\n",
    "# Display the table schema:\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-run-sql-queries\"></a>\n",
    "### Step 3: Run interactive SQL queries\n",
    "\n",
    "Use the `%sql` Jupyter magic to run an SQL queries on the \"stocks_tab\" table that was created in the previous step.\n",
    "(The queries is processed using Presto.)\n",
    "The example runs a `SELECT` query that reads the first ten table items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " * presto://iguazio:***@presto-api-presto.default-tenant.app.dev34.lab.iguazeng.com:443/v3io?protocol=https\n",
      "Done.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <th>securitydesc</th>\n",
       "        <th>securitytype</th>\n",
       "        <th>time</th>\n",
       "        <th>isin</th>\n",
       "        <th>minprice</th>\n",
       "        <th>date</th>\n",
       "        <th>endprice</th>\n",
       "        <th>numberoftrades</th>\n",
       "        <th>mnemonic</th>\n",
       "        <th>currency</th>\n",
       "        <th>securityid</th>\n",
       "        <th>maxprice</th>\n",
       "        <th>tradedvolume</th>\n",
       "        <th>startprice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>SENVION S.A.     EUR -,01</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:05</td>\n",
       "        <td>LU1377527517</td>\n",
       "        <td>9.95</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>9.95</td>\n",
       "        <td>1</td>\n",
       "        <td>SEN</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2506162</td>\n",
       "        <td>9.95</td>\n",
       "        <td>11</td>\n",
       "        <td>9.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>AT+S AUSTR.T.+SYSTEMT.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:04</td>\n",
       "        <td>AT0000969985</td>\n",
       "        <td>22.3</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>22.4</td>\n",
       "        <td>3</td>\n",
       "        <td>AUS</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504191</td>\n",
       "        <td>22.4</td>\n",
       "        <td>761</td>\n",
       "        <td>22.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>LY.MSCI AL.CO.WO.UETF CEO</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:27</td>\n",
       "        <td>FR0011079466</td>\n",
       "        <td>220.65</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>220.65</td>\n",
       "        <td>1</td>\n",
       "        <td>LYY0</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505321</td>\n",
       "        <td>220.65</td>\n",
       "        <td>200</td>\n",
       "        <td>220.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ENVITEC BIOGAS O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:11</td>\n",
       "        <td>DE000A0MVLS8</td>\n",
       "        <td>7.25</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>7.25</td>\n",
       "        <td>1</td>\n",
       "        <td>ETG</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504388</td>\n",
       "        <td>7.25</td>\n",
       "        <td>45</td>\n",
       "        <td>7.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ZURICH INSUR.GR.NA.SF0,10</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:14</td>\n",
       "        <td>CH0011075394</td>\n",
       "        <td>259.1</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>259.1</td>\n",
       "        <td>1</td>\n",
       "        <td>ZFIN</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504215</td>\n",
       "        <td>259.1</td>\n",
       "        <td>45</td>\n",
       "        <td>259.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>HOCHTIEF AG</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:02</td>\n",
       "        <td>DE0006070006</td>\n",
       "        <td>147.9</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>147.9</td>\n",
       "        <td>1</td>\n",
       "        <td>HOT</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505009</td>\n",
       "        <td>147.9</td>\n",
       "        <td>13</td>\n",
       "        <td>147.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>X(IE)-GERM.MITTEL.MCAP 1D</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:55</td>\n",
       "        <td>IE00B9MRJJ36</td>\n",
       "        <td>24.665</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>24.665</td>\n",
       "        <td>1</td>\n",
       "        <td>XDGM</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505788</td>\n",
       "        <td>24.665</td>\n",
       "        <td>1102</td>\n",
       "        <td>24.665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ISHS IV-AUTO.+ROBOTIC.ETF</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:01</td>\n",
       "        <td>IE00BYZK4552</td>\n",
       "        <td>6.423</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>6.423</td>\n",
       "        <td>1</td>\n",
       "        <td>2B76</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505551</td>\n",
       "        <td>6.423</td>\n",
       "        <td>6</td>\n",
       "        <td>6.423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>A.SPRINGER SE VNA</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:03</td>\n",
       "        <td>DE0005501357</td>\n",
       "        <td>68.0</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>68.05</td>\n",
       "        <td>3</td>\n",
       "        <td>SPR</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504947</td>\n",
       "        <td>68.05</td>\n",
       "        <td>143</td>\n",
       "        <td>68.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>PANDORA A/S         DK 1</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:36</td>\n",
       "        <td>DK0060252690</td>\n",
       "        <td>87.0</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>87.0</td>\n",
       "        <td>1</td>\n",
       "        <td>3P7</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505139</td>\n",
       "        <td>87.0</td>\n",
       "        <td>9</td>\n",
       "        <td>87.0</td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "[('SENVION S.A.     EUR -,01', 'Common stock', '08:05', 'LU1377527517', 9.95, '2018-03-26 00:00:00.000', 9.95, 1, 'SEN', 'EUR', 2506162, 9.95, 11, 9.95),\n",
       " ('AT+S AUSTR.T.+SYSTEMT.', 'Common stock', '08:04', 'AT0000969985', 22.3, '2018-03-26 00:00:00.000', 22.4, 3, 'AUS', 'EUR', 2504191, 22.4, 761, 22.3),\n",
       " ('LY.MSCI AL.CO.WO.UETF CEO', 'ETF', '08:27', 'FR0011079466', 220.65, '2018-03-26 00:00:00.000', 220.65, 1, 'LYY0', 'EUR', 2505321, 220.65, 200, 220.65),\n",
       " ('ENVITEC BIOGAS O.N.', 'Common stock', '08:11', 'DE000A0MVLS8', 7.25, '2018-03-26 00:00:00.000', 7.25, 1, 'ETG', 'EUR', 2504388, 7.25, 45, 7.25),\n",
       " ('ZURICH INSUR.GR.NA.SF0,10', 'Common stock', '08:14', 'CH0011075394', 259.1, '2018-03-26 00:00:00.000', 259.1, 1, 'ZFIN', 'EUR', 2504215, 259.1, 45, 259.1),\n",
       " ('HOCHTIEF AG', 'Common stock', '08:02', 'DE0006070006', 147.9, '2018-03-26 00:00:00.000', 147.9, 1, 'HOT', 'EUR', 2505009, 147.9, 13, 147.9),\n",
       " ('X(IE)-GERM.MITTEL.MCAP 1D', 'ETF', '08:55', 'IE00B9MRJJ36', 24.665, '2018-03-26 00:00:00.000', 24.665, 1, 'XDGM', 'EUR', 2505788, 24.665, 1102, 24.665),\n",
       " ('ISHS IV-AUTO.+ROBOTIC.ETF', 'ETF', '08:01', 'IE00BYZK4552', 6.423, '2018-03-26 00:00:00.000', 6.423, 1, '2B76', 'EUR', 2505551, 6.423, 6, 6.423),\n",
       " ('A.SPRINGER SE VNA', 'Common stock', '08:03', 'DE0005501357', 68.0, '2018-03-26 00:00:00.000', 68.05, 3, 'SPR', 'EUR', 2504947, 68.05, 143, 68.0),\n",
       " ('PANDORA A/S         DK 1', 'Common stock', '08:36', 'DK0060252690', 87.0, '2018-03-26 00:00:00.000', 87.0, 1, '3P7', 'EUR', 2505139, 87.0, 9, 87.0)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "presto_nosql_table_path = os.path.join('v3io.users.\"'+os.getenv('V3IO_USERNAME')+'/examples/stocks_tab\"')\n",
    "%sql select * from $presto_nosql_table_path limit 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-convert-data-to-parquet\"></a>\n",
    "### Step 4: Convert the data to a Parquet table\n",
    "\n",
    "Use a Spark DataFrame `write` command to write the data in the Spark DaraFrame &mdash; which was created from the CSV file and used to create the NoSQL table in [Step 2](#getting-started-example-step-convert-csv-to-nosql-table) &mdash; to a new **users/&lt;running user&gt;/examples/stocks_prqt** Parquet table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the DataFrame data that was read from the CSV file in Step 2 to a Parquet table in a platform data container\n",
    "prqt_table_path = dir_path + \"stocks_prqt\"\n",
    "df.write.mode('overwrite').parquet(prqt_table_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-browse-the-examples-dir\"></a>\n",
    "### Step 5: Browse the example container directory\n",
    "\n",
    "Use a file-system bash-shell command to list the contents of the **users/&lt;running user&gt;/examples** data-container directory to which all the ingested data in the previous steps were saved.\n",
    "You should see in this directory the **stocks.csv** file, **stocks_tab** NoSQL table directory, and **stocks_prqt** Parquet table directory that you created in the previous steps.\n",
    "The following cells demonstrate how to issue the same command using the local file system and using Hadoop FS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 0\n",
      "drwxrwxr-x 2 iguazio iguazio      0 Jun 11 20:04 stocks_example_tab\n",
      "-rw-r--r-- 1 iguazio iguazio 882055 Jun 11 20:05 stocks.csv\n",
      "drwxrwxrwx 2 iguazio iguazio      0 Jun 11 20:05 stocks_tab\n",
      "drwxr-xr-x 2 iguazio iguazio      0 Jun 11 20:07 stocks_prqt\n"
     ]
    }
   ],
   "source": [
    "# List the contents of the users/<running user>/examples directory using a local file-system command\n",
    "!ls -lrt /User/examples\n",
    "# The following are equivalent commands that demonstrate different ways to reference your user home directory:\n",
    "#!ls -lrt /v3io/${V3IO_HOME}/examples\n",
    "#!ls -lrt /v3io/users/${V3IO_USERNAME}/examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 4 items\n",
      "-rw-r--r--   1 iguazio iguazio     882055 2019-06-11 20:05 v3io://users/iguazio/examples/stocks.csv\n",
      "drwxrwxr-x   - iguazio iguazio          0 2019-06-11 20:04 v3io://users/iguazio/examples/stocks_example_tab\n",
      "drwxr-xr-x   - iguazio iguazio          0 2019-06-11 20:07 v3io://users/iguazio/examples/stocks_prqt\n",
      "drwxrwxrwx   - iguazio iguazio          0 2019-06-11 20:05 v3io://users/iguazio/examples/stocks_tab\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19/06/11 20:07:14 INFO slf_4j.Slf4jLogger: Slf4jLogger started\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "# List the contents of the users/<running user>/examples directory using an Hadoop FS command\n",
    "hadoop fs -ls ${V3IO_HOME_URL}/examples\n",
    "# The following are equivalent commands that demonstrate different ways to reference your user home directory:\n",
    "#hadoop fs -ls v3io://${V3IO_HOME}/examples\n",
    "#hadoop fs -ls v3io://users/${V3IO_USERNAME}/examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-deleting-data\"></a>\n",
    "### Deleting Data\n",
    "\n",
    "When are you are done, you can select to delete the any of the directories or files that you created.\n",
    "See the instructions in the [Creating and Deleting Container Directories](https://www.iguazio.com/docs/tutorials/latest-release/getting-started/containers/#create-delete-container-dirs) tutorial.\n",
    "The following example uses a local file-system bash-shell command to delete the entire contents of the **users/&lt;running user&gt;/examples** directory that was created in this example, but not the directory itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the contents of the examples directory:\n",
    "!rm -rf /User/examples/*\n",
    "# You can also delete the examples directory iteself (and all its contents):\n",
    "#!rm -rf /User/examples/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-release-spark-resources\"></a>\n",
    "### Releasing Spark Resources\n",
    "\n",
    "When you're done, run the following command to stop your Spark session and release its computation and memory resources:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 2
}