T13b_Hadoop - Simple Hivemall query for Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "997252e6-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99725476-495a-11e8-a057-0242ac130002",
     "previous": null
    }
   },
   "source": [
    "# About: Simple Hivemall query for Test\n",
    "\n",
    "---\n",
    "\n",
    "Hivemallの動作確認として、 [a9a binary classification](https://github.com/myui/hivemall/wiki#a9a-binary-classification) で示されたLogistic Regressionの動作確認をしてみる。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99725476-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997255d4-495a-11e8-a057-0242ac130002",
     "previous": "997252e6-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "## *Operation Note*\n",
    "\n",
    "*This is a cell for your own recording.  ここに経緯を記述*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "997255d4-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972571e-495a-11e8-a057-0242ac130002",
     "previous": "99725476-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "# Notebookと環境のBinding\n",
    "\n",
    "Inventory中のgroup名でBind対象を指示する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "lc_cell_meme": {
     "current": "9972571e-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972585e-495a-11e8-a057-0242ac130002",
     "previous": "997255d4-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "target_group = 'hadoop_all_testcluster'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9972585e-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972599e-495a-11e8-a057-0242ac130002",
     "previous": "9972571e-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "# Datasetの準備\n",
    "\n",
    "https://github.com/myui/hivemall/wiki/a9a-binary-dataset に従い、DatasetをHDFSにコピーし、tableを作成する。\n",
    "\n",
    "http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html#a9a から、以下のファイルをダウンロードする。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "lc_cell_meme": {
     "current": "9972599e-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99725ade-495a-11e8-a057-0242ac130002",
     "previous": "9972585e-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/tmp/tmpmkdyRn'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tempfile\n",
    "work_dir = tempfile.mkdtemp()\n",
    "work_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "lc_cell_meme": {
     "current": "99725ade-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99725c1e-495a-11e8-a057-0242ac130002",
     "previous": "9972599e-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WORK_DIR=/tmp/tmpmkdyRn\n"
     ]
    }
   ],
   "source": [
    "%env WORK_DIR={work_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "lc_cell_meme": {
     "current": "99725c1e-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99725d5e-495a-11e8-a057-0242ac130002",
     "previous": "99725ade-495a-11e8-a057-0242ac130002"
    },
    "run_control": {
     "marked": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100 2275k  100 2275k    0     0  5954k      0 --:--:-- --:--:-- --:--:-- 5971k\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100 1137k  100 1137k    0     0  4090k      0 --:--:-- --:--:-- --:--:-- 4105k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cd ${WORK_DIR}\n",
    "curl -O http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/a9a\n",
    "curl -O http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/a9a.t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99725d5e-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99725e94-495a-11e8-a057-0242ac130002",
     "previous": "99725c1e-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "整形をおこなう。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "lc_cell_meme": {
     "current": "99725e94-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99725fd4-495a-11e8-a057-0242ac130002",
     "previous": "99725d5e-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   167  100   167    0     0    536      0 --:--:-- --:--:-- --:--:--   538\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cd ${WORK_DIR}\n",
    "curl -O https://raw.githubusercontent.com/myui/hivemall/master/resources/misc/conv.awk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "lc_cell_meme": {
     "current": "99725fd4-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972610a-495a-11e8-a057-0242ac130002",
     "previous": "99725e94-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd ${WORK_DIR}\n",
    "awk -f conv.awk a9a | sed -e \"s/+1/1/\" | sed -e \"s/-1/0/\" > a9a.train\n",
    "awk -f conv.awk a9a.t | sed -e \"s/+1/1/\" | sed -e \"s/-1/0/\" > a9a.test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9972610a-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972624a-495a-11e8-a057-0242ac130002",
     "previous": "99725fd4-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "整形したデータをHDFSにアップロードする。\n",
    "\n",
    "まずこのNotebook環境の一時ディレクトリから `hadoop_client` マシンにコピーしてから、 `hadoop fs` コマンドでコピーする。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "9972624a-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972679a-495a-11e8-a057-0242ac130002",
     "previous": "9972610a-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "hdfs_dataset_dir = '/dataset/a9a'\n",
    "hadoop_client_dir = '~/a9a'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9972679a-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99726c22-495a-11e8-a057-0242ac130002",
     "previous": "9972624a-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "このNotebook環境から `hadoop_client` マシンへのコピー・・・"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "lc_cell_meme": {
     "current": "99726c22-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99726fe2-495a-11e8-a057-0242ac130002",
     "previous": "9972679a-495a-11e8-a057-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"checksum\": \"0105fad83d99e5c97c199572ace5cb71fe477313\", \n",
      "    \"dest\": \"/home/ansible/a9a/a9a.train\", \n",
      "    \"gid\": 500, \n",
      "    \"group\": \"ansible\", \n",
      "    \"mode\": \"0664\", \n",
      "    \"owner\": \"ansible\", \n",
      "    \"path\": \"/home/ansible/a9a/a9a.train\", \n",
      "    \"size\": 2449013, \n",
      "    \"state\": \"file\", \n",
      "    \"uid\": 500\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"checksum\": \"21117ece0f1f42a6b0a551a5cdb45535025f2361\", \n",
      "    \"dest\": \"/home/ansible/a9a/a9a.test\", \n",
      "    \"gid\": 500, \n",
      "    \"group\": \"ansible\", \n",
      "    \"mode\": \"0664\", \n",
      "    \"owner\": \"ansible\", \n",
      "    \"path\": \"/home/ansible/a9a/a9a.test\", \n",
      "    \"size\": 1218646, \n",
      "    \"state\": \"file\", \n",
      "    \"uid\": 500\n",
      "}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -m copy -a 'src={work_dir}/a9a.train dest={hadoop_client_dir}/' hadoop_client -l {target_group}\n",
    "!ansible -m copy -a 'src={work_dir}/a9a.test dest={hadoop_client_dir}/' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99726fe2-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997271fe-495a-11e8-a057-0242ac130002",
     "previous": "99726c22-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "`hadoop fs` コマンドを実行する。\n",
    "\n",
    "以下、すでに `/dataset/a9a` が存在している場合はコメントを外して使用する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "lc_cell_meme": {
     "current": "997271fe-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997273e8-495a-11e8-a057-0242ac130002",
     "previous": "99726fe2-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "Deleted /dataset/a9a16/08/26 07:28:52 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "#!ansible -m shell -a \"chdir={hadoop_client_dir} \\\n",
    "#                      hadoop fs -rm -r {hdfs_dataset_dir}\" hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "lc_cell_meme": {
     "current": "997273e8-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997275c8-495a-11e8-a057-0242ac130002",
     "previous": "997271fe-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -m shell -a \"chdir={hadoop_client_dir} \\\n",
    "                      hadoop fs -mkdir -p {hdfs_dataset_dir}/train && \\\n",
    "                      hadoop fs -mkdir -p {hdfs_dataset_dir}/test && \\\n",
    "                      hadoop fs -copyFromLocal a9a.train {hdfs_dataset_dir}/train && \\\n",
    "                      hadoop fs -copyFromLocal a9a.test {hdfs_dataset_dir}/test\" hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "997275c8-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997277b2-495a-11e8-a057-0242ac130002",
     "previous": "997273e8-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "念のため、HDFSにファイルがアップロードされていることを確認しておく。\n",
    "\n",
    "以下の `dataset_dir/train`, `dataset_dir/test` それぞれのディレクトリに、 `a9a.train`, `a9a.test` が作成されていればOK。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "lc_cell_meme": {
     "current": "997277b2-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997279a6-495a-11e8-a057-0242ac130002",
     "previous": "997275c8-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "Found 1 items\n",
      "-rw-r--r--   3 ansible supergroup    2449013 2016-08-26 07:29 /dataset/a9a/train/a9a.train\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "Found 1 items\n",
      "-rw-r--r--   3 ansible supergroup    1218646 2016-08-26 07:29 /dataset/a9a/test/a9a.test\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -a 'hadoop fs -ls {hdfs_dataset_dir}/train' hadoop_client -l {target_group}\n",
    "!ansible -a 'hadoop fs -ls {hdfs_dataset_dir}/test' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "997279a6-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99727b40-495a-11e8-a057-0242ac130002",
     "previous": "997277b2-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "データベースおよびテーブルを作成する。まずはHiveのクエリを作成する。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99727b40-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99727cbc-495a-11e8-a057-0242ac130002",
     "previous": "997279a6-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "## データベースのdrop\n",
    "\n",
    "必要に応じてコメントを外して使用する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "lc_cell_meme": {
     "current": "99727cbc-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99727e38-495a-11e8-a057-0242ac130002",
     "previous": "99727b40-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "#%%writefile {work_dir}/drop.query\n",
    "#use a9a;\n",
    "#drop table a9atrain;\n",
    "#drop table a9atest;\n",
    "#drop table a9a_model1;\n",
    "#drop view a9a_predict1;\n",
    "#drop view a9a_submit1;\n",
    "#drop view a9atest_exploded;\n",
    "#drop database a9a;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "lc_cell_meme": {
     "current": "99727e38-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99727fbe-495a-11e8-a057-0242ac130002",
     "previous": "99727cbc-495a-11e8-a057-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#!ansible -m copy -a 'src={work_dir}/drop.query dest={hadoop_client_dir}/' hadoop_client -l {target_group}\n",
    "#!ansible -a 'hive -f {hadoop_client_dir}/drop.query' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99727fbe-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99728130-495a-11e8-a057-0242ac130002",
     "previous": "99727e38-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "## テーブルの作成\n",
    "\n",
    "テーブルを作成する。まずはHiveクエリの作成。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "lc_cell_meme": {
     "current": "99728130-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997282ac-495a-11e8-a057-0242ac130002",
     "previous": "99727fbe-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "create database a9a;\r\n",
      "use a9a;\r\n",
      "\r\n",
      "create external table a9atrain (\r\n",
      "  rowid int,\r\n",
      "  label float,\r\n",
      "  features ARRAY<STRING>\r\n",
      ") ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t' COLLECTION ITEMS TERMINATED BY \",\" STORED AS TEXTFILE LOCATION '/dataset/a9a/train';\r\n",
      "\r\n",
      "create external table a9atest (\r\n",
      "  rowid int, \r\n",
      "  label float,\r\n",
      "  features ARRAY<STRING>\r\n",
      ") ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t' COLLECTION ITEMS TERMINATED BY \",\" STORED AS TEXTFILE LOCATION '/dataset/a9a/test';\r\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "with open(os.path.join(work_dir, 'hive.query'), 'w') as f:\n",
    "    f.write('''\n",
    "create database a9a;\n",
    "use a9a;\n",
    "\n",
    "create external table a9atrain (\n",
    "  rowid int,\n",
    "  label float,\n",
    "  features ARRAY<STRING>\n",
    ") ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\\\t' COLLECTION ITEMS TERMINATED BY \",\" STORED AS TEXTFILE LOCATION '{hdfs_dataset_dir}/train';\n",
    "\n",
    "create external table a9atest (\n",
    "  rowid int, \n",
    "  label float,\n",
    "  features ARRAY<STRING>\n",
    ") ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\\\t' COLLECTION ITEMS TERMINATED BY \",\" STORED AS TEXTFILE LOCATION '{hdfs_dataset_dir}/test';\n",
    "'''.format(hdfs_dataset_dir=hdfs_dataset_dir))\n",
    "\n",
    "!cat {work_dir}/hive.query"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "997282ac-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972841e-495a-11e8-a057-0242ac130002",
     "previous": "99728130-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "クエリを実行する。\n",
    "\n",
    "hiveコマンド実行時に `WARNING: Use \"yarn jar\" to launch YARN applications.` なる警告が表示されるが、これは無視する。\n",
    "([Warning \"yarn jar\" instead of \"hadoop jar\" in hadoop 2.7.0](https://issues.apache.org/jira/browse/HIVE-10100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "lc_cell_meme": {
     "current": "9972841e-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99728586-495a-11e8-a057-0242ac130002",
     "previous": "997282ac-495a-11e8-a057-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;33mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"checksum\": \"b27b2b59425d8e81ae2f83486c31b61d9b293fe4\", \n",
      "    \"dest\": \"/home/ansible/a9a/hive.query\", \n",
      "    \"gid\": 500, \n",
      "    \"group\": \"ansible\", \n",
      "    \"md5sum\": \"9d398e1bdd9a6e994be68d1b796e23f1\", \n",
      "    \"mode\": \"0664\", \n",
      "    \"owner\": \"ansible\", \n",
      "    \"size\": 472, \n",
      "    \"src\": \"/home/ansible/.ansible/tmp/ansible-tmp-1472164627.77-216599425494495/source\", \n",
      "    \"state\": \"file\", \n",
      "    \"uid\": 500\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "WARNING: Use \"yarn jar\" to launch YARN applications.\n",
      "\n",
      "Logging initialized using configuration in file:/etc/hive/conf/hive-log4j.properties\n",
      "OK\n",
      "Time taken: 0.278 seconds\n",
      "OK\n",
      "Time taken: 0.012 seconds\n",
      "OK\n",
      "Time taken: 0.412 seconds\n",
      "OK\n",
      "Time taken: 0.056 seconds\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -m copy -a 'src={work_dir}/hive.query dest={hadoop_client_dir}/' hadoop_client -l {target_group}\n",
    "!ansible -a 'hive -f {hadoop_client_dir}/hive.query' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99728586-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997286f8-495a-11e8-a057-0242ac130002",
     "previous": "9972841e-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "これでDatasetの準備OK。データ数の確認をしておく。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "lc_cell_meme": {
     "current": "997286f8-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972886a-495a-11e8-a057-0242ac130002",
     "previous": "99728586-495a-11e8-a057-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "WARNING: Use \"yarn jar\" to launch YARN applications.\r\n",
      "\r\n",
      "Logging initialized using configuration in file:/etc/hive/conf/hive-log4j.properties\r\n",
      "OK\r\n",
      "Time taken: 0.056 seconds\r\n",
      "Query ID = ansible_20160826073958_ae1d093b-ce00-41b5-b42f-7d99aa02980f\r\n",
      "Total jobs = 1\r\n",
      "Launching Job 1 out of 1\r\n",
      "\r\n",
      "\r\n",
      "Status: Running (Executing on YARN cluster with App id application_1471559811619_0065)\r\n",
      "\r\n",
      "Map 1: -/-\tReducer 2: 0/1\t\r\n",
      "Map 1: 0/1\tReducer 2: 0/1\t\r\n",
      "Map 1: 0/1\tReducer 2: 0/1\t\r\n",
      "Map 1: 0(+1)/1\tReducer 2: 0/1\t\r\n",
      "Map 1: 1/1\tReducer 2: 0/1\t\r\n",
      "Map 1: 1/1\tReducer 2: 0(+1)/1\t\r\n",
      "Map 1: 1/1\tReducer 2: 1/1\t\r\n",
      "OK\r\n",
      "Time taken: 11.838 seconds, Fetched: 1 row(s)\r\n",
      "Query ID = ansible_20160826074010_bd0ecf02-a14a-4aa7-9e2e-c2092ca9ff27\r\n",
      "Total jobs = 1\r\n",
      "Launching Job 1 out of 1\r\n",
      "\r\n",
      "\r\n",
      "Status: Running (Executing on YARN cluster with App id application_1471559811619_0065)\r\n",
      "\r\n",
      "Map 1: 0(+1)/1\tReducer 2: 0/1\t\r\n",
      "Map 1: 1/1\tReducer 2: 0/1\t\r\n",
      "Map 1: 1/1\tReducer 2: 0(+1)/1\t\r\n",
      "Map 1: 1/1\tReducer 2: 1/1\t\r\n",
      "OK\r\n",
      "Time taken: 1.539 seconds, Fetched: 1 row(s)\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -m shell -a 'hive -e \"use a9a; select count(1) from a9atrain; select count(1) from a9atest;\" \\\n",
    "                      > {hadoop_client_dir}/count_rows.result' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "lc_cell_meme": {
     "current": "9972886a-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99728a40-495a-11e8-a057-0242ac130002",
     "previous": "997286f8-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "32561\r\n",
      "16281\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -a 'cat {hadoop_client_dir}/count_rows.result' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99728a40-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99728c0c-495a-11e8-a057-0242ac130002",
     "previous": "9972886a-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "出力結果に格納される2つの数値のうち、1つめが `a2atrain` の行数、2つめが `a2atest` の行数となる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "99728c0c-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99728d56-495a-11e8-a057-0242ac130002",
     "previous": "99728a40-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "a9atrain_count = 32561\n",
    "a9atest_count = 16281"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99728d56-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99728ea0-495a-11e8-a057-0242ac130002",
     "previous": "99728c0c-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "# training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "lc_cell_meme": {
     "current": "99728ea0-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99728fe0-495a-11e8-a057-0242ac130002",
     "previous": "99728d56-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /tmp/tmpmkdyRn/hive.query\n"
     ]
    }
   ],
   "source": [
    "%%writefile {work_dir}/hive.query\n",
    "use a9a;\n",
    "\n",
    "create table a9a_model1 \n",
    "as\n",
    "select \n",
    " cast(feature as int) as feature,\n",
    " avg(weight) as weight\n",
    "from \n",
    " (select \n",
    "     logress(addBias(features),label) as (feature,weight)\n",
    "  from \n",
    "     a9atrain\n",
    " ) t \n",
    "group by feature;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "lc_cell_meme": {
     "current": "99728fe0-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729120-495a-11e8-a057-0242ac130002",
     "previous": "99728ea0-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;33mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"checksum\": \"315c607f32e7ff6e0f4f96dfab918a6d2d2204c7\", \n",
      "    \"dest\": \"/home/ansible/a9a/hive.query\", \n",
      "    \"gid\": 500, \n",
      "    \"group\": \"ansible\", \n",
      "    \"md5sum\": \"3d3a53c1a199e264e53223ee4a529354\", \n",
      "    \"mode\": \"0664\", \n",
      "    \"owner\": \"ansible\", \n",
      "    \"size\": 228, \n",
      "    \"src\": \"/home/ansible/.ansible/tmp/ansible-tmp-1472164878.24-14894325444976/source\", \n",
      "    \"state\": \"file\", \n",
      "    \"uid\": 500\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "WARNING: Use \"yarn jar\" to launch YARN applications.\n",
      "\n",
      "Logging initialized using configuration in file:/etc/hive/conf/hive-log4j.properties\n",
      "OK\n",
      "Time taken: 0.072 seconds\n",
      "Query ID = ansible_20160826074146_d5e23fda-5749-41de-b6dc-e1f14aef575c\n",
      "Total jobs = 1\n",
      "Launching Job 1 out of 1\n",
      "\n",
      "\n",
      "Status: Running (Executing on YARN cluster with App id application_1471559811619_0066)\n",
      "\n",
      "Map 1: -/-\tReducer 2: 0/1\t\n",
      "Map 1: 0/1\tReducer 2: 0/1\t\n",
      "Map 1: 0/1\tReducer 2: 0/1\t\n",
      "Map 1: 0(+1)/1\tReducer 2: 0/1\t\n",
      "Map 1: 0(+1)/1\tReducer 2: 0/1\t\n",
      "Map 1: 1/1\tReducer 2: 0(+1)/1\t\n",
      "Map 1: 1/1\tReducer 2: 1/1\t\n",
      "Moving data to: hdfs://hdfs-cluster/user/ansible/warehouse/a9a.db/a9a_model1\n",
      "Table a9a.a9a_model1 stats: [numFiles=1, numRows=124, totalSize=2894, rawDataSize=2770]\n",
      "OK\n",
      "Time taken: 14.887 seconds\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -m copy -a 'src={work_dir}/hive.query dest={hadoop_client_dir}/' hadoop_client -l {target_group}\n",
    "!ansible -a 'hive -f {hadoop_client_dir}/hive.query' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99729120-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729256-495a-11e8-a057-0242ac130002",
     "previous": "99728fe0-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "# prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "lc_cell_meme": {
     "current": "99729256-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729396-495a-11e8-a057-0242ac130002",
     "previous": "99729120-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /tmp/tmpmkdyRn/hive.query\n"
     ]
    }
   ],
   "source": [
    "%%writefile {work_dir}/hive.query\n",
    "use a9a;\n",
    "\n",
    "create or replace view a9atest_exploded as\n",
    "select \n",
    "  rowid,\n",
    "  label,\n",
    "  extract_feature(feature) as feature,\n",
    "  extract_weight(feature) as value\n",
    "from \n",
    "  a9atest LATERAL VIEW explode(addBias(features)) t AS feature;\n",
    "\n",
    "create or replace view a9a_predict1 as\n",
    "select\n",
    "  t.rowid, \n",
    "  sigmoid(sum(m.weight * t.value)) as prob,\n",
    "  CAST((case when sigmoid(sum(m.weight * t.value)) >= 0.5 then 1.0 else 0.0 end) as FLOAT) as label\n",
    "from \n",
    "  a9atest_exploded t LEFT OUTER JOIN\n",
    "  a9a_model1 m ON (t.feature = m.feature)\n",
    "group by\n",
    "  t.rowid;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "lc_cell_meme": {
     "current": "99729396-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997294d6-495a-11e8-a057-0242ac130002",
     "previous": "99729256-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;33mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"checksum\": \"c8cb50ab53e4ee3e0db3daee48139ed888f58c45\", \n",
      "    \"dest\": \"/home/ansible/a9a/hive.query\", \n",
      "    \"gid\": 500, \n",
      "    \"group\": \"ansible\", \n",
      "    \"md5sum\": \"97b3bef77bda574b045f1af43c851aa1\", \n",
      "    \"mode\": \"0664\", \n",
      "    \"owner\": \"ansible\", \n",
      "    \"size\": 537, \n",
      "    \"src\": \"/home/ansible/.ansible/tmp/ansible-tmp-1472164924.24-47476934008382/source\", \n",
      "    \"state\": \"file\", \n",
      "    \"uid\": 500\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "WARNING: Use \"yarn jar\" to launch YARN applications.\n",
      "\n",
      "Logging initialized using configuration in file:/etc/hive/conf/hive-log4j.properties\n",
      "OK\n",
      "Time taken: 0.028 seconds\n",
      "OK\n",
      "Time taken: 1.285 seconds\n",
      "OK\n",
      "Time taken: 0.305 seconds\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -m copy -a 'src={work_dir}/hive.query dest={hadoop_client_dir}/' hadoop_client -l {target_group}\n",
    "!ansible -a 'hive -f {hadoop_client_dir}/hive.query' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "997294d6-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729616-495a-11e8-a057-0242ac130002",
     "previous": "99729396-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "# evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "lc_cell_meme": {
     "current": "99729616-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729756-495a-11e8-a057-0242ac130002",
     "previous": "997294d6-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "use a9a;\r\n",
      "\r\n",
      "create or replace view a9a_submit1 as\r\n",
      "select \r\n",
      "  t.label as actual, \r\n",
      "  pd.label as predicted, \r\n",
      "  pd.prob as probability\r\n",
      "from \r\n",
      "  a9atest t JOIN a9a_predict1 pd \r\n",
      "    on (t.rowid = pd.rowid);\r\n",
      "\r\n",
      "select count(1) / 16281 from a9a_submit1 \r\n",
      "where actual == predicted;"
     ]
    }
   ],
   "source": [
    "with open(os.path.join(work_dir, 'hive.query'), 'w') as f:\n",
    "    f.write('''use a9a;\n",
    "\n",
    "create or replace view a9a_submit1 as\n",
    "select \n",
    "  t.label as actual, \n",
    "  pd.label as predicted, \n",
    "  pd.prob as probability\n",
    "from \n",
    "  a9atest t JOIN a9a_predict1 pd \n",
    "    on (t.rowid = pd.rowid);\n",
    "\n",
    "select count(1) / {num_test_instances} from a9a_submit1 \n",
    "where actual == predicted;'''.format(num_test_instances=a9atest_count))\n",
    "    \n",
    "!cat {work_dir}/hive.query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "lc_cell_meme": {
     "current": "99729756-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729896-495a-11e8-a057-0242ac130002",
     "previous": "99729616-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;33mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"checksum\": \"b9205dbd578f59c6f89d4fe5c6557bc6735f3417\", \n",
      "    \"dest\": \"/home/ansible/a9a/hive.query\", \n",
      "    \"gid\": 500, \n",
      "    \"group\": \"ansible\", \n",
      "    \"md5sum\": \"2df1d172658f1321e825cced32b7b45a\", \n",
      "    \"mode\": \"0664\", \n",
      "    \"owner\": \"ansible\", \n",
      "    \"size\": 267, \n",
      "    \"src\": \"/home/ansible/.ansible/tmp/ansible-tmp-1472164957.98-160344252851358/source\", \n",
      "    \"state\": \"file\", \n",
      "    \"uid\": 500\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "0.845771144278607WARNING: Use \"yarn jar\" to launch YARN applications.\n",
      "\n",
      "Logging initialized using configuration in file:/etc/hive/conf/hive-log4j.properties\n",
      "OK\n",
      "Time taken: 0.05 seconds\n",
      "OK\n",
      "Time taken: 1.491 seconds\n",
      "Query ID = ansible_20160826074309_1848469a-dcd0-41ee-af32-8f0e51517965\n",
      "Total jobs = 1\n",
      "Launching Job 1 out of 1\n",
      "\n",
      "\n",
      "Status: Running (Executing on YARN cluster with App id application_1471559811619_0068)\n",
      "\n",
      "Map 1: -/-\tMap 4: -/-\tMap 5: -/-\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 0/1\tMap 4: 0/1\tMap 5: 0/1\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 0/1\tMap 4: 0/1\tMap 5: 0/1\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 0/1\tMap 4: 0/1\tMap 5: 0(+1)/1\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 0/1\tMap 4: 0(+1)/1\tMap 5: 0(+1)/1\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 0/1\tMap 4: 1/1\tMap 5: 0(+1)/1\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 0(+1)/1\tMap 4: 1/1\tMap 5: 0(+1)/1\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 0(+1)/1\tMap 4: 1/1\tMap 5: 1/1\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 0(+1)/1\tMap 4: 1/1\tMap 5: 1/1\tReducer 2: 0/1\tReducer 3: 0/1\t\n",
      "Map 1: 1/1\tMap 4: 1/1\tMap 5: 1/1\tReducer 2: 0(+1)/1\tReducer 3: 0/1\t\n",
      "Map 1: 1/1\tMap 4: 1/1\tMap 5: 1/1\tReducer 2: 1/1\tReducer 3: 0/1\t\n",
      "Map 1: 1/1\tMap 4: 1/1\tMap 5: 1/1\tReducer 2: 1/1\tReducer 3: 0(+1)/1\t\n",
      "Map 1: 1/1\tMap 4: 1/1\tMap 5: 1/1\tReducer 2: 1/1\tReducer 3: 0/1\t\n",
      "Map 1: 1/1\tMap 4: 1/1\tMap 5: 1/1\tReducer 2: 1/1\tReducer 3: 1/1\t\n",
      "OK\n",
      "Time taken: 21.007 seconds, Fetched: 1 row(s)\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -m copy -a 'src={work_dir}/hive.query dest={hadoop_client_dir}/' hadoop_client -l {target_group}\n",
    "!ansible -a 'hive -f {hadoop_client_dir}/hive.query' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99729896-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "997299cc-495a-11e8-a057-0242ac130002",
     "previous": "99729756-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "# 実行経過の確認"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "997299cc-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729b02-495a-11e8-a057-0242ac130002",
     "previous": "99729896-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "## ログの確認\n",
    "\n",
    "ログは、 `yarn logs` コマンドにより確認することができる。この際、Application IDが必要。この情報はログから確認することができる。\n",
    "\n",
    "> `Status: Running (Executing on YARN cluster with App id application_1471559811619_0068)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "lc_cell_meme": {
     "current": "99729b02-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729c42-495a-11e8-a057-0242ac130002",
     "previous": "997299cc-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "yarn_application_id = 'application_1471559811619_0068'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99729c42-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729d78-495a-11e8-a057-0242ac130002",
     "previous": "99729b02-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "## Swimlanesによる可視化\n",
    "\n",
    "SwimlaneインストールNotebookによりTez-toolsをインストールしていれば、Swimlaneにより可視化することもできる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "lc_cell_meme": {
     "current": "99729d78-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729eae-495a-11e8-a057-0242ac130002",
     "previous": "99729c42-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /tmp/tmpmkdyRn/swimlane.yml\n"
     ]
    }
   ],
   "source": [
    "%%writefile {work_dir}/swimlane.yml\n",
    "- hosts: hadoop_swimlane\n",
    "  tasks:\n",
    "    - name: delete_swimlane_old_output\n",
    "      become: yes\n",
    "      file: path={{ swimlane_install_path }}/apache-tez-{{ swimlane_tez_version }}-src/tez-tools/swimlanes/{{ yarn_application_id }}.svg state=absent\n",
    "  \n",
    "    - name: create_swimlane_analyze\n",
    "      shell: bash yarn-swimlanes.sh {{ yarn_application_id }}\n",
    "      args:\n",
    "        chdir: \"{{ swimlane_install_path }}/apache-tez-{{ swimlane_tez_version }}-src/tez-tools/swimlanes/\"\n",
    "  \n",
    "    - name: download_svg\n",
    "      fetch: src={{ swimlane_install_path }}/apache-tez-{{ swimlane_tez_version }}-src/tez-tools/swimlanes/{{ yarn_application_id }}.svg dest={{ local_svg_path }} flat=yes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99729eae-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "99729fee-495a-11e8-a057-0242ac130002",
     "previous": "99729d78-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "SVGは一時ディレクトリにダウンロードし、Notebookにレンダリングすることにする。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "99729fee-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972a124-495a-11e8-a057-0242ac130002",
     "previous": "99729eae-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "local_svg_path = os.path.join(work_dir, 'result.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9972a124-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972a264-495a-11e8-a057-0242ac130002",
     "previous": "99729fee-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "swimlaneのスクリプトを実行し、SVGファイルをダウンロードする。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "lc_cell_meme": {
     "current": "9972a264-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972a39a-495a-11e8-a057-0242ac130002",
     "previous": "9972a124-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_swimlane] *********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [delete_swimlane_old_output] **********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [create_swimlane_analyze] *************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [download_svg] ************************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m4\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m2\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "extra_vars = ['-e yarn_application_id={}'.format(yarn_application_id), '-e local_svg_path={}'.format(local_svg_path)]\n",
    "!ansible-playbook {' '.join(extra_vars)} {work_dir}/swimlane.yml -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9972a39a-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972a4d0-495a-11e8-a057-0242ac130002",
     "previous": "9972a264-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "ダウンロードしたSVGファイルをレンダリングする。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "lc_cell_meme": {
     "current": "9972a4d0-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972a606-495a-11e8-a057-0242ac130002",
     "previous": "9972a39a-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"304\" version=\"1.1\" width=\"2581\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "\t\t<script type=\"text/ecmascript\" xlink:href=\"http://code.jquery.com/jquery-2.1.1.min.js\"/>\n",
       "\t\t<text style=\"font-size: 32px; text-anchor: middle\" transform=\"\" x=\"1062\" y=\"32\">appattempt_1471559811619_0068_000001</text><text style=\"text-anchor:end; font-size: 16px;\" transform=\"\" x=\"84\" y=\"96\">Container ID</text><text style=\"text-anchor:end; font-size: 16px;\" transform=\"\" x=\"96\" y=\"152\">container_e01_1471559811619_0068_01_000003</text><line style=\"stroke: #ccc\" x1=\"100\" x2=\"2225\" y1=\"152\" y2=\"152\"/><text style=\"text-anchor:end; font-size: 16px;\" transform=\"\" x=\"96\" y=\"176\">container_e01_1471559811619_0068_01_000004</text><line style=\"stroke: #ccc\" x1=\"100\" x2=\"2225\" y1=\"176\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"100\" y=\"116\">0.00 s</text><line style=\"stroke: #ddd\" x1=\"100\" x2=\"100\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"230\" y=\"116\">1.00 s</text><line style=\"stroke: #ddd\" x1=\"230\" x2=\"230\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"360\" y=\"116\">3.00 s</text><line style=\"stroke: #ddd\" x1=\"360\" x2=\"360\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"490\" y=\"116\">5.00 s</text><line style=\"stroke: #ddd\" x1=\"490\" x2=\"490\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"620\" y=\"116\">6.00 s</text><line style=\"stroke: #ddd\" x1=\"620\" x2=\"620\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"750\" y=\"116\">8.00 s</text><line style=\"stroke: #ddd\" x1=\"750\" x2=\"750\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"880\" y=\"116\">10.00 s</text><line style=\"stroke: #ddd\" x1=\"880\" x2=\"880\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"2225\" y=\"116\">27.00 s</text><line style=\"stroke: #ddd\" x1=\"2225\" x2=\"2225\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1010\" y=\"116\">11.00 s</text><line style=\"stroke: #ddd\" x1=\"1010\" x2=\"1010\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1140\" y=\"116\">13.00 s</text><line style=\"stroke: #ddd\" x1=\"1140\" x2=\"1140\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1270\" y=\"116\">15.00 s</text><line style=\"stroke: #ddd\" x1=\"1270\" x2=\"1270\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1400\" y=\"116\">16.00 s</text><line style=\"stroke: #ddd\" x1=\"1400\" x2=\"1400\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1530\" y=\"116\">18.00 s</text><line style=\"stroke: #ddd\" x1=\"1530\" x2=\"1530\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1660\" y=\"116\">20.00 s</text><line style=\"stroke: #ddd\" x1=\"1660\" x2=\"1660\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1790\" y=\"116\">21.00 s</text><line style=\"stroke: #ddd\" x1=\"1790\" x2=\"1790\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"1920\" y=\"116\">23.00 s</text><line style=\"stroke: #ddd\" x1=\"1920\" x2=\"1920\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"2050\" y=\"116\">25.00 s</text><line style=\"stroke: #ddd\" x1=\"2050\" x2=\"2050\" y1=\"116\" y2=\"176\"/><text style=\"text-anchor: middle; font-size: 12px\" transform=\"\" x=\"2180\" y=\"116\">27.00 s</text><line style=\"stroke: #ddd\" x1=\"2180\" x2=\"2180\" y1=\"116\" y2=\"176\"/><line style=\"stroke: #000\" x1=\"100\" x2=\"2225\" y1=\"128\" y2=\"128\"/><line style=\"stroke: #000\" x1=\"100\" x2=\"2225\" y1=\"176\" y2=\"176\"/><line style=\"stroke: #000\" x1=\"100\" x2=\"100\" y1=\"128\" y2=\"176\"/><line style=\"stroke: #000\" x1=\"2225\" x2=\"2225\" y1=\"128\" y2=\"176\"/><line style=\"stroke: green\" x1=\"1123\" x2=\"1123\" y1=\"152\" y2=\"176\"/><line style=\"stroke: green\" x1=\"2649\" x2=\"2649\" y1=\"152\" y2=\"176\"/><rect height=\"24\" style=\"fill: #ccc; opacity: 0.3\" width=\"1526\" x=\"1123\" y=\"152\"><title/></rect><line style=\"stroke: green\" x1=\"1123\" x2=\"1123\" y1=\"128\" y2=\"152\"/><line style=\"stroke: green\" x1=\"2649\" x2=\"2649\" y1=\"128\" y2=\"152\"/><rect height=\"24\" style=\"fill: #ccc; opacity: 0.3\" width=\"1526\" x=\"1123\" y=\"128\"><title/></rect><line stroke-dasharray=\"8,4\" style=\"stroke: black;\" x1=\"901\" x2=\"901\" y1=\"104\" y2=\"176\"/><line stroke-dasharray=\"8,4\" style=\"stroke: black;\" x1=\"2227\" x2=\"2227\" y1=\"104\" y2=\"176\"/><line style=\"stroke: black\" x1=\"901\" x2=\"2227\" y1=\"104\" y2=\"104\"/><text style=\"text-anchor: middle; font-size: 12px;\" transform=\"\" x=\"1564\" y=\"96\">dag_1471559811619_0068_1 (17.2 s)</text><rect height=\"22\" style=\"fill: #E2F2D8; stroke: #ccc;\" width=\"239\" x=\"1323\" y=\"153\"><title>attempt_1471559811619_0068_1_00_000000_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1442\" y=\"163\">Map 4 (00000_0)</text><rect height=\"22\" style=\"fill: #62C2A2; stroke: #ccc;\" width=\"316\" x=\"1300\" y=\"129\"><title>attempt_1471559811619_0068_1_01_000000_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1458\" y=\"139\">Map 5 (00000_0)</text><rect height=\"22\" style=\"fill: #E2F6E1; stroke: #ccc;\" width=\"361\" x=\"1584\" y=\"153\"><title>attempt_1471559811619_0068_1_02_000000_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"1764\" y=\"163\">Map 1 (00000_0)</text><rect height=\"22\" style=\"fill: #D8DAD7; stroke: #ccc;\" width=\"220\" x=\"1957\" y=\"129\"><title>attempt_1471559811619_0068_1_03_000000_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"2067\" y=\"139\">Reducer 2 (00000_0)</text><rect height=\"22\" style=\"fill: #A9DDB4; stroke: #ccc;\" width=\"45\" x=\"2180\" y=\"153\"><title>attempt_1471559811619_0068_1_04_000000_0</title></rect><text style=\"text-anchor: middle; font-size: 9px;\" transform=\"\" x=\"2202\" y=\"163\">Reducer 3</text></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename=local_svg_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9972a606-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972a796-495a-11e8-a057-0242ac130002",
     "previous": "9972a4d0-495a-11e8-a057-0242ac130002"
    }
   },
   "source": [
    "# 後始末\n",
    "\n",
    "一時ディレクトリを削除する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "9972a796-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": "9972a8f4-495a-11e8-a057-0242ac130002",
     "previous": "9972a606-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "!rm -fr {work_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "9972a8f4-495a-11e8-a057-0242ac130002",
     "history": [],
     "next": null,
     "previous": "9972a796-495a-11e8-a057-0242ac130002"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  },
  "lc_notebook_meme": {
   "current": "99725034-495a-11e8-a057-0242ac130002",
   "history": [],
   "root_cells": [
    "997252e6-495a-11e8-a057-0242ac130002",
    "99725476-495a-11e8-a057-0242ac130002",
    "997255d4-495a-11e8-a057-0242ac130002",
    "9972571e-495a-11e8-a057-0242ac130002",
    "9972585e-495a-11e8-a057-0242ac130002",
    "9972599e-495a-11e8-a057-0242ac130002",
    "99725ade-495a-11e8-a057-0242ac130002",
    "99725c1e-495a-11e8-a057-0242ac130002",
    "99725d5e-495a-11e8-a057-0242ac130002",
    "99725e94-495a-11e8-a057-0242ac130002",
    "99725fd4-495a-11e8-a057-0242ac130002",
    "9972610a-495a-11e8-a057-0242ac130002",
    "9972624a-495a-11e8-a057-0242ac130002",
    "9972679a-495a-11e8-a057-0242ac130002",
    "99726c22-495a-11e8-a057-0242ac130002",
    "99726fe2-495a-11e8-a057-0242ac130002",
    "997271fe-495a-11e8-a057-0242ac130002",
    "997273e8-495a-11e8-a057-0242ac130002",
    "997275c8-495a-11e8-a057-0242ac130002",
    "997277b2-495a-11e8-a057-0242ac130002",
    "997279a6-495a-11e8-a057-0242ac130002",
    "99727b40-495a-11e8-a057-0242ac130002",
    "99727cbc-495a-11e8-a057-0242ac130002",
    "99727e38-495a-11e8-a057-0242ac130002",
    "99727fbe-495a-11e8-a057-0242ac130002",
    "99728130-495a-11e8-a057-0242ac130002",
    "997282ac-495a-11e8-a057-0242ac130002",
    "9972841e-495a-11e8-a057-0242ac130002",
    "99728586-495a-11e8-a057-0242ac130002",
    "997286f8-495a-11e8-a057-0242ac130002",
    "9972886a-495a-11e8-a057-0242ac130002",
    "99728a40-495a-11e8-a057-0242ac130002",
    "99728c0c-495a-11e8-a057-0242ac130002",
    "99728d56-495a-11e8-a057-0242ac130002",
    "99728ea0-495a-11e8-a057-0242ac130002",
    "99728fe0-495a-11e8-a057-0242ac130002",
    "99729120-495a-11e8-a057-0242ac130002",
    "99729256-495a-11e8-a057-0242ac130002",
    "99729396-495a-11e8-a057-0242ac130002",
    "997294d6-495a-11e8-a057-0242ac130002",
    "99729616-495a-11e8-a057-0242ac130002",
    "99729756-495a-11e8-a057-0242ac130002",
    "99729896-495a-11e8-a057-0242ac130002",
    "997299cc-495a-11e8-a057-0242ac130002",
    "99729b02-495a-11e8-a057-0242ac130002",
    "99729c42-495a-11e8-a057-0242ac130002",
    "99729d78-495a-11e8-a057-0242ac130002",
    "99729eae-495a-11e8-a057-0242ac130002",
    "99729fee-495a-11e8-a057-0242ac130002",
    "9972a124-495a-11e8-a057-0242ac130002",
    "9972a264-495a-11e8-a057-0242ac130002",
    "9972a39a-495a-11e8-a057-0242ac130002",
    "9972a4d0-495a-11e8-a057-0242ac130002",
    "9972a606-495a-11e8-a057-0242ac130002",
    "9972a796-495a-11e8-a057-0242ac130002",
    "9972a8f4-495a-11e8-a057-0242ac130002"
   ]
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}