01_05_Diagnostics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "a6fba2a2-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fba2a2-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fba4f0-14ed-11e7-b31a-0242ac12000d",
       "previous": null
      }
     ],
     "next": "a6fba6e4-14ed-11e7-b31a-0242ac12000d",
     "previous": null
    }
   },
   "source": [
    "# Elastic Stackの構築：状態の診断\n",
    "----\n",
    "稼働しているElasticsearchに異常がないか調べ、 異常がある場合の対処手順について説明します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "lc_cell_meme": {
     "current": "a6fba6e4-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fba6e4-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fba8ce-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fba4f0-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a6fba8ce-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fba2a2-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## 準備\n",
    "本章のコマンドを実行するための設定を行います。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fba8ce-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbaaea-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fba6e4-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "セットアップ済みの環境のうち、コマンドを送る先であるCoordinating(Client) Nodeのホストアドレスを、次のセルに記入して実行し、保存してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbaaea-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbacc0-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fba8ce-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: ES_HOST=XXX.XXX.XXX.231:9200\n"
     ]
    }
   ],
   "source": [
    "%env ES_HOST=XXX.XXX.XXX.231:9200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbacc0-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbae8c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbaaea-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "診断の対象としたいインデックスを指定してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbae8c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbb058-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbacc0-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: INDEX=meteorological-data-*\n"
     ]
    }
   ],
   "source": [
    "%env INDEX=meteorological-data-*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbb058-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbb224-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbae8c-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "各サーバにAnsibleコマンドを発行するため、 Ansibleを実行可能なユーザアカウントとそのSSHキーを次のセルで設定してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbb224-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbb404-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbb058-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [],
   "source": [
    "#elasticsearchをインストールするサーバーでansibleコマンドを実行するユーザー\n",
    "USER='ansible'\n",
    "\n",
    "#公開鍵認証を行う場合の秘密鍵のパス\n",
    "#KEYPATH='/notebooks/etc/key/id_rsa'\n",
    "KEYPATH = '~/.ssh/ansible_id_rsa'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbb404-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbb5da-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbb224-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "※この後のコマンドは、[01_03_Set_Inventory.ipynb](01_03_Set_Inventory.ipynb) で生成したインベントリに対して発行されるように構築されています。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "lc_cell_meme": {
     "current": "a6fbb5da-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbb7a6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbb404-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## 診断\n",
    "現在の状態を把握するために、各種の状態を診断します。  \n",
    "本章の一連のコマンドを実行してください。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbb7a6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbb968-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbb5da-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "「■確認内容」で異常があると診断できた場合、次章の[症状と対処](#症状と対処)の内容に従い状態を正常化してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbb968-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbbb2a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbb7a6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### サーバマシンが稼働しているか？\n",
    "サーバごとにpingコマンドを実行し、サーバが稼働しているか確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbbb2a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbbcec-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbb968-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m ping -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbbcec-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbbeae-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbbb2a-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbbeae-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbc07a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbbcec-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "サーバごとに次のような応答があれば正常です。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbc07a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbc232-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbbeae-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "<各サーバのIPアドレス> | SUCCESS => {\n",
    "    \"changed\": false, \n",
    "    \"ping\": \"pong\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbc232-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbc3f4-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbc07a-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### プロセスが稼働しているか？\n",
    "サーバごとにpsコマンドを実行し、Elasticsearchプロセスが稼働しているか確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbc3f4-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbc5c0-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbc232-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n",
      "elastic+  9608     1  0 Apr04 ?        00:10:05 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n",
      "elastic+  8826     1  0 Apr04 ?        00:09:51 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n",
      "elastic+ 12091     1  0 Apr04 ?        00:12:50 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n",
      "elastic+ 30867     1  0 Apr04 ?        00:09:36 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n",
      "elastic+ 32010     1  0 Apr04 ?        00:11:24 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m shell -a \"ps -aef |grep Elasticsearch |grep -v grep\" -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbc5c0-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbc782-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbc3f4-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbc782-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbc944-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbc5c0-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Elastisearchが稼働しているはずのサーバごとに、次のコマンドのプロセスが存在していれば正常です。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbc944-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbcb06-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbc782-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "/bin/java ～～ org.elasticsearch.bootstrap.Elasticsearch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbcb06-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbccd2-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbc944-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### ディスクの使用量に余裕があるか？\n",
    "サーバごとにdfコマンドを実行し、ディスクサイズに余裕があるか確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbccd2-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbce94-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbcb06-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n",
      "Filesystem      1K-blocks    Used  Available Use% Mounted on\n",
      "/dev/sda2       480482404 2608712  453443492   1% /\n",
      "devtmpfs         49405296       0   49405296   0% /dev\n",
      "tmpfs            49413876       0   49413876   0% /dev/shm\n",
      "tmpfs            49413876   74352   49339524   1% /run\n",
      "tmpfs            49413876       0   49413876   0% /sys/fs/cgroup\n",
      "/dev/sda5      1928003752   68956 1829974880   1% /mnt\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n",
      "Filesystem      1K-blocks    Used  Available Use% Mounted on\n",
      "/dev/sda2       480482404 2608420  453443784   1% /\n",
      "devtmpfs         49405296       0   49405296   0% /dev\n",
      "tmpfs            49413876       0   49413876   0% /dev/shm\n",
      "tmpfs            49413876   74348   49339528   1% /run\n",
      "tmpfs            49413876       0   49413876   0% /sys/fs/cgroup\n",
      "/dev/sda5      1928003752   68956 1829974880   1% /mnt\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n",
      "Filesystem      1K-blocks    Used  Available Use% Mounted on\n",
      "/dev/sda2       480482404 2608476  453443728   1% /\n",
      "devtmpfs         49405296       0   49405296   0% /dev\n",
      "tmpfs            49413876       0   49413876   0% /dev/shm\n",
      "tmpfs            49413876   74348   49339528   1% /run\n",
      "tmpfs            49413876       0   49413876   0% /sys/fs/cgroup\n",
      "/dev/sda5      1928003752   68956 1829974880   1% /mnt\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n",
      "Filesystem      1K-blocks    Used  Available Use% Mounted on\n",
      "/dev/sda2       480482404 2608644  453443560   1% /\n",
      "devtmpfs         49405296       0   49405296   0% /dev\n",
      "tmpfs            49413876       0   49413876   0% /dev/shm\n",
      "tmpfs            49413876   66148   49347728   1% /run\n",
      "tmpfs            49413876       0   49413876   0% /sys/fs/cgroup\n",
      "/dev/sda5      1928003752   68956 1829974880   1% /mnt\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n",
      "Filesystem      1K-blocks    Used  Available Use% Mounted on\n",
      "/dev/sda2       480482404 3046048  453006156   1% /\n",
      "devtmpfs         49405296       0   49405296   0% /dev\n",
      "tmpfs            49413876       0   49413876   0% /dev/shm\n",
      "tmpfs            49413876   66148   49347728   1% /run\n",
      "tmpfs            49413876       0   49413876   0% /sys/fs/cgroup\n",
      "/dev/sda5      1928003752   68956 1829974880   1% /mnt\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m shell -a \"df\" -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbce94-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbd06a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbccd2-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbd06a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbd240-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbce94-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Elasticsearchのデフォルトの設定では、ディスク使用率が85%を越えると、そのノードに新たなシャードを割り当てません。そのため、**\"Use%\"**の列に出力されるディスク使用率が85%を越えないようにしてください。  \n",
    "ディスク使用率に余裕があれば問題ありません。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbd240-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbd40c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbd06a-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### クラスタと通信できるか？\n",
    "APIを発行し、Elasticsearchのクラウドと通信可能か確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbd40c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbd5ce-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbd240-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"name\" : \"LXj8U7-\",\r\n",
      "  \"cluster_name\" : \"es-cluster\",\r\n",
      "  \"cluster_uuid\" : \"4nC7EdwsRLeMCfKFaIiltA\",\r\n",
      "  \"version\" : {\r\n",
      "    \"number\" : \"5.0.0\",\r\n",
      "    \"build_hash\" : \"253032b\",\r\n",
      "    \"build_date\" : \"2016-10-26T04:37:51.531Z\",\r\n",
      "    \"build_snapshot\" : false,\r\n",
      "    \"lucene_version\" : \"6.2.0\"\r\n",
      "  },\r\n",
      "  \"tagline\" : \"You Know, for Search\"\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!curl -XGET \"http://$ES_HOST/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbd5ce-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbd790-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbd40c-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbd790-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbd95c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbd5ce-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "JSON形式の応答が表示されれば問題ありません。  \n",
    "表示されない場合は、クラスタの状態か、クラスタまでの通信経路に問題があります。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbd95c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbdb1e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbd790-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### クラスタに全Nodeが参加しているか？\n",
    "クラスタのヘルスチェック用のAPIを発行し、参加しているノードの状態等を確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbdb1e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbdce0-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbd95c-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"cluster_name\" : \"es-cluster\",\r\n",
      "  \"status\" : \"green\",\r\n",
      "  \"timed_out\" : false,\r\n",
      "  \"number_of_nodes\" : 5,\r\n",
      "  \"number_of_data_nodes\" : 5,\r\n",
      "  \"active_primary_shards\" : 0,\r\n",
      "  \"active_shards\" : 0,\r\n",
      "  \"relocating_shards\" : 0,\r\n",
      "  \"initializing_shards\" : 0,\r\n",
      "  \"unassigned_shards\" : 0,\r\n",
      "  \"delayed_unassigned_shards\" : 0,\r\n",
      "  \"number_of_pending_tasks\" : 0,\r\n",
      "  \"number_of_in_flight_fetch\" : 0,\r\n",
      "  \"task_max_waiting_in_queue_millis\" : 0,\r\n",
      "  \"active_shards_percent_as_number\" : 100.0\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!curl -XGET \"http://$ES_HOST/_cluster/health?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbdce0-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbdea2-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbdb1e-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbdea2-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbe078-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbdce0-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "出力結果から次の内容が確認できれば正常です。\n",
    "\n",
    "1. \"status\" が \"green\" かどうか？\n",
    "  - **green**  \n",
    "      ：正常です。\n",
    "  - **yellow**  \n",
    "    ：Primary Shardは割り当てられていますが、Replica Shardが割り当てられていません。\n",
    "  - **red**  \n",
    "    ：特定のShardがクラスタに割り当てられていません\n",
    "2. \"number_of_nodes\" の数が、起動しているはずの全Node数と一致しているか？\n",
    "3. \"number_of_data_nodes\"の数が、起動しているはずのData Node数と一致しているか？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbe078-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbe276-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbdea2-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### エラーログが無いか？\n",
    "各ElasticsearchプロセスがERRORメッセージをログに出力していないか確認します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbe276-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbe442-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbe078-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "ログはデフォルトでは次の場所に出力されています。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbe442-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbe60e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbe276-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "/var/log/elasticsearch/**{クラスタ名}**.log  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbe60e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbe7da-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbe442-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "その他のログも含め、ログディレクトリ配下のファイルについてERRORが出力がされていないかgrepします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbe7da-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbe99c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbe60e-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n",
      "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:12,497][ERROR][o.e.b.Bootstrap          ] Exception\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n",
      "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:12,476][ERROR][o.e.b.Bootstrap          ] Exception\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n",
      "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:12,690][ERROR][o.e.b.Bootstrap          ] Exception\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n",
      "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:13,381][ERROR][o.e.b.Bootstrap          ] Exception\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n",
      "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:13,712][ERROR][o.e.b.Bootstrap          ] Exception\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m shell -a 'grep ERROR /var/log/elasticsearch/*.log' -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbe99c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbeb54-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbe7da-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbeb54-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbed20-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbe99c-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "    XXX.XXX.XXX.111 | FAILED | rc=1 >>\n",
    "\n",
    "のように出力される場合は、grepした結果でERRORが0行ということなので、問題がありません。  \n",
    "1行以上、何かの出力がある場合は問題があります。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbed20-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbeef6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbeb54-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### Slowログが無いか？  \n",
    "各ElasticsearchプロセスがSlowログを出力していないか確認します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbeef6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbf0b8-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbed20-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Slowログはデフォルトでは次の場所に出力されています。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbf0b8-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbf27a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbeef6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "/var/log/elasticsearch/**{クラスタ名}**_index_indexing_slowlog.log  \n",
    "/var/log/elasticsearch/**{クラスタ名}**_index_search_slowlog.log  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbf27a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbf43c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbf0b8-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n",
      "\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m shell -a 'cat /var/log/elasticsearch/*slowlog.log' -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbf43c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbf644-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbf27a-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbf644-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbf81a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbf43c-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "    XXX.XXX.XXX.111 | SUCCESS | rc=0 >>\n",
    "\n",
    "のように出力される場合は、Slowログが0行ということなので、問題がありません。  \n",
    "1行以上、何かの出力がある場合は問題があります。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbf81a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbf9dc-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbf644-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### クラスタ内のサーバのCPU・メモリが枯渇してないか？\n",
    "CPU使用率、メモリ使用率が利用できる限界に近づいていないか確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbf9dc-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbfb9e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbf81a-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n",
      "top - 10:51:43 up 2 days, 17:22,  1 user,  load average: 0.00, 0.01, 0.05\n",
      "Tasks: 296 total,   1 running, 293 sleeping,   0 stopped,   2 zombie\n",
      "%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\n",
      "KiB Mem : 98827752 total, 62325280 free, 35258496 used,  1243976 buff/cache\n",
      "KiB Swap: 93750000 total, 93750000 free,        0 used. 63183308 avail Mem \n",
      "\n",
      "  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\n",
      "16115 ansible   20   0  130152   1784   1196 R   6.2  0.0   0:00.01 top\n",
      "    1 root      20   0   54304   4136   2480 S   0.0  0.0   0:03.72 systemd\n",
      "    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd\n",
      "    3 root      20   0       0      0      0 S   0.0  0.0   0:00.20 ksoftirqd/0\n",
      "    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.31 migration/0\n",
      "    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh\n",
      "   10 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0\n",
      "   11 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1\n",
      "   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2\n",
      "   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3\n",
      "   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/4\n",
      "   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/5\n",
      "   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/6\n",
      "   17 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/7\n",
      "   18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/8\n",
      "   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/9\n",
      "   20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/10\n",
      "   21 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/11\n",
      "   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/12\n",
      "   23 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/13\n",
      "   24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/14\n",
      "   25 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/15\n",
      "   26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/16\n",
      "   27 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/17\n",
      "   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/18\n",
      "   29 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/19\n",
      "   30 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/20\n",
      "   31 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/21\n",
      "   32 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/22\n",
      "   33 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/23\n",
      "   34 root      20   0       0      0      0 S   0.0  0.0   5:57.94 rcu_sched\n",
      "   35 root      20   0       0      0      0 S   0.0  0.0   0:04.33 rcuos/0\n",
      "   36 root      20   0       0      0      0 S   0.0  0.0   0:24.10 rcuos/1\n",
      "   37 root      20   0       0      0      0 S   0.0  0.0   0:11.13 rcuos/2\n",
      "   38 root      20   0       0      0      0 S   0.0  0.0   0:07.94 rcuos/3\n",
      "   39 root      20   0       0      0      0 S   0.0  0.0   0:05.56 rcuos/4\n",
      "   40 root      20   0       0      0      0 S   0.0  0.0   0:04.49 rcuos/5\n",
      "   41 root      20   0       0      0      0 S   0.0  0.0   0:05.39 rcuos/6\n",
      "   42 root      20   0       0      0      0 S   0.0  0.0   0:04.52 rcuos/7\n",
      "   43 root      20   0       0      0      0 S   0.0  0.0   0:08.40 rcuos/8\n",
      "   44 root      20   0       0      0      0 S   0.0  0.0   0:06.96 rcuos/9\n",
      "   45 root      20   0       0      0      0 S   0.0  0.0   1:28.10 rcuos/10\n",
      "   46 root      20   0       0      0      0 S   0.0  0.0   0:05.13 rcuos/11\n",
      "   47 root      20   0       0      0      0 S   0.0  0.0   0:01.53 rcuos/12\n",
      "   48 root      20   0       0      0      0 S   0.0  0.0   0:11.72 rcuos/13\n",
      "   49 root      20   0       0      0      0 S   0.0  0.0   0:02.49 rcuos/14\n",
      "   50 root      20   0       0      0      0 S   0.0  0.0   0:02.17 rcuos/15\n",
      "   51 root      20   0       0      0      0 S   0.0  0.0   0:01.81 rcuos/16\n",
      "   52 root      20   0       0      0      0 S   0.0  0.0   0:02.54 rcuos/17\n",
      "   53 root      20   0       0      0      0 S   0.0  0.0   0:02.11 rcuos/18\n",
      "   54 root      20   0       0      0      0 S   0.0  0.0   0:12.20 rcuos/19\n",
      "   55 root      20   0       0      0      0 S   0.0  0.0   0:03.22 rcuos/20\n",
      "   56 root      20   0       0      0      0 S   0.0  0.0   0:02.40 rcuos/21\n",
      "   57 root      20   0       0      0      0 S   0.0  0.0   0:09.34 rcuos/22\n",
      "   58 root      20   0       0      0      0 S   0.0  0.0   0:01.94 rcuos/23\n",
      "   59 root      rt   0       0      0      0 S   0.0  0.0   0:00.55 watchdog/0\n",
      "   60 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 watchdog/1\n",
      "   61 root      rt   0       0      0      0 S   0.0  0.0   0:00.21 migration/1\n",
      "   62 root      20   0       0      0      0 S   0.0  0.0   0:00.03 ksoftirqd/1\n",
      "   64 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      "   65 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 watchdog/2\n",
      "   66 root      rt   0       0      0      0 S   0.0  0.0   0:00.32 migration/2\n",
      "   67 root      20   0       0      0      0 S   0.0  0.0   0:00.22 ksoftirqd/2\n",
      "   69 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:+\n",
      "   70 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/3\n",
      "   71 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/3\n",
      "   72 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/3\n",
      "   74 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      "   75 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/4\n",
      "   76 root      rt   0       0      0      0 S   0.0  0.0   0:00.22 migration/4\n",
      "   77 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/4\n",
      "   79 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      "   80 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/5\n",
      "   81 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 migration/5\n",
      "   82 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/5\n",
      "   84 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      "   85 root      rt   0       0      0      0 S   0.0  0.0   0:00.52 watchdog/6\n",
      "   86 root      rt   0       0      0      0 S   0.0  0.0   0:00.32 migration/6\n",
      "   87 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/6\n",
      "   89 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/6:+\n",
      "   91 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/7\n",
      "   92 root      rt   0       0      0      0 S   0.0  0.0   0:00.31 migration/7\n",
      "   93 root      20   0       0      0      0 S   0.0  0.0   0:00.49 ksoftirqd/7\n",
      "   95 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/7:+\n",
      "   96 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/8\n",
      "   97 root      rt   0       0      0      0 S   0.0  0.0   0:00.23 migration/8\n",
      "   98 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/8\n",
      "  100 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/8:+\n",
      "  101 root      rt   0       0      0      0 S   0.0  0.0   0:00.52 watchdog/9\n",
      "  102 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/9\n",
      "  103 root      20   0       0      0      0 S   0.0  0.0   0:00.50 ksoftirqd/9\n",
      "  105 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/9:+\n",
      "  106 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/10\n",
      "  107 root      rt   0       0      0      0 S   0.0  0.0   0:00.21 migration/+\n",
      "  108 root      20   0       0      0      0 S   0.0  0.0   0:00.16 ksoftirqd/+\n",
      "  110 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      "  111 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/11\n",
      "  112 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/+\n",
      "  113 root      20   0       0      0      0 S   0.0  0.0   0:00.05 ksoftirqd/+\n",
      "  115 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "  116 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/12\n",
      "  117 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/+\n",
      "  118 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/+\n",
      "  120 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      "  121 root      rt   0       0      0      0 S   0.0  0.0   0:00.52 watchdog/13\n",
      "  122 root      rt   0       0      0      0 S   0.0  0.0   0:00.87 migration/+\n",
      "  123 root      20   0       0      0      0 S   0.0  0.0   0:00.36 ksoftirqd/+\n",
      "  125 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "  126 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/14\n",
      "  127 root      rt   0       0      0      0 S   0.0  0.0   0:00.43 migration/+\n",
      "  128 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  130 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "  131 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/15\n",
      "  132 root      rt   0       0      0      0 S   0.0  0.0   0:00.21 migration/+\n",
      "  133 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  135 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "  136 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/16\n",
      "  137 root      rt   0       0      0      0 S   0.0  0.0   0:00.26 migration/+\n",
      "  138 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  140 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/16+\n",
      "  141 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/17\n",
      "  142 root      rt   0       0      0      0 S   0.0  0.0   0:00.16 migration/+\n",
      "  143 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  145 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "  146 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/18\n",
      "  147 root      rt   0       0      0      0 S   0.0  0.0   0:00.28 migration/+\n",
      "  148 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  149 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "  150 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "  151 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/19\n",
      "  152 root      rt   0       0      0      0 S   0.0  0.0   0:00.75 migration/+\n",
      "  153 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/+\n",
      "  155 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "  156 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/20\n",
      "  157 root      rt   0       0      0      0 S   0.0  0.0   0:00.36 migration/+\n",
      "  158 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  160 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "  161 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/21\n",
      "  162 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/+\n",
      "  163 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  165 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "  166 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/22\n",
      "  167 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/+\n",
      "  168 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  170 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      "  171 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/23\n",
      "  172 root      rt   0       0      0      0 S   0.0  0.0   0:00.62 migration/+\n",
      "  173 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  174 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "  175 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "  176 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 khelper\n",
      "  177 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs\n",
      "  178 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns\n",
      "  179 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback\n",
      "  180 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd\n",
      "  181 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset\n",
      "  182 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd\n",
      "  184 root      20   0       0      0      0 S   0.0  0.0   0:00.00 khubd\n",
      "  185 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 md\n",
      "  187 root      20   0       0      0      0 S   0.0  0.0   0:00.22 khungtaskd\n",
      "  188 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd0\n",
      "  189 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd1\n",
      "  190 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd\n",
      "  191 root      39  19       0      0      0 S   0.0  0.0   0:01.90 khugepaged\n",
      "  192 root      20   0       0      0      0 S   0.0  0.0   0:00.00 fsnotify_m+\n",
      "  193 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto\n",
      "  201 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld\n",
      "  205 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kmpath_rda+\n",
      "  217 root      20   0       0      0      0 S   0.0  0.0   0:02.82 kworker/12+\n",
      "  220 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kworker/22+\n",
      "  223 root      20   0       0      0      0 S   0.0  0.0   0:00.61 kworker/18+\n",
      "  224 root      20   0       0      0      0 S   0.0  0.0   0:00.05 kworker/20+\n",
      "  227 root      20   0       0      0      0 S   0.0  0.0   0:00.60 kworker/21+\n",
      "  230 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kpsmoused\n",
      "  249 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 deferwq\n",
      "  279 root      20   0       0      0      0 S   0.0  0.0   0:00.32 kauditd\n",
      "  412 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_0\n",
      "  413 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_0\n",
      "  416 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ata_sff\n",
      "  427 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_1\n",
      "  428 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_1\n",
      "  429 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_2\n",
      "  430 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_2\n",
      "  431 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_3\n",
      "  432 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_3\n",
      "  433 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_4\n",
      "  434 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_4\n",
      "  435 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_5\n",
      "  436 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_5\n",
      "  437 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_6\n",
      "  438 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_6\n",
      "  441 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u5+\n",
      "  442 root      20   0       0      0      0 S   0.0  0.0   0:02.00 kworker/u5+\n",
      "  449 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u4+\n",
      "  510 root      20   0       0      0      0 S   0.0  0.0   0:02.61 jbd2/sda2-8\n",
      "  511 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  512 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  581 root      20   0  177560  94620  94284 S   0.0  0.1   0:37.21 systemd-jo+\n",
      "  600 root      20   0   42932   2360   1224 S   0.0  0.0   0:00.15 systemd-ud+\n",
      "  690 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 edac-poller\n",
      "  735 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kvm-irqfd-+\n",
      "  742 root      20   0       0      0      0 S   0.0  0.0   0:00.00 jbd2/sda5-8\n",
      "  743 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  744 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  758 root      16  -4  116684   1636   1240 S   0.0  0.0   0:01.23 auditd\n",
      "  781 root      20   0  339468  24192   6880 S   0.0  0.0   0:01.50 firewalld\n",
      "  784 root      20   0  550144  18188   5628 S   0.0  0.0   0:13.20 tuned\n",
      "  791 root      20   0   19284   1284    948 S   0.0  0.0   0:29.36 irqbalance\n",
      "  793 avahi     20   0   28072   1684   1392 S   0.0  0.0   0:00.30 avahi-daem+\n",
      "  796 root      20   0  328256  52280  49640 S   0.0  0.1   0:20.25 rsyslogd\n",
      "  797 root      20   0   34684   1720   1408 S   0.0  0.0   0:01.56 systemd-lo+\n",
      "  798 dbus      20   0   26708   1936   1344 S   0.0  0.0   0:02.85 dbus-daemon\n",
      "  800 root      20   0   25932    944    744 S   0.0  0.0   0:00.00 atd\n",
      "  802 root      20   0  126304   1600    984 S   0.0  0.0   0:00.70 crond\n",
      "  810 root      20   0  110004    836    712 S   0.0  0.0   0:00.00 agetty\n",
      "  812 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "  813 avahi     20   0   27944    444    216 S   0.0  0.0   0:00.00 avahi-daem+\n",
      "  814 ntp       20   0   31432   2072   1468 S   0.0  0.0   0:00.78 ntpd\n",
      "  908 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      "  982 root      20   0  454020   8944   6812 S   0.0  0.0   0:04.68 NetworkMan+\n",
      " 1040 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      " 1132 polkitd   20   0  514436  10564   4740 S   0.0  0.0   0:01.98 polkitd\n",
      " 1202 root       0 -20       0      0      0 S   0.0  0.0   0:00.08 kworker/2:+\n",
      " 1266 root      20   0       0      0      0 S   0.0  0.0   0:01.81 kworker/u4+\n",
      " 1680 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      " 1681 root      20   0  223228  11088   6412 S   0.0  0.0   2:33.06 snmpd\n",
      " 1684 root      20   0  358620  25916   4704 S   0.0  0.0   0:16.92 dodai-inst+\n",
      " 1781 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      " 1953 root      20   0       0      0      0 S   0.0  0.0   0:15.55 kworker/13+\n",
      " 2096 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      " 2138 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/9:0\n",
      " 2385 root      20   0  159408   9308   2144 S   0.0  0.0   0:08.25 munin-node\n",
      " 2920 root      20   0   91084   2172   1144 S   0.0  0.0   0:00.87 master\n",
      " 3008 postfix   20   0   91256   3900   2908 S   0.0  0.0   0:00.16 qmgr\n",
      " 3102 root      20   0       0      0      0 S   0.0  0.0   0:00.12 kworker/23+\n",
      " 3453 root      20   0       0      0      0 S   0.0  0.0   0:00.04 kworker/20+\n",
      " 3612 root      20   0       0      0      0 S   0.0  0.0   0:12.22 kworker/0:0\n",
      " 4617 root      20   0       0      0      0 S   0.0  0.0   0:07.18 kworker/4:2\n",
      " 4777 root      20   0       0      0      0 S   0.0  0.0   0:00.04 kworker/16+\n",
      " 4929 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:2\n",
      " 4930 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/5:0\n",
      " 5909 root      20   0       0      0      0 S   0.0  0.0   0:05.00 kworker/6:0\n",
      " 6210 root      20   0       0      0      0 S   0.0  0.0   1:00.66 kworker/u4+\n",
      " 6906 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0\n",
      " 8222 root      20   0       0      0      0 S   0.0  0.0   0:00.14 kworker/5:1\n",
      " 8688 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      " 8689 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/8:+\n",
      " 8690 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/9:+\n",
      " 8691 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      " 9760 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      " 9850 root      20   0       0      0      0 S   0.0  0.0   0:00.22 kworker/17+\n",
      "11346 root      20   0       0      0      0 S   0.0  0.0   0:00.21 kworker/14+\n",
      "11846 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "11875 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "12051 root      20   0       0      0      0 S   0.0  0.0   0:02.61 kworker/1:0\n",
      "12091 elastic+  20   0 44.131g 0.032t  15828 S   0.0 34.7  12:50.73 java\n",
      "12643 root      20   0       0      0      0 Z   0.0  0.0   0:00.00 sh\n",
      "12653 root      20   0       0      0      0 Z   0.0  0.0   0:00.00 sh\n",
      "12870 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/4:0\n",
      "13062 postfix   20   0   91188   3884   2888 S   0.0  0.0   0:00.00 pickup\n",
      "13238 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bond0\n",
      "13332 root      20   0       0      0      0 S   0.0  0.0   0:06.63 kworker/2:1\n",
      "13707 root      20   0       0      0      0 S   0.0  0.0   0:00.39 kworker/17+\n",
      "13869 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "14034 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/7:0\n",
      "14274 root      20   0       0      0      0 S   0.0  0.0   0:01.69 kworker/19+\n",
      "14634 root      20   0       0      0      0 S   0.0  0.0   0:02.98 kworker/22+\n",
      "15160 root      20   0       0      0      0 S   0.0  0.0   0:00.13 kworker/15+\n",
      "15520 root      20   0       0      0      0 S   0.0  0.0   0:00.26 kworker/16+\n",
      "15618 root      20   0  135216   4848   3680 S   0.0  0.0   0:00.03 sshd\n",
      "15620 ansible   20   0  135348   2612   1356 S   0.0  0.0   0:00.03 sshd\n",
      "16099 ansible   20   0  113116   1196   1020 S   0.0  0.0   0:00.00 sh\n",
      "16113 ansible   20   0  191232   7992   3584 S   0.0  0.0   0:00.04 python\n",
      "16114 ansible   20   0  208384  12716   3848 S   0.0  0.0   0:00.07 python\n",
      "16829 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "18171 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0\n",
      "19646 root      20   0       0      0      0 S   0.0  0.0   0:02.19 kworker/8:0\n",
      "19811 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      "20196 root      20   0   82508   3644   2788 S   0.0  0.0   0:00.01 sshd\n",
      "20609 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "20610 root      20   0       0      0      0 S   0.0  0.0   0:01.14 kworker/6:2\n",
      "21506 root      20   0       0      0      0 S   0.0  0.0   0:07.91 kworker/u4+\n",
      "22125 root      20   0       0      0      0 S   0.0  0.0   0:01.28 kworker/9:1\n",
      "22629 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:2\n",
      "23443 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "25416 root      20   0       0      0      0 S   0.0  0.0   0:04.88 kworker/10+\n",
      "25455 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "25503 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/6:+\n",
      "25523 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/7:+\n",
      "27375 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "28210 root      20   0       0      0      0 S   0.0  0.0   0:06.71 kworker/11+\n",
      "28255 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/8:1\n",
      "28583 root      20   0       0      0      0 S   0.0  0.0   0:04.10 kworker/3:1\n",
      "30560 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "31217 root      20   0       0      0      0 S   0.0  0.0   0:08.36 kworker/7:2\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n",
      "top - 10:51:43 up 2 days, 17:22,  1 user,  load average: 0.00, 0.01, 0.05\n",
      "Tasks: 297 total,   1 running, 294 sleeping,   0 stopped,   2 zombie\n",
      "%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\n",
      "KiB Mem : 98827752 total, 62652100 free, 34940596 used,  1235056 buff/cache\n",
      "KiB Swap: 93750000 total, 93750000 free,        0 used. 63501916 avail Mem \n",
      "\n",
      "  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\n",
      "13415 ansible   20   0  130152   1776   1196 R   6.2  0.0   0:00.01 top\n",
      "    1 root      20   0   54336   4124   2480 S   0.0  0.0   0:03.72 systemd\n",
      "    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd\n",
      "    3 root      20   0       0      0      0 S   0.0  0.0   0:00.19 ksoftirqd/0\n",
      "    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/0\n",
      "    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh\n",
      "   10 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0\n",
      "   11 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1\n",
      "   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2\n",
      "   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3\n",
      "   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/4\n",
      "   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/5\n",
      "   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/6\n",
      "   17 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/7\n",
      "   18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/8\n",
      "   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/9\n",
      "   20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/10\n",
      "   21 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/11\n",
      "   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/12\n",
      "   23 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/13\n",
      "   24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/14\n",
      "   25 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/15\n",
      "   26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/16\n",
      "   27 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/17\n",
      "   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/18\n",
      "   29 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/19\n",
      "   30 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/20\n",
      "   31 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/21\n",
      "   32 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/22\n",
      "   33 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/23\n",
      "   34 root      20   0       0      0      0 S   0.0  0.0   5:34.14 rcu_sched\n",
      "   35 root      20   0       0      0      0 S   0.0  0.0   0:05.56 rcuos/0\n",
      "   36 root      20   0       0      0      0 S   0.0  0.0   0:04.91 rcuos/1\n",
      "   37 root      20   0       0      0      0 S   0.0  0.0   0:06.63 rcuos/2\n",
      "   38 root      20   0       0      0      0 S   0.0  0.0   0:06.21 rcuos/3\n",
      "   39 root      20   0       0      0      0 S   0.0  0.0   0:04.79 rcuos/4\n",
      "   40 root      20   0       0      0      0 S   0.0  0.0   0:05.27 rcuos/5\n",
      "   41 root      20   0       0      0      0 S   0.0  0.0   0:05.95 rcuos/6\n",
      "   42 root      20   0       0      0      0 S   0.0  0.0   0:04.48 rcuos/7\n",
      "   43 root      20   0       0      0      0 S   0.0  0.0   0:09.22 rcuos/8\n",
      "   44 root      20   0       0      0      0 S   0.0  0.0   0:06.00 rcuos/9\n",
      "   45 root      20   0       0      0      0 S   0.0  0.0   0:32.02 rcuos/10\n",
      "   46 root      20   0       0      0      0 S   0.0  0.0   0:08.65 rcuos/11\n",
      "   47 root      20   0       0      0      0 S   0.0  0.0   1:11.71 rcuos/12\n",
      "   48 root      20   0       0      0      0 S   0.0  0.0   0:11.60 rcuos/13\n",
      "   49 root      20   0       0      0      0 S   0.0  0.0   0:02.62 rcuos/14\n",
      "   50 root      20   0       0      0      0 S   0.0  0.0   0:01.94 rcuos/15\n",
      "   51 root      20   0       0      0      0 S   0.0  0.0   0:02.68 rcuos/16\n",
      "   52 root      20   0       0      0      0 S   0.0  0.0   0:02.28 rcuos/17\n",
      "   53 root      20   0       0      0      0 S   0.0  0.0   0:01.67 rcuos/18\n",
      "   54 root      20   0       0      0      0 S   0.0  0.0   0:11.56 rcuos/19\n",
      "   55 root      20   0       0      0      0 S   0.0  0.0   0:02.69 rcuos/20\n",
      "   56 root      20   0       0      0      0 S   0.0  0.0   0:02.93 rcuos/21\n",
      "   57 root      20   0       0      0      0 S   0.0  0.0   0:02.77 rcuos/22\n",
      "   58 root      20   0       0      0      0 S   0.0  0.0   0:02.27 rcuos/23\n",
      "   59 root      rt   0       0      0      0 S   0.0  0.0   0:00.56 watchdog/0\n",
      "   60 root      rt   0       0      0      0 S   0.0  0.0   0:00.56 watchdog/1\n",
      "   61 root      rt   0       0      0      0 S   0.0  0.0   0:00.28 migration/1\n",
      "   62 root      20   0       0      0      0 S   0.0  0.0   0:00.46 ksoftirqd/1\n",
      "   64 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      "   65 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/2\n",
      "   66 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/2\n",
      "   67 root      20   0       0      0      0 S   0.0  0.0   0:00.15 ksoftirqd/2\n",
      "   69 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:+\n",
      "   70 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/3\n",
      "   71 root      rt   0       0      0      0 S   0.0  0.0   0:00.42 migration/3\n",
      "   72 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/3\n",
      "   74 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      "   75 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/4\n",
      "   76 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/4\n",
      "   77 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/4\n",
      "   79 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      "   80 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/5\n",
      "   81 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/5\n",
      "   82 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/5\n",
      "   84 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      "   85 root      rt   0       0      0      0 S   0.0  0.0   0:00.52 watchdog/6\n",
      "   86 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/6\n",
      "   87 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/6\n",
      "   89 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/6:+\n",
      "   90 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u5+\n",
      "   91 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/7\n",
      "   92 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 migration/7\n",
      "   93 root      20   0       0      0      0 S   0.0  0.0   0:00.49 ksoftirqd/7\n",
      "   95 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/7:+\n",
      "   96 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/8\n",
      "   97 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/8\n",
      "   98 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/8\n",
      "  100 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/8:+\n",
      "  101 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/9\n",
      "  102 root      rt   0       0      0      0 S   0.0  0.0   0:00.23 migration/9\n",
      "  103 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/9\n",
      "  105 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/9:+\n",
      "  106 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/10\n",
      "  107 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/+\n",
      "  108 root      20   0       0      0      0 S   0.0  0.0   0:00.39 ksoftirqd/+\n",
      "  110 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      "  111 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/11\n",
      "  112 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/+\n",
      "  113 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/+\n",
      "  115 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "  116 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/12\n",
      "  117 root      rt   0       0      0      0 S   0.0  0.0   0:00.22 migration/+\n",
      "  118 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/+\n",
      "  120 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      "  121 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/13\n",
      "  122 root      rt   0       0      0      0 S   0.0  0.0   0:01.00 migration/+\n",
      "  123 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  125 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "  126 root      rt   0       0      0      0 S   0.0  0.0   0:00.57 watchdog/14\n",
      "  127 root      rt   0       0      0      0 S   0.0  0.0   0:00.36 migration/+\n",
      "  128 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  130 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "  131 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/15\n",
      "  132 root      rt   0       0      0      0 S   0.0  0.0   0:00.22 migration/+\n",
      "  133 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  135 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "  136 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/16\n",
      "  137 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 migration/+\n",
      "  138 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  140 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/16+\n",
      "  141 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/17\n",
      "  142 root      rt   0       0      0      0 S   0.0  0.0   0:00.16 migration/+\n",
      "  143 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  145 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "  146 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/18\n",
      "  147 root      rt   0       0      0      0 S   0.0  0.0   0:00.21 migration/+\n",
      "  148 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  150 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "  151 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/19\n",
      "  152 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 migration/+\n",
      "  153 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/+\n",
      "  154 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "  155 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "  156 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/20\n",
      "  157 root      rt   0       0      0      0 S   0.0  0.0   0:00.37 migration/+\n",
      "  158 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  160 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "  161 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/21\n",
      "  162 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/+\n",
      "  163 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  165 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "  166 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/22\n",
      "  167 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 migration/+\n",
      "  168 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  170 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      "  171 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/23\n",
      "  172 root      rt   0       0      0      0 S   0.0  0.0   0:00.62 migration/+\n",
      "  173 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  175 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "  176 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 khelper\n",
      "  177 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs\n",
      "  178 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns\n",
      "  179 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback\n",
      "  180 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd\n",
      "  181 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset\n",
      "  182 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd\n",
      "  184 root      20   0       0      0      0 S   0.0  0.0   0:00.00 khubd\n",
      "  185 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 md\n",
      "  187 root      20   0       0      0      0 S   0.0  0.0   0:00.21 khungtaskd\n",
      "  188 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd0\n",
      "  189 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd1\n",
      "  190 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd\n",
      "  191 root      39  19       0      0      0 S   0.0  0.0   0:01.88 khugepaged\n",
      "  192 root      20   0       0      0      0 S   0.0  0.0   0:00.00 fsnotify_m+\n",
      "  193 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto\n",
      "  201 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld\n",
      "  205 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kmpath_rda+\n",
      "  214 root      20   0       0      0      0 S   0.0  0.0   0:01.83 kworker/19+\n",
      "  215 root      20   0       0      0      0 S   0.0  0.0   0:04.67 kworker/18+\n",
      "  218 root      20   0       0      0      0 S   0.0  0.0   0:00.88 kworker/17+\n",
      "  227 root      20   0       0      0      0 S   0.0  0.0   0:00.72 kworker/20+\n",
      "  229 root      20   0       0      0      0 S   0.0  0.0   0:00.17 kworker/23+\n",
      "  231 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kpsmoused\n",
      "  250 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 deferwq\n",
      "  280 root      20   0       0      0      0 S   0.0  0.0   0:00.34 kauditd\n",
      "  418 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_0\n",
      "  419 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_0\n",
      "  420 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ata_sff\n",
      "  431 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_1\n",
      "  432 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_1\n",
      "  433 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_2\n",
      "  434 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_2\n",
      "  436 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_3\n",
      "  437 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_3\n",
      "  438 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_4\n",
      "  439 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_4\n",
      "  441 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_5\n",
      "  442 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_5\n",
      "  443 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_6\n",
      "  444 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_6\n",
      "  449 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u4+\n",
      "  450 root      20   0       0      0      0 S   0.0  0.0   0:01.49 kworker/u4+\n",
      "  511 root      20   0       0      0      0 S   0.0  0.0   0:02.54 jbd2/sda2-8\n",
      "  512 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  513 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  581 root      20   0  177640  94228  93888 S   0.0  0.1   0:36.02 systemd-jo+\n",
      "  600 root      20   0   42932   2368   1224 S   0.0  0.0   0:00.15 systemd-ud+\n",
      "  666 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 edac-poller\n",
      "  732 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kvm-irqfd-+\n",
      "  741 root      20   0       0      0      0 S   0.0  0.0   0:00.00 jbd2/sda5-8\n",
      "  742 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  743 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  757 root      16  -4  116684   1632   1240 S   0.0  0.0   0:01.09 auditd\n",
      "  781 root      20   0  339472  24196   6896 S   0.0  0.0   0:01.44 firewalld\n",
      "  785 avahi     20   0   28072   1680   1392 S   0.0  0.0   0:00.31 avahi-daem+\n",
      "  786 root      20   0  550144  16148   5628 S   0.0  0.0   0:13.27 tuned\n",
      "  792 avahi     20   0   27944    440    216 S   0.0  0.0   0:00.00 avahi-daem+\n",
      "  794 root      20   0   19284   1284    948 S   0.0  0.0   0:28.98 irqbalance\n",
      "  799 root      20   0  394084  52168  49528 S   0.0  0.1   0:18.73 rsyslogd\n",
      "  800 dbus      20   0   26708   1936   1344 S   0.0  0.0   0:02.62 dbus-daemon\n",
      "  801 root      20   0   34684   1724   1408 S   0.0  0.0   0:01.54 systemd-lo+\n",
      "  804 root      20   0   25932    944    744 S   0.0  0.0   0:00.00 atd\n",
      "  805 root      20   0  126300   1596    980 S   0.0  0.0   0:00.70 crond\n",
      "  811 root      20   0  110004    836    712 S   0.0  0.0   0:00.00 agetty\n",
      "  813 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "  817 ntp       20   0   31432   2068   1464 S   0.0  0.0   0:00.76 ntpd\n",
      "  864 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/9:+\n",
      "  983 root      20   0  527860  11024   6812 S   0.0  0.0   0:04.61 NetworkMan+\n",
      "  994 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/10+\n",
      " 1022 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/11+\n",
      " 1043 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/6:+\n",
      " 1068 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/7:+\n",
      " 1079 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/8:+\n",
      " 1137 polkitd   20   0  514384  10392   4704 S   0.0  0.0   0:01.58 polkitd\n",
      " 1215 root      20   0       0      0      0 S   0.0  0.0   0:02.25 kworker/u5+\n",
      " 1233 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      " 1569 root      20   0       0      0      0 S   0.0  0.0   0:00.15 kworker/3:1\n",
      " 1686 root      20   0  223228  11080   6412 S   0.0  0.0   2:33.09 snmpd\n",
      " 1688 root      20   0  358616  25916   4704 S   0.0  0.0   0:16.89 dodai-inst+\n",
      " 1931 root      20   0       0      0      0 S   0.0  0.0   0:00.26 kworker/15+\n",
      " 1959 root      20   0       0      0      0 S   0.0  0.0   0:01.01 kworker/13+\n",
      " 2215 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      " 2499 root      20   0  159408   9308   2144 S   0.0  0.0   0:07.75 munin-node\n",
      " 2835 root      20   0   91084   2172   1144 S   0.0  0.0   0:00.81 master\n",
      " 2893 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/6:2\n",
      " 2909 postfix   20   0   91256   3896   2904 S   0.0  0.0   0:00.14 qmgr\n",
      " 2921 root      20   0       0      0      0 S   0.0  0.0   0:00.55 kworker/21+\n",
      " 3400 root      20   0       0      0      0 S   0.0  0.0   0:02.66 kworker/7:0\n",
      " 3555 root      20   0       0      0      0 S   0.0  0.0   0:15.34 kworker/10+\n",
      " 4214 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/9:0\n",
      " 4370 root      20   0       0      0      0 S   0.0  0.0   0:07.89 kworker/0:1\n",
      " 4719 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      " 5844 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      " 6191 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/8:2\n",
      " 6678 root      20   0       0      0      0 S   0.0  0.0   0:00.76 kworker/11+\n",
      " 7382 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/7:1\n",
      " 7878 root      20   0       0      0      0 S   0.0  0.0   0:01.80 kworker/5:1\n",
      " 8194 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0\n",
      " 8686 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      " 9512 root      20   0       0      0      0 S   0.0  0.0   0:00.04 kworker/8:0\n",
      " 9606 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      " 9608 elastic+  20   0 43.707g 0.032t  15840 S   0.0 34.3  10:05.08 java\n",
      "10173 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0\n",
      "10368 postfix   20   0   91188   3892   2888 S   0.0  0.0   0:00.01 pickup\n",
      "12642 root      20   0       0      0      0 Z   0.0  0.0   0:00.00 sh\n",
      "12652 root      20   0       0      0      0 Z   0.0  0.0   0:00.00 sh\n",
      "12808 root      20   0       0      0      0 S   0.0  0.0   0:00.55 kworker/6:0\n",
      "12918 root      20   0  135216   4848   3680 S   0.0  0.0   0:00.03 sshd\n",
      "12920 ansible   20   0  135348   2612   1356 S   0.0  0.0   0:00.03 sshd\n",
      "12963 root      20   0       0      0      0 S   0.0  0.0   0:00.98 kworker/2:1\n",
      "13237 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bond0\n",
      "13399 ansible   20   0  113116   1196   1020 S   0.0  0.0   0:00.00 sh\n",
      "13413 ansible   20   0  191232   7992   3584 S   0.0  0.0   0:00.04 python\n",
      "13414 ansible   20   0  208384  12720   3848 S   0.0  0.0   0:00.08 python\n",
      "13465 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/16+\n",
      "14600 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "14634 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "14768 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "15598 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/4:1\n",
      "15991 root      20   0       0      0      0 S   0.0  0.0   0:37.35 kworker/u4+\n",
      "18106 root      20   0       0      0      0 S   0.0  0.0   0:00.40 kworker/9:1\n",
      "19068 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0\n",
      "19723 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "19920 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "20196 root      20   0   82508   3648   2788 S   0.0  0.0   0:00.01 sshd\n",
      "20386 root      20   0       0      0      0 S   0.0  0.0   0:00.76 kworker/4:2\n",
      "21238 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "21791 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "22071 root      20   0       0      0      0 S   0.0  0.0   0:39.73 kworker/u4+\n",
      "22885 root      20   0       0      0      0 S   0.0  0.0   0:00.50 kworker/23+\n",
      "23405 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:2\n",
      "23729 root      20   0       0      0      0 S   0.0  0.0   0:08.91 kworker/1:2\n",
      "24155 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "24379 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      "24723 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "25379 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "26041 root      20   0       0      0      0 S   0.0  0.0   0:00.36 kworker/5:0\n",
      "27065 root       0 -20       0      0      0 S   0.0  0.0   0:00.05 kworker/2:+\n",
      "27430 root      20   0       0      0      0 S   0.0  0.0   0:00.74 kworker/14+\n",
      "27746 root      20   0       0      0      0 S   0.0  0.0   0:00.52 kworker/12+\n",
      "28698 root      20   0       0      0      0 S   0.0  0.0   0:00.68 kworker/22+\n",
      "28881 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "29495 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      "31835 root      20   0       0      0      0 S   0.0  0.0   0:00.62 kworker/16+\n",
      "32609 root      20   0       0      0      0 S   0.0  0.0   0:00.40 kworker/12+\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n",
      "top - 10:51:43 up 2 days, 17:21,  1 user,  load average: 0.02, 0.02, 0.05\n",
      "Tasks: 297 total,   1 running, 294 sleeping,   0 stopped,   2 zombie\n",
      "%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\n",
      "KiB Mem : 98827752 total, 62656732 free, 34935120 used,  1235900 buff/cache\n",
      "KiB Swap: 93750000 total, 93750000 free,        0 used. 63506656 avail Mem \n",
      "\n",
      "  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\n",
      "12549 ansible   20   0  130152   1780   1196 R   6.2  0.0   0:00.01 top\n",
      "    1 root      20   0   54296   4128   2480 S   0.0  0.0   0:03.54 systemd\n",
      "    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd\n",
      "    3 root      20   0       0      0      0 S   0.0  0.0   0:00.03 ksoftirqd/0\n",
      "    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "    7 root      20   0       0      0      0 S   0.0  0.0   0:02.54 kworker/u4+\n",
      "    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/0\n",
      "    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh\n",
      "   10 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0\n",
      "   11 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1\n",
      "   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2\n",
      "   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3\n",
      "   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/4\n",
      "   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/5\n",
      "   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/6\n",
      "   17 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/7\n",
      "   18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/8\n",
      "   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/9\n",
      "   20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/10\n",
      "   21 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/11\n",
      "   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/12\n",
      "   23 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/13\n",
      "   24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/14\n",
      "   25 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/15\n",
      "   26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/16\n",
      "   27 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/17\n",
      "   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/18\n",
      "   29 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/19\n",
      "   30 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/20\n",
      "   31 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/21\n",
      "   32 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/22\n",
      "   33 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/23\n",
      "   34 root      20   0       0      0      0 S   0.0  0.0   5:50.95 rcu_sched\n",
      "   35 root      20   0       0      0      0 S   0.0  0.0   0:03.64 rcuos/0\n",
      "   36 root      20   0       0      0      0 S   0.0  0.0   0:03.36 rcuos/1\n",
      "   37 root      20   0       0      0      0 S   0.0  0.0   0:07.10 rcuos/2\n",
      "   38 root      20   0       0      0      0 S   0.0  0.0   0:06.11 rcuos/3\n",
      "   39 root      20   0       0      0      0 S   0.0  0.0   0:05.20 rcuos/4\n",
      "   40 root      20   0       0      0      0 S   0.0  0.0   0:03.99 rcuos/5\n",
      "   41 root      20   0       0      0      0 S   0.0  0.0   0:04.15 rcuos/6\n",
      "   42 root      20   0       0      0      0 S   0.0  0.0   0:04.99 rcuos/7\n",
      "   43 root      20   0       0      0      0 S   0.0  0.0   0:08.37 rcuos/8\n",
      "   44 root      20   0       0      0      0 S   0.0  0.0   0:06.44 rcuos/9\n",
      "   45 root      20   0       0      0      0 S   0.0  0.0   0:04.80 rcuos/10\n",
      "   46 root      20   0       0      0      0 S   0.0  0.0   0:04.53 rcuos/11\n",
      "   47 root      20   0       0      0      0 S   0.0  0.0   0:02.51 rcuos/12\n",
      "   48 root      20   0       0      0      0 S   0.0  0.0   0:15.41 rcuos/13\n",
      "   49 root      20   0       0      0      0 S   0.0  0.0   0:24.33 rcuos/14\n",
      "   50 root      20   0       0      0      0 S   0.0  0.0   0:02.35 rcuos/15\n",
      "   51 root      20   0       0      0      0 S   0.0  0.0   0:01.85 rcuos/16\n",
      "   52 root      20   0       0      0      0 S   0.0  0.0   0:02.00 rcuos/17\n",
      "   53 root      20   0       0      0      0 S   0.0  0.0   0:32.03 rcuos/18\n",
      "   54 root      20   0       0      0      0 S   0.0  0.0   0:12.12 rcuos/19\n",
      "   55 root      20   0       0      0      0 S   0.0  0.0   0:02.61 rcuos/20\n",
      "   56 root      20   0       0      0      0 S   0.0  0.0   0:02.22 rcuos/21\n",
      "   57 root      20   0       0      0      0 S   0.0  0.0   0:56.09 rcuos/22\n",
      "   58 root      20   0       0      0      0 S   0.0  0.0   0:01.39 rcuos/23\n",
      "   59 root      rt   0       0      0      0 S   0.0  0.0   0:00.56 watchdog/0\n",
      "   60 root      rt   0       0      0      0 S   0.0  0.0   0:00.56 watchdog/1\n",
      "   61 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/1\n",
      "   62 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/1\n",
      "   64 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      "   65 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 watchdog/2\n",
      "   66 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/2\n",
      "   67 root      20   0       0      0      0 S   0.0  0.0   0:00.18 ksoftirqd/2\n",
      "   69 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:+\n",
      "   70 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/3\n",
      "   71 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/3\n",
      "   72 root      20   0       0      0      0 S   0.0  0.0   0:00.03 ksoftirqd/3\n",
      "   74 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      "   75 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/4\n",
      "   76 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/4\n",
      "   77 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/4\n",
      "   79 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      "   80 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/5\n",
      "   81 root      rt   0       0      0      0 S   0.0  0.0   0:00.34 migration/5\n",
      "   82 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/5\n",
      "   84 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      "   85 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/6\n",
      "   86 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/6\n",
      "   87 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/6\n",
      "   89 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/6:+\n",
      "   91 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/7\n",
      "   92 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 migration/7\n",
      "   93 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/7\n",
      "   95 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/7:+\n",
      "   96 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/8\n",
      "   97 root      rt   0       0      0      0 S   0.0  0.0   0:00.29 migration/8\n",
      "   98 root      20   0       0      0      0 S   0.0  0.0   0:00.50 ksoftirqd/8\n",
      "  100 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/8:+\n",
      "  101 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/9\n",
      "  102 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/9\n",
      "  103 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/9\n",
      "  105 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/9:+\n",
      "  106 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/10\n",
      "  107 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/+\n",
      "  108 root      20   0       0      0      0 S   0.0  0.0   0:00.52 ksoftirqd/+\n",
      "  110 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      "  111 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/11\n",
      "  112 root      rt   0       0      0      0 S   0.0  0.0   0:00.33 migration/+\n",
      "  113 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/+\n",
      "  115 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "  116 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/12\n",
      "  117 root      rt   0       0      0      0 S   0.0  0.0   0:00.30 migration/+\n",
      "  118 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  120 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      "  121 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/13\n",
      "  122 root      rt   0       0      0      0 S   0.0  0.0   0:00.76 migration/+\n",
      "  123 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  125 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "  126 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/14\n",
      "  127 root      rt   0       0      0      0 S   0.0  0.0   0:00.36 migration/+\n",
      "  128 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  130 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "  131 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/15\n",
      "  132 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/+\n",
      "  133 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  135 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "  136 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/16\n",
      "  137 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 migration/+\n",
      "  138 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  140 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/16+\n",
      "  141 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/17\n",
      "  142 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/+\n",
      "  143 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  145 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "  146 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/18\n",
      "  147 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 migration/+\n",
      "  148 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  150 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "  151 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 watchdog/19\n",
      "  152 root      rt   0       0      0      0 S   0.0  0.0   0:00.83 migration/+\n",
      "  153 root      20   0       0      0      0 S   0.0  0.0   0:00.19 ksoftirqd/+\n",
      "  155 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "  156 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/20\n",
      "  157 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 migration/+\n",
      "  158 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  160 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "  161 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/21\n",
      "  162 root      rt   0       0      0      0 S   0.0  0.0   0:00.28 migration/+\n",
      "  163 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  165 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "  166 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/22\n",
      "  167 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 migration/+\n",
      "  168 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/+\n",
      "  170 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      "  171 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/23\n",
      "  172 root      rt   0       0      0      0 S   0.0  0.0   0:00.58 migration/+\n",
      "  173 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  175 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "  176 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 khelper\n",
      "  177 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs\n",
      "  178 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns\n",
      "  179 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback\n",
      "  180 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd\n",
      "  181 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset\n",
      "  182 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd\n",
      "  184 root      20   0       0      0      0 S   0.0  0.0   0:00.00 khubd\n",
      "  185 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 md\n",
      "  187 root      20   0       0      0      0 S   0.0  0.0   0:00.22 khungtaskd\n",
      "  188 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd0\n",
      "  189 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd1\n",
      "  190 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd\n",
      "  191 root      39  19       0      0      0 S   0.0  0.0   0:01.71 khugepaged\n",
      "  192 root      20   0       0      0      0 S   0.0  0.0   0:00.00 fsnotify_m+\n",
      "  193 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto\n",
      "  201 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld\n",
      "  205 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kmpath_rda+\n",
      "  216 root      20   0       0      0      0 S   0.0  0.0   0:02.57 kworker/22+\n",
      "  218 root      20   0       0      0      0 S   0.0  0.0   0:00.41 kworker/16+\n",
      "  220 root      20   0       0      0      0 S   0.0  0.0   0:11.03 kworker/18+\n",
      "  226 root      20   0       0      0      0 S   0.0  0.0   0:00.73 kworker/15+\n",
      "  229 root      20   0       0      0      0 S   0.0  0.0   0:02.24 kworker/23+\n",
      "  230 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kpsmoused\n",
      "  249 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 deferwq\n",
      "  279 root      20   0       0      0      0 S   0.0  0.0   0:00.31 kauditd\n",
      "  391 root      20   0       0      0      0 S   0.0  0.0   0:00.58 kworker/3:0\n",
      "  406 root      20   0       0      0      0 S   0.0  0.0   0:06.93 kworker/11+\n",
      "  410 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_0\n",
      "  411 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_0\n",
      "  413 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ata_sff\n",
      "  426 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_1\n",
      "  427 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_1\n",
      "  428 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_2\n",
      "  429 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_2\n",
      "  430 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_3\n",
      "  431 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_3\n",
      "  432 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_4\n",
      "  433 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_4\n",
      "  434 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_5\n",
      "  435 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_5\n",
      "  436 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_6\n",
      "  437 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_6\n",
      "  440 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u5+\n",
      "  441 root      20   0       0      0      0 S   0.0  0.0   0:01.22 kworker/u5+\n",
      "  448 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u4+\n",
      "  510 root      20   0       0      0      0 S   0.0  0.0   0:02.50 jbd2/sda2-8\n",
      "  511 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  512 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  581 root      20   0  177688  94444  94092 S   0.0  0.1   0:39.94 systemd-jo+\n",
      "  599 root      20   0   42932   2364   1224 S   0.0  0.0   0:00.16 systemd-ud+\n",
      "  685 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 edac-poller\n",
      "  734 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kvm-irqfd-+\n",
      "  740 root      20   0       0      0      0 S   0.0  0.0   0:00.00 jbd2/sda5-8\n",
      "  741 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  742 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  757 root      16  -4  116684   1636   1240 S   0.0  0.0   0:01.18 auditd\n",
      "  779 root      20   0  339472  24176   6880 S   0.0  0.0   0:01.44 firewalld\n",
      "  788 root      20   0   19284   1288    948 S   0.0  0.0   0:28.91 irqbalance\n",
      "  790 root      20   0  550144  16152   5628 S   0.0  0.0   0:13.05 tuned\n",
      "  791 root      20   0  394132  52108  49456 S   0.0  0.1   0:20.10 rsyslogd\n",
      "  794 root      20   0   34684   1724   1408 S   0.0  0.0   0:01.51 systemd-lo+\n",
      "  795 avahi     20   0   28072   1532   1288 S   0.0  0.0   0:00.29 avahi-daem+\n",
      "  796 dbus      20   0   26708   1928   1344 S   0.0  0.0   0:02.56 dbus-daemon\n",
      "  798 root      20   0  126300   1596    980 S   0.0  0.0   0:00.67 crond\n",
      "  799 root      20   0   25932    940    744 S   0.0  0.0   0:00.00 atd\n",
      "  810 root      20   0  110004    844    712 S   0.0  0.0   0:00.00 agetty\n",
      "  811 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "  812 avahi     20   0   27944    444    216 S   0.0  0.0   0:00.00 avahi-daem+\n",
      "  816 ntp       20   0   31432   2076   1468 S   0.0  0.0   0:00.78 ntpd\n",
      "  935 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "  981 root      20   0  454324  11276   6908 S   0.0  0.0   0:04.47 NetworkMan+\n",
      " 1122 polkitd   20   0  514436  13444   4704 S   0.0  0.0   0:01.51 polkitd\n",
      " 1202 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      " 1333 root      20   0       0      0      0 S   0.0  0.0   0:35.98 kworker/u4+\n",
      " 1476 root       0 -20       0      0      0 S   0.0  0.0   0:00.06 kworker/2:+\n",
      " 1686 root      20   0  360108  26536   4704 S   0.0  0.0   0:17.61 dodai-inst+\n",
      " 1693 root      20   0  223232  11088   6412 S   0.0  0.0   2:32.04 snmpd\n",
      " 1798 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      " 2267 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/8:2\n",
      " 2359 root      20   0  159408   9304   2144 S   0.0  0.0   0:07.65 munin-node\n",
      " 2830 root      20   0   91084   2172   1148 S   0.0  0.0   0:00.83 master\n",
      " 2859 postfix   20   0   91256   3896   2904 S   0.0  0.0   0:00.16 qmgr\n",
      " 3936 root      20   0       0      0      0 S   0.0  0.0   0:02.01 kworker/8:1\n",
      " 4020 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      " 4363 root      20   0       0      0      0 S   0.0  0.0   1:15.51 kworker/19+\n",
      " 4463 root      20   0       0      0      0 S   0.0  0.0   0:03.13 kworker/4:0\n",
      " 5388 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      " 5389 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      " 6599 root      20   0       0      0      0 S   0.0  0.0   0:00.50 kworker/7:2\n",
      " 6726 root      20   0       0      0      0 S   0.0  0.0   0:00.19 kworker/6:0\n",
      " 7076 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      " 7366 root      20   0       0      0      0 S   0.0  0.0   0:01.69 kworker/0:0\n",
      " 8573 root      20   0       0      0      0 S   0.0  0.0   0:00.13 kworker/16+\n",
      " 8687 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/8:+\n",
      " 8688 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/9:+\n",
      " 8689 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/10+\n",
      " 8764 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      " 8765 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      " 8770 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      " 8826 elastic+  20   0 43.644g 0.032t  15844 S   0.0 34.3   9:51.07 java\n",
      " 9496 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      " 9587 postfix   20   0   91188   3884   2888 S   0.0  0.0   0:00.00 pickup\n",
      " 9750 root      20   0       0      0      0 S   0.0  0.0   0:32.66 kworker/u4+\n",
      "10714 root      20   0       0      0      0 S   0.0  0.0   0:00.66 kworker/9:0\n",
      "12029 root      20   0       0      0      0 Z   0.0  0.0   0:00.00 sh\n",
      "12033 root      20   0       0      0      0 S   0.0  0.0   0:00.64 kworker/6:2\n",
      "12039 root      20   0       0      0      0 Z   0.0  0.0   0:00.00 sh\n",
      "12139 root      20   0  135216   4848   3680 S   0.0  0.0   0:00.03 sshd\n",
      "12141 ansible   20   0  135348   2612   1356 S   0.0  0.0   0:00.03 sshd\n",
      "12533 ansible   20   0  113116   1192   1020 S   0.0  0.0   0:00.00 sh\n",
      "12547 ansible   20   0  191232   7992   3584 S   0.0  0.0   0:00.03 python\n",
      "12548 ansible   20   0  208384  12716   3848 S   0.0  0.0   0:00.08 python\n",
      "12623 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bond0\n",
      "12690 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "13495 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "13812 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/7:+\n",
      "14037 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/11+\n",
      "14040 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/6:+\n",
      "15258 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "16342 root      20   0       0      0      0 S   0.0  0.0   0:09.90 kworker/10+\n",
      "16519 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "16834 root      20   0       0      0      0 S   0.0  0.0   0:00.56 kworker/20+\n",
      "17835 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      "19148 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      "19154 root      20   0       0      0      0 S   0.0  0.0   0:00.45 kworker/5:1\n",
      "19474 root      20   0   82508   3636   2788 S   0.0  0.0   0:00.00 sshd\n",
      "19978 root      20   0       0      0      0 S   0.0  0.0   0:03.95 kworker/2:0\n",
      "20473 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/5:0\n",
      "21284 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0\n",
      "23134 root      20   0       0      0      0 S   0.0  0.0   0:00.41 kworker/1:2\n",
      "24914 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:2\n",
      "24921 root      20   0       0      0      0 S   0.0  0.0   0:00.07 kworker/21+\n",
      "24949 root      20   0       0      0      0 S   0.0  0.0   0:04.28 kworker/17+\n",
      "25649 root      20   0       0      0      0 S   0.0  0.0   0:00.43 kworker/14+\n",
      "26267 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "26774 root      20   0       0      0      0 S   0.0  0.0   0:00.62 kworker/12+\n",
      "27612 root      20   0       0      0      0 S   0.0  0.0   0:01.60 kworker/13+\n",
      "28056 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/3:1\n",
      "29214 root      20   0       0      0      0 S   0.0  0.0   0:00.26 kworker/7:0\n",
      "29215 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/9:2\n",
      "29873 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      "30361 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      "30835 root      20   0       0      0      0 S   0.0  0.0   0:00.55 kworker/23+\n",
      "30836 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/2:1\n",
      "31329 root      20   0       0      0      0 S   0.0  0.0   0:01.24 kworker/4:1\n",
      "31564 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n",
      "top - 10:51:44 up 2 days, 17:13,  1 user,  load average: 0.00, 0.01, 0.05\n",
      "Tasks: 295 total,   1 running, 294 sleeping,   0 stopped,   0 zombie\n",
      "%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\n",
      "KiB Mem : 98827752 total, 62807528 free, 34798252 used,  1221972 buff/cache\n",
      "KiB Swap: 93750000 total, 93750000 free,        0 used. 63653308 avail Mem \n",
      "\n",
      "  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\n",
      " 2100 ansible   20   0  130152   1808   1196 R   6.2  0.0   0:00.01 top\n",
      "    1 root      20   0   54292   4112   2476 S   0.0  0.0   0:03.72 systemd\n",
      "    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd\n",
      "    3 root      20   0       0      0      0 S   0.0  0.0   0:00.45 ksoftirqd/0\n",
      "    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "    7 root      20   0       0      0      0 S   0.0  0.0   0:00.68 kworker/u4+\n",
      "    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.16 migration/0\n",
      "    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh\n",
      "   10 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0\n",
      "   11 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1\n",
      "   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2\n",
      "   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3\n",
      "   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/4\n",
      "   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/5\n",
      "   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/6\n",
      "   17 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/7\n",
      "   18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/8\n",
      "   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/9\n",
      "   20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/10\n",
      "   21 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/11\n",
      "   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/12\n",
      "   23 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/13\n",
      "   24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/14\n",
      "   25 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/15\n",
      "   26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/16\n",
      "   27 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/17\n",
      "   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/18\n",
      "   29 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/19\n",
      "   30 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/20\n",
      "   31 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/21\n",
      "   32 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/22\n",
      "   33 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/23\n",
      "   34 root      20   0       0      0      0 S   0.0  0.0   5:44.32 rcu_sched\n",
      "   35 root      20   0       0      0      0 S   0.0  0.0   0:04.38 rcuos/0\n",
      "   36 root      20   0       0      0      0 S   0.0  0.0   0:06.00 rcuos/1\n",
      "   37 root      20   0       0      0      0 S   0.0  0.0   0:08.86 rcuos/2\n",
      "   38 root      20   0       0      0      0 S   0.0  0.0   0:08.07 rcuos/3\n",
      "   39 root      20   0       0      0      0 S   0.0  0.0   0:05.83 rcuos/4\n",
      "   40 root      20   0       0      0      0 S   0.0  0.0   0:05.14 rcuos/5\n",
      "   41 root      20   0       0      0      0 S   0.0  0.0   0:04.98 rcuos/6\n",
      "   42 root      20   0       0      0      0 S   0.0  0.0   0:04.06 rcuos/7\n",
      "   43 root      20   0       0      0      0 S   0.0  0.0   0:07.17 rcuos/8\n",
      "   44 root      20   0       0      0      0 S   0.0  0.0   0:27.90 rcuos/9\n",
      "   45 root      20   0       0      0      0 S   0.0  0.0   0:08.72 rcuos/10\n",
      "   46 root      20   0       0      0      0 S   0.0  0.0   0:06.14 rcuos/11\n",
      "   47 root      20   0       0      0      0 S   0.0  0.0   0:01.93 rcuos/12\n",
      "   48 root      20   0       0      0      0 S   0.0  0.0   0:12.85 rcuos/13\n",
      "   49 root      20   0       0      0      0 S   0.0  0.0   1:18.25 rcuos/14\n",
      "   50 root      20   0       0      0      0 S   0.0  0.0   0:02.50 rcuos/15\n",
      "   51 root      20   0       0      0      0 S   0.0  0.0   0:02.24 rcuos/16\n",
      "   52 root      20   0       0      0      0 S   0.0  0.0   0:02.00 rcuos/17\n",
      "   53 root      20   0       0      0      0 S   0.0  0.0   0:01.60 rcuos/18\n",
      "   54 root      20   0       0      0      0 S   0.0  0.0   0:11.04 rcuos/19\n",
      "   55 root      20   0       0      0      0 S   0.0  0.0   0:02.47 rcuos/20\n",
      "   56 root      20   0       0      0      0 S   0.0  0.0   0:03.60 rcuos/21\n",
      "   57 root      20   0       0      0      0 S   0.0  0.0   0:01.74 rcuos/22\n",
      "   58 root      20   0       0      0      0 S   0.0  0.0   0:01.15 rcuos/23\n",
      "   59 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 watchdog/0\n",
      "   60 root      rt   0       0      0      0 S   0.0  0.0   0:00.56 watchdog/1\n",
      "   61 root      rt   0       0      0      0 S   0.0  0.0   0:00.23 migration/1\n",
      "   62 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/1\n",
      "   64 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      "   65 root      rt   0       0      0      0 S   0.0  0.0   0:00.52 watchdog/2\n",
      "   66 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/2\n",
      "   67 root      20   0       0      0      0 S   0.0  0.0   0:00.14 ksoftirqd/2\n",
      "   69 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:+\n",
      "   70 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/3\n",
      "   71 root      rt   0       0      0      0 S   0.0  0.0   0:00.31 migration/3\n",
      "   72 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/3\n",
      "   74 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      "   75 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/4\n",
      "   76 root      rt   0       0      0      0 S   0.0  0.0   0:00.16 migration/4\n",
      "   77 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/4\n",
      "   79 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      "   80 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/5\n",
      "   81 root      rt   0       0      0      0 S   0.0  0.0   0:00.26 migration/5\n",
      "   82 root      20   0       0      0      0 S   0.0  0.0   0:00.44 ksoftirqd/5\n",
      "   84 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      "   85 root      rt   0       0      0      0 S   0.0  0.0   0:00.52 watchdog/6\n",
      "   86 root      rt   0       0      0      0 S   0.0  0.0   0:00.26 migration/6\n",
      "   87 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/6\n",
      "   89 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/6:+\n",
      "   91 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/7\n",
      "   92 root      rt   0       0      0      0 S   0.0  0.0   0:00.23 migration/7\n",
      "   93 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/7\n",
      "   95 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/7:+\n",
      "   96 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/8\n",
      "   97 root      rt   0       0      0      0 S   0.0  0.0   0:00.16 migration/8\n",
      "   98 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/8\n",
      "  100 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/8:+\n",
      "  101 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/9\n",
      "  102 root      rt   0       0      0      0 S   0.0  0.0   0:00.26 migration/9\n",
      "  103 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/9\n",
      "  105 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/9:+\n",
      "  106 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/10\n",
      "  107 root      rt   0       0      0      0 S   0.0  0.0   0:00.16 migration/+\n",
      "  108 root      20   0       0      0      0 S   0.0  0.0   0:00.08 ksoftirqd/+\n",
      "  110 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      "  111 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/11\n",
      "  112 root      rt   0       0      0      0 S   0.0  0.0   0:00.27 migration/+\n",
      "  113 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/+\n",
      "  115 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "  116 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/12\n",
      "  117 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 migration/+\n",
      "  118 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  119 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      "  120 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      "  121 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/13\n",
      "  122 root      rt   0       0      0      0 S   0.0  0.0   0:00.75 migration/+\n",
      "  123 root      20   0       0      0      0 S   0.0  0.0   0:00.35 ksoftirqd/+\n",
      "  125 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "  126 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/14\n",
      "  127 root      rt   0       0      0      0 S   0.0  0.0   0:00.38 migration/+\n",
      "  128 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/+\n",
      "  130 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "  131 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/15\n",
      "  132 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/+\n",
      "  133 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  135 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "  136 root      rt   0       0      0      0 S   0.0  0.0   0:00.46 watchdog/16\n",
      "  137 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 migration/+\n",
      "  138 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  140 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/16+\n",
      "  141 root      rt   0       0      0      0 S   0.0  0.0   0:00.46 watchdog/17\n",
      "  142 root      rt   0       0      0      0 S   0.0  0.0   0:00.34 migration/+\n",
      "  143 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  145 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "  146 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/18\n",
      "  147 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/+\n",
      "  148 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  150 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "  151 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/19\n",
      "  152 root      rt   0       0      0      0 S   0.0  0.0   0:00.73 migration/+\n",
      "  153 root      20   0       0      0      0 S   0.0  0.0   0:00.36 ksoftirqd/+\n",
      "  155 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "  156 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/20\n",
      "  157 root      rt   0       0      0      0 S   0.0  0.0   0:00.36 migration/+\n",
      "  158 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  160 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "  161 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/21\n",
      "  162 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/+\n",
      "  163 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  165 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "  166 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/22\n",
      "  167 root      rt   0       0      0      0 S   0.0  0.0   0:00.18 migration/+\n",
      "  168 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  170 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      "  171 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/23\n",
      "  172 root      rt   0       0      0      0 S   0.0  0.0   0:00.55 migration/+\n",
      "  173 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/+\n",
      "  175 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "  176 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 khelper\n",
      "  177 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs\n",
      "  178 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns\n",
      "  179 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback\n",
      "  180 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd\n",
      "  181 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset\n",
      "  182 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd\n",
      "  184 root      20   0       0      0      0 S   0.0  0.0   0:00.00 khubd\n",
      "  185 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 md\n",
      "  188 root      20   0       0      0      0 S   0.0  0.0   0:00.21 khungtaskd\n",
      "  189 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd0\n",
      "  190 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd1\n",
      "  191 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd\n",
      "  192 root      39  19       0      0      0 S   0.0  0.0   0:01.75 khugepaged\n",
      "  193 root      20   0       0      0      0 S   0.0  0.0   0:00.00 fsnotify_m+\n",
      "  194 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto\n",
      "  202 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld\n",
      "  206 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kmpath_rda+\n",
      "  213 root      20   0       0      0      0 S   0.0  0.0   0:01.08 kworker/14+\n",
      "  214 root      20   0       0      0      0 S   0.0  0.0   0:36.57 kworker/19+\n",
      "  215 root      20   0       0      0      0 S   0.0  0.0   0:05.54 kworker/20+\n",
      "  221 root      20   0       0      0      0 S   0.0  0.0   0:00.99 kworker/16+\n",
      "  224 root      20   0       0      0      0 S   0.0  0.0   0:00.88 kworker/18+\n",
      "  225 root      20   0       0      0      0 S   0.0  0.0   0:01.34 kworker/21+\n",
      "  226 root      20   0       0      0      0 S   0.0  0.0   0:00.76 kworker/12+\n",
      "  229 root      20   0       0      0      0 S   0.0  0.0   0:02.81 kworker/23+\n",
      "  230 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kpsmoused\n",
      "  249 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 deferwq\n",
      "  279 root      20   0       0      0      0 S   0.0  0.0   0:00.30 kauditd\n",
      "  303 postfix   20   0   91188   3888   2888 S   0.0  0.0   0:00.00 pickup\n",
      "  422 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_0\n",
      "  423 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_0\n",
      "  426 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ata_sff\n",
      "  437 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_1\n",
      "  438 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_1\n",
      "  439 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_2\n",
      "  440 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_2\n",
      "  441 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_3\n",
      "  442 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_3\n",
      "  443 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_4\n",
      "  444 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_4\n",
      "  445 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_5\n",
      "  446 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_5\n",
      "  447 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_6\n",
      "  448 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_6\n",
      "  451 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u5+\n",
      "  452 root      20   0       0      0      0 S   0.0  0.0   0:03.03 kworker/u5+\n",
      "  459 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u4+\n",
      "  507 root      20   0       0      0      0 S   0.0  0.0   0:06.82 kworker/11+\n",
      "  510 root      20   0       0      0      0 S   0.0  0.0   0:02.33 jbd2/sda2-8\n",
      "  511 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  512 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  581 root      20   0  161888  92012  91672 S   0.0  0.1   0:35.41 systemd-jo+\n",
      "  600 root      20   0   42344   1776   1224 S   0.0  0.0   0:00.11 systemd-ud+\n",
      "  645 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 edac-poller\n",
      "  716 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kvm-irqfd-+\n",
      "  729 root      20   0       0      0      0 S   0.0  0.0   0:00.00 jbd2/sda5-8\n",
      "  730 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  731 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  744 root      16  -4  116684   1632   1240 S   0.0  0.0   0:01.04 auditd\n",
      "  766 root      20   0  339500  24164   6880 S   0.0  0.0   0:01.18 firewalld\n",
      "  770 avahi     20   0   28072   1684   1392 S   0.0  0.0   0:00.27 avahi-daem+\n",
      "  771 root      20   0   19284   1284    948 S   0.0  0.0   0:28.55 irqbalance\n",
      "  777 avahi     20   0   27944    440    216 S   0.0  0.0   0:00.00 avahi-daem+\n",
      "  778 root      20   0  314184  50864  48220 S   0.0  0.1   0:18.81 rsyslogd\n",
      "  779 root      20   0  550144  18168   5628 S   0.0  0.0   0:13.14 tuned\n",
      "  781 dbus      20   0   26708   1920   1344 S   0.0  0.0   0:02.56 dbus-daemon\n",
      "  783 root      20   0   34684   1724   1408 S   0.0  0.0   0:01.54 systemd-lo+\n",
      "  784 root      20   0   25932    944    744 S   0.0  0.0   0:00.00 atd\n",
      "  785 root      20   0  126304   1600    984 S   0.0  0.0   0:00.63 crond\n",
      "  799 root      20   0  110004    840    712 S   0.0  0.0   0:00.00 agetty\n",
      "  800 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/8:+\n",
      "  801 ntp       20   0   31432   2068   1468 S   0.0  0.0   0:00.76 ntpd\n",
      "  843 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/10+\n",
      "  903 root      20   0  454052   8760   6748 S   0.0  0.0   0:03.49 NetworkMan+\n",
      "  925 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/11+\n",
      "  970 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/9:+\n",
      " 1012 polkitd   20   0  514436  11416   4708 S   0.0  0.0   0:01.55 polkitd\n",
      " 1029 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/6:+\n",
      " 1071 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kworker/0:0\n",
      " 1093 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/7:+\n",
      " 1568 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bond0\n",
      " 1687 root      20   0  135216   4848   3680 S   0.0  0.0   0:00.03 sshd\n",
      " 1689 ansible   20   0  135348   2612   1356 S   0.0  0.0   0:00.03 sshd\n",
      " 1817 root      20   0  223264  11076   6412 S   0.0  0.0   2:31.96 snmpd\n",
      " 1820 root      20   0  217128  19940   4528 S   0.0  0.0   0:03.58 dodai-inst+\n",
      " 2084 ansible   20   0  113116   1196   1020 S   0.0  0.0   0:00.00 sh\n",
      " 2098 ansible   20   0  191232   7992   3584 S   0.0  0.0   0:00.03 python\n",
      " 2099 ansible   20   0  208384  12716   3848 S   0.0  0.0   0:00.06 python\n",
      " 2407 root      20   0  155272   9208   2068 S   0.0  0.0   0:08.70 munin-node\n",
      " 2529 root      20   0   91084   2172   1148 S   0.0  0.0   0:00.86 master\n",
      " 2575 postfix   20   0   91256   3908   2908 S   0.0  0.0   0:00.16 qmgr\n",
      " 2906 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      " 2966 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      " 3319 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      " 4636 root      20   0       0      0      0 S   0.0  0.0   0:00.35 kworker/17+\n",
      " 4637 root      20   0       0      0      0 S   0.0  0.0   0:03.50 kworker/10+\n",
      " 4738 root      20   0       0      0      0 S   0.0  0.0   0:00.77 kworker/3:0\n",
      " 6807 root       0 -20       0      0      0 S   0.0  0.0   0:00.04 kworker/2:+\n",
      " 6928 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      " 7109 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      " 7372 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/9:1\n",
      " 8687 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/4:0\n",
      " 8904 root      20   0       0      0      0 S   0.0  0.0   0:24.33 kworker/u4+\n",
      " 9063 root      20   0   82508   3640   2788 S   0.0  0.0   0:00.00 sshd\n",
      " 9508 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "10028 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "10222 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/16+\n",
      "10511 root      20   0       0      0      0 S   0.0  0.0   0:01.43 kworker/8:2\n",
      "10843 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:1\n",
      "11356 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/3:2\n",
      "11997 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "12188 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0\n",
      "14571 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      "14820 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/8:0\n",
      "15033 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "15232 root      20   0       0      0      0 S   0.0  0.0   0:00.38 kworker/15+\n",
      "15817 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "16140 root      20   0       0      0      0 S   0.0  0.0   0:00.70 kworker/5:1\n",
      "16553 root      20   0       0      0      0 S   0.0  0.0   0:00.47 kworker/22+\n",
      "16652 root      20   0       0      0      0 S   0.0  0.0   0:02.06 kworker/0:2\n",
      "17048 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "17460 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0\n",
      "18336 root      20   0       0      0      0 S   0.0  0.0   0:21.93 kworker/u4+\n",
      "18781 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "19184 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "19394 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "19930 root      20   0       0      0      0 S   0.0  0.0   0:00.57 kworker/1:1\n",
      "22008 root      20   0       0      0      0 S   0.0  0.0   0:24.71 kworker/13+\n",
      "23266 root      20   0       0      0      0 S   0.0  0.0   0:00.09 kworker/14+\n",
      "23268 root      20   0       0      0      0 S   0.0  0.0   0:00.78 kworker/6:2\n",
      "24478 root      20   0       0      0      0 S   0.0  0.0   0:04.01 kworker/7:1\n",
      "24544 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      "24586 root      20   0       0      0      0 S   0.0  0.0   0:01.03 kworker/9:0\n",
      "24764 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "25394 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "25570 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "27349 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/7:2\n",
      "28054 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/5:2\n",
      "29873 root      20   0       0      0      0 S   0.0  0.0   0:02.99 kworker/2:2\n",
      "30450 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "30579 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/6:0\n",
      "30867 elastic+  20   0 43.487g 0.031t  15868 S   0.0 34.2   9:36.72 java\n",
      "30922 root      20   0       0      0      0 S   0.0  0.0   0:04.05 kworker/4:2\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n",
      "top - 10:51:44 up 2 days, 17:14,  1 user,  load average: 0.00, 0.01, 0.05\n",
      "Tasks: 293 total,   1 running, 292 sleeping,   0 stopped,   0 zombie\n",
      "%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\n",
      "KiB Mem : 98827752 total, 62337512 free, 34777856 used,  1712384 buff/cache\n",
      "KiB Swap: 93750000 total, 93750000 free,        0 used. 63656840 avail Mem \n",
      "\n",
      "  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\n",
      "    1 root      20   0   54328   4124   2476 S   0.0  0.0   0:03.62 systemd\n",
      "    2 root      20   0       0      0      0 S   0.0  0.0   0:00.01 kthreadd\n",
      "    3 root      20   0       0      0      0 S   0.0  0.0   0:00.03 ksoftirqd/0\n",
      "    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "    7 root      20   0       0      0      0 S   0.0  0.0   0:02.07 kworker/u4+\n",
      "    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 migration/0\n",
      "    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh\n",
      "   10 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0\n",
      "   11 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1\n",
      "   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2\n",
      "   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3\n",
      "   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/4\n",
      "   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/5\n",
      "   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/6\n",
      "   17 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/7\n",
      "   18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/8\n",
      "   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/9\n",
      "   20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/10\n",
      "   21 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/11\n",
      "   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/12\n",
      "   23 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/13\n",
      "   24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/14\n",
      "   25 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/15\n",
      "   26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/16\n",
      "   27 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/17\n",
      "   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/18\n",
      "   29 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/19\n",
      "   30 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/20\n",
      "   31 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/21\n",
      "   32 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/22\n",
      "   33 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/23\n",
      "   34 root      20   0       0      0      0 S   0.0  0.0   5:41.28 rcu_sched\n",
      "   35 root      20   0       0      0      0 S   0.0  0.0   0:04.93 rcuos/0\n",
      "   36 root      20   0       0      0      0 S   0.0  0.0   0:05.51 rcuos/1\n",
      "   37 root      20   0       0      0      0 S   0.0  0.0   0:08.44 rcuos/2\n",
      "   38 root      20   0       0      0      0 S   0.0  0.0   0:06.34 rcuos/3\n",
      "   39 root      20   0       0      0      0 S   0.0  0.0   0:05.27 rcuos/4\n",
      "   40 root      20   0       0      0      0 S   0.0  0.0   0:04.87 rcuos/5\n",
      "   41 root      20   0       0      0      0 S   0.0  0.0   0:12.52 rcuos/6\n",
      "   42 root      20   0       0      0      0 S   0.0  0.0   0:04.18 rcuos/7\n",
      "   43 root      20   0       0      0      0 S   0.0  0.0   0:08.07 rcuos/8\n",
      "   44 root      20   0       0      0      0 S   0.0  0.0   0:05.46 rcuos/9\n",
      "   45 root      20   0       0      0      0 S   0.0  0.0   0:04.56 rcuos/10\n",
      "   46 root      20   0       0      0      0 S   0.0  0.0   0:04.72 rcuos/11\n",
      "   47 root      20   0       0      0      0 S   0.0  0.0   0:21.14 rcuos/12\n",
      "   48 root      20   0       0      0      0 S   0.0  0.0   0:11.12 rcuos/13\n",
      "   49 root      20   0       0      0      0 S   0.0  0.0   1:12.74 rcuos/14\n",
      "   50 root      20   0       0      0      0 S   0.0  0.0   0:01.89 rcuos/15\n",
      "   51 root      20   0       0      0      0 S   0.0  0.0   0:01.69 rcuos/16\n",
      "   52 root      20   0       0      0      0 S   0.0  0.0   0:01.77 rcuos/17\n",
      "   53 root      20   0       0      0      0 S   0.0  0.0   0:04.58 rcuos/18\n",
      "   54 root      20   0       0      0      0 S   0.0  0.0   0:12.88 rcuos/19\n",
      "   55 root      20   0       0      0      0 S   0.0  0.0   0:03.04 rcuos/20\n",
      "   56 root      20   0       0      0      0 S   0.0  0.0   0:02.36 rcuos/21\n",
      "   57 root      20   0       0      0      0 S   0.0  0.0   0:02.34 rcuos/22\n",
      "   58 root      20   0       0      0      0 S   0.0  0.0   0:01.49 rcuos/23\n",
      "   59 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 watchdog/0\n",
      "   60 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 watchdog/1\n",
      "   61 root      rt   0       0      0      0 S   0.0  0.0   0:00.23 migration/1\n",
      "   62 root      20   0       0      0      0 S   0.0  0.0   0:00.46 ksoftirqd/1\n",
      "   64 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      "   65 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 watchdog/2\n",
      "   66 root      rt   0       0      0      0 S   0.0  0.0   0:00.21 migration/2\n",
      "   67 root      20   0       0      0      0 S   0.0  0.0   0:00.65 ksoftirqd/2\n",
      "   69 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:+\n",
      "   70 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/3\n",
      "   71 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/3\n",
      "   72 root      20   0       0      0      0 S   0.0  0.0   0:00.03 ksoftirqd/3\n",
      "   74 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      "   75 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/4\n",
      "   76 root      rt   0       0      0      0 S   0.0  0.0   0:00.33 migration/4\n",
      "   77 root      20   0       0      0      0 S   0.0  0.0   0:00.03 ksoftirqd/4\n",
      "   79 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      "   80 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/5\n",
      "   81 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/5\n",
      "   82 root      20   0       0      0      0 S   0.0  0.0   0:00.03 ksoftirqd/5\n",
      "   84 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      "   85 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 watchdog/6\n",
      "   86 root      rt   0       0      0      0 S   0.0  0.0   0:00.27 migration/6\n",
      "   87 root      20   0       0      0      0 S   0.0  0.0   0:00.05 ksoftirqd/6\n",
      "   89 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/6:+\n",
      "   91 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/7\n",
      "   92 root      rt   0       0      0      0 S   0.0  0.0   0:00.34 migration/7\n",
      "   93 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/7\n",
      "   95 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/7:+\n",
      "   96 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/8\n",
      "   97 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/8\n",
      "   98 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/8\n",
      "  100 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/8:+\n",
      "  101 root      rt   0       0      0      0 S   0.0  0.0   0:00.51 watchdog/9\n",
      "  102 root      rt   0       0      0      0 S   0.0  0.0   0:00.25 migration/9\n",
      "  103 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/9\n",
      "  105 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/9:+\n",
      "  106 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/10\n",
      "  107 root      rt   0       0      0      0 S   0.0  0.0   0:00.29 migration/+\n",
      "  108 root      20   0       0      0      0 S   0.0  0.0   0:00.08 ksoftirqd/+\n",
      "  110 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      "  111 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/11\n",
      "  112 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/+\n",
      "  113 root      20   0       0      0      0 S   0.0  0.0   0:00.04 ksoftirqd/+\n",
      "  115 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "  116 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/12\n",
      "  117 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/+\n",
      "  118 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  120 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      "  121 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/13\n",
      "  122 root      rt   0       0      0      0 S   0.0  0.0   0:00.66 migration/+\n",
      "  123 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  125 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "  126 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/14\n",
      "  127 root      rt   0       0      0      0 S   0.0  0.0   0:00.35 migration/+\n",
      "  128 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/+\n",
      "  130 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "  131 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/15\n",
      "  132 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/+\n",
      "  133 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  135 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "  136 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/16\n",
      "  137 root      rt   0       0      0      0 S   0.0  0.0   0:00.24 migration/+\n",
      "  138 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  140 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/16+\n",
      "  141 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/17\n",
      "  142 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/+\n",
      "  143 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  145 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "  146 root      rt   0       0      0      0 S   0.0  0.0   0:00.49 watchdog/18\n",
      "  147 root      rt   0       0      0      0 S   0.0  0.0   0:00.19 migration/+\n",
      "  148 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  149 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "  150 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      "  151 root      rt   0       0      0      0 S   0.0  0.0   0:00.52 watchdog/19\n",
      "  152 root      rt   0       0      0      0 S   0.0  0.0   0:00.85 migration/+\n",
      "  153 root      20   0       0      0      0 S   0.0  0.0   0:00.36 ksoftirqd/+\n",
      "  154 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "  155 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      "  156 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/20\n",
      "  157 root      rt   0       0      0      0 S   0.0  0.0   0:00.37 migration/+\n",
      "  158 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  160 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/20+\n",
      "  161 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/21\n",
      "  162 root      rt   0       0      0      0 S   0.0  0.0   0:00.20 migration/+\n",
      "  163 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  164 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "  165 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/21+\n",
      "  166 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/22\n",
      "  167 root      rt   0       0      0      0 S   0.0  0.0   0:00.29 migration/+\n",
      "  168 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  170 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      "  171 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 watchdog/23\n",
      "  172 root      rt   0       0      0      0 S   0.0  0.0   0:00.60 migration/+\n",
      "  173 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/+\n",
      "  174 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "  175 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      "  176 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 khelper\n",
      "  177 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs\n",
      "  178 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns\n",
      "  179 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback\n",
      "  180 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd\n",
      "  181 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset\n",
      "  182 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd\n",
      "  184 root      20   0       0      0      0 S   0.0  0.0   0:00.00 khubd\n",
      "  185 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 md\n",
      "  187 root      20   0       0      0      0 S   0.0  0.0   0:00.21 khungtaskd\n",
      "  188 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd0\n",
      "  189 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd1\n",
      "  190 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd\n",
      "  191 root      39  19       0      0      0 S   0.0  0.0   0:01.76 khugepaged\n",
      "  192 root      20   0       0      0      0 S   0.0  0.0   0:00.00 fsnotify_m+\n",
      "  193 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto\n",
      "  201 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld\n",
      "  205 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kmpath_rda+\n",
      "  214 root      20   0       0      0      0 S   0.0  0.0   0:01.52 kworker/18+\n",
      "  216 root      20   0       0      0      0 S   0.0  0.0   1:18.28 kworker/19+\n",
      "  217 root      20   0       0      0      0 S   0.0  0.0   0:00.87 kworker/22+\n",
      "  218 root      20   0       0      0      0 S   0.0  0.0   0:01.76 kworker/13+\n",
      "  220 root      20   0       0      0      0 S   0.0  0.0   0:07.74 kworker/12+\n",
      "  229 root      20   0       0      0      0 S   0.0  0.0   0:00.95 kworker/23+\n",
      "  230 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kpsmoused\n",
      "  249 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 deferwq\n",
      "  279 root      20   0       0      0      0 S   0.0  0.0   0:00.30 kauditd\n",
      "  410 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_0\n",
      "  411 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_0\n",
      "  413 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ata_sff\n",
      "  425 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_1\n",
      "  426 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_1\n",
      "  428 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_2\n",
      "  429 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_2\n",
      "  430 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_3\n",
      "  431 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_3\n",
      "  432 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_4\n",
      "  433 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_4\n",
      "  434 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_5\n",
      "  435 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_5\n",
      "  436 root      20   0       0      0      0 S   0.0  0.0   0:00.00 scsi_eh_6\n",
      "  437 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 scsi_tmf_6\n",
      "  440 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u5+\n",
      "  441 root      20   0       0      0      0 S   0.0  0.0   0:01.94 kworker/u5+\n",
      "  497 root      20   0       0      0      0 S   0.0  0.0   0:00.97 kworker/21+\n",
      "  512 root      20   0       0      0      0 S   0.0  0.0   0:02.49 jbd2/sda2-8\n",
      "  513 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  514 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  582 root      20   0  161860  92048  91700 S   0.0  0.1   0:39.08 systemd-jo+\n",
      "  601 root      20   0   42432   1868   1224 S   0.0  0.0   0:00.10 systemd-ud+\n",
      "  647 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 edac-poller\n",
      "  712 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kvm-irqfd-+\n",
      "  724 root      20   0       0      0      0 S   0.0  0.0   0:00.00 jbd2/sda5-8\n",
      "  725 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-rsv-c+\n",
      "  726 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ext4-unrsv+\n",
      "  739 root      16  -4  116684   1632   1240 S   0.0  0.0   0:01.10 auditd\n",
      "  761 root      20   0  339508  24180   6880 S   0.0  0.0   0:01.17 firewalld\n",
      "  765 root      20   0   19284   1284    948 S   0.0  0.0   0:28.09 irqbalance\n",
      "  766 avahi     20   0   28072   1684   1392 S   0.0  0.0   0:00.30 avahi-daem+\n",
      "  770 root      20   0  314156  50824  48184 S   0.0  0.1   0:19.00 rsyslogd\n",
      "  771 root      20   0  550144  16152   5628 S   0.0  0.0   0:13.31 tuned\n",
      "  774 dbus      20   0   26708   1924   1344 S   0.0  0.0   0:02.54 dbus-daemon\n",
      "  776 avahi     20   0   27944    440    212 S   0.0  0.0   0:00.00 avahi-daem+\n",
      "  777 root      20   0   34684   1732   1408 S   0.0  0.0   0:01.54 systemd-lo+\n",
      "  779 root      20   0   25932    940    744 S   0.0  0.0   0:00.00 atd\n",
      "  783 root      20   0  126300   1600    984 S   0.0  0.0   0:00.67 crond\n",
      "  793 root      20   0  110004    840    712 S   0.0  0.0   0:00.00 agetty\n",
      "  794 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:+\n",
      "  795 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+\n",
      "  796 ntp       20   0   31432   2076   1468 S   0.0  0.0   0:00.74 ntpd\n",
      "  905 root      20   0  527808   8764   6748 S   0.0  0.0   0:03.49 NetworkMan+\n",
      "  922 root      20   0       0      0      0 S   0.0  0.0   0:32.60 kworker/u4+\n",
      "  955 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/9:+\n",
      "  973 root      20   0       0      0      0 S   0.0  0.0   0:03.74 kworker/6:1\n",
      "  976 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/10+\n",
      " 1007 polkitd   20   0  514404  12436   4708 S   0.0  0.0   0:01.52 polkitd\n",
      " 1008 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/11+\n",
      " 1026 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/6:+\n",
      " 1050 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/7:+\n",
      " 1071 root       0 -20       0      0      0 S   0.0  0.0   0:00.01 kworker/8:+\n",
      " 1194 root      20   0       0      0      0 S   0.0  0.0   0:02.91 kworker/5:1\n",
      " 1396 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/5:+\n",
      " 1407 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u4+\n",
      " 1489 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/12+\n",
      " 1566 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bond0\n",
      " 1698 postfix   20   0   91188   3888   2888 S   0.0  0.0   0:00.00 pickup\n",
      " 1816 root      20   0  217012  19936   4528 S   0.0  0.0   0:03.70 dodai-inst+\n",
      " 1822 root      20   0  223264  11076   6412 S   0.0  0.0   2:32.76 snmpd\n",
      " 2414 root      20   0  155272   9212   2068 S   0.0  0.0   0:08.82 munin-node\n",
      " 2597 root      20   0   91084   2168   1144 S   0.0  0.0   0:00.80 master\n",
      " 2629 postfix   20   0   91256   3908   2908 S   0.0  0.0   0:00.16 qmgr\n",
      " 3020 root      20   0  135216   4852   3680 S   0.0  0.0   0:00.03 sshd\n",
      " 3022 ansible   20   0  135348   2616   1356 S   0.0  0.0   0:00.04 sshd\n",
      " 3115 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/8:1\n",
      " 3415 ansible   20   0  113116   1196   1020 S   0.0  0.0   0:00.00 sh\n",
      " 3429 ansible   20   0  191232   7992   3584 S   0.0  0.0   0:00.03 python\n",
      " 3430 ansible   20   0  208384  12720   3848 S   0.0  0.0   0:00.07 python\n",
      " 3431 ansible   20   0  130152   1808   1196 R   0.0  0.0   0:00.00 top\n",
      " 3622 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/22+\n",
      " 3916 root      20   0       0      0      0 S   0.0  0.0   0:01.79 kworker/10+\n",
      " 4057 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/23+\n",
      " 4261 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/18+\n",
      " 4668 root      20   0       0      0      0 S   0.0  0.0   0:00.19 kworker/16+\n",
      " 5236 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/10+\n",
      " 6375 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/19+\n",
      " 6754 root      20   0       0      0      0 S   0.0  0.0   0:01.02 kworker/20+\n",
      " 7659 root      20   0       0      0      0 S   0.0  0.0   0:00.85 kworker/15+\n",
      " 8369 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:+\n",
      " 9062 root      20   0   82508   3644   2788 S   0.0  0.0   0:00.01 sshd\n",
      " 9541 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/6:0\n",
      "10954 root      20   0       0      0      0 S   0.0  0.0   0:05.41 kworker/7:1\n",
      "11158 root      20   0       0      0      0 S   0.0  0.0   0:00.31 kworker/14+\n",
      "11366 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/13+\n",
      "11374 root      20   0       0      0      0 S   0.0  0.0   0:01.03 kworker/11+\n",
      "12815 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/11+\n",
      "13681 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/16+\n",
      "13682 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/9:1\n",
      "15763 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:+\n",
      "15927 root      20   0       0      0      0 S   0.0  0.0   0:05.03 kworker/u4+\n",
      "17093 root      20   0       0      0      0 S   0.0  0.0   0:04.57 kworker/9:2\n",
      "17469 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/4:0\n",
      "17555 root      20   0       0      0      0 S   0.0  0.0   0:02.01 kworker/17+\n",
      "17985 root      20   0       0      0      0 S   0.0  0.0   0:02.00 kworker/2:2\n",
      "19082 root      20   0       0      0      0 S   0.0  0.0   0:00.02 kworker/20+\n",
      "21446 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/2:1\n",
      "24163 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/15+\n",
      "25910 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/5:0\n",
      "27711 root      20   0       0      0      0 S   0.0  0.0   0:01.41 kworker/8:0\n",
      "28025 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0\n",
      "28215 root      20   0       0      0      0 S   0.0  0.0   0:01.26 kworker/4:2\n",
      "29531 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/7:0\n",
      "29959 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/14+\n",
      "30688 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/3:1\n",
      "30690 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/17+\n",
      "31205 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0\n",
      "31694 root      20   0       0      0      0 S   0.0  0.0   0:00.75 kworker/0:1\n",
      "31703 root       0 -20       0      0      0 S   0.0  0.0   0:00.07 kworker/2:+\n",
      "32010 elastic+  20   0 43.455g 0.031t  15852 S   0.0 34.2  11:24.22 java\n",
      "32296 root      20   0       0      0      0 S   0.0  0.0   0:09.54 kworker/1:1\n",
      "32583 root      20   0       0      0      0 S   0.0  0.0   0:02.39 kworker/3:2\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m shell -a 'top n 1 b ' -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbfb9e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbfd56-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbf9dc-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbfd56-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbfb9e-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "CPU使用率、メモリ使用率が継続的に100%近い数値になっていないか確認します。  \n",
    "継続的に100%近い数値になっている場合、リソースが枯渇していると考えられます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbff22-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc00e4-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbfd56-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### ヒープが枯渇していないか？\n",
    "ノード状態を調査するAPIを発行し、ヒープの利用状況を確認します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc00e4-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc02a6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbff22-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Nodes Stats APIを、JVMの状態を見るオプションを付与して発行します。  \n",
    "APIの詳細はElasticsearch Referenceの[Nodes Stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html)を参照してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc02a6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc045e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc00e4-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"_nodes\" : {\r\n",
      "    \"total\" : 5,\r\n",
      "    \"successful\" : 5,\r\n",
      "    \"failed\" : 0\r\n",
      "  },\r\n",
      "  \"cluster_name\" : \"es-cluster\",\r\n",
      "  \"nodes\" : {\r\n",
      "    \"z0-yqIssTTeGPdynW-UEEw\" : {\r\n",
      "      \"timestamp\" : 1491443504886,\r\n",
      "      \"name\" : \"z0-yqIs\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.235:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.235\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.235:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"jvm\" : {\r\n",
      "        \"timestamp\" : 1491443504886,\r\n",
      "        \"uptime_in_millis\" : 162357336,\r\n",
      "        \"mem\" : {\r\n",
      "          \"heap_used_in_bytes\" : 969223056,\r\n",
      "          \"heap_used_percent\" : 2,\r\n",
      "          \"heap_committed_in_bytes\" : 33128972288,\r\n",
      "          \"heap_max_in_bytes\" : 33128972288,\r\n",
      "          \"non_heap_used_in_bytes\" : 68031648,\r\n",
      "          \"non_heap_committed_in_bytes\" : 72294400,\r\n",
      "          \"pools\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"used_in_bytes\" : 513253704,\r\n",
      "              \"max_in_bytes\" : 1256259584,\r\n",
      "              \"peak_used_in_bytes\" : 1256259584,\r\n",
      "              \"peak_max_in_bytes\" : 1256259584\r\n",
      "            },\r\n",
      "            \"survivor\" : {\r\n",
      "              \"used_in_bytes\" : 4482560,\r\n",
      "              \"max_in_bytes\" : 157024256,\r\n",
      "              \"peak_used_in_bytes\" : 157024248,\r\n",
      "              \"peak_max_in_bytes\" : 157024256\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"used_in_bytes\" : 451486792,\r\n",
      "              \"max_in_bytes\" : 31715688448,\r\n",
      "              \"peak_used_in_bytes\" : 451486792,\r\n",
      "              \"peak_max_in_bytes\" : 31715688448\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"threads\" : {\r\n",
      "          \"count\" : 114,\r\n",
      "          \"peak_count\" : 118\r\n",
      "        },\r\n",
      "        \"gc\" : {\r\n",
      "          \"collectors\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"collection_count\" : 3,\r\n",
      "              \"collection_time_in_millis\" : 289\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"collection_count\" : 1,\r\n",
      "              \"collection_time_in_millis\" : 404\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"buffer_pools\" : {\r\n",
      "          \"direct\" : {\r\n",
      "            \"count\" : 50,\r\n",
      "            \"used_in_bytes\" : 397951788,\r\n",
      "            \"total_capacity_in_bytes\" : 397951787\r\n",
      "          },\r\n",
      "          \"mapped\" : {\r\n",
      "            \"count\" : 0,\r\n",
      "            \"used_in_bytes\" : 0,\r\n",
      "            \"total_capacity_in_bytes\" : 0\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"classes\" : {\r\n",
      "          \"current_loaded_count\" : 9065,\r\n",
      "          \"total_loaded_count\" : 9065,\r\n",
      "          \"total_unloaded_count\" : 0\r\n",
      "        }\r\n",
      "      }\r\n",
      "    },\r\n",
      "    \"kMWX4wbYTC-jYH-UJHmVmg\" : {\r\n",
      "      \"timestamp\" : 1491443504886,\r\n",
      "      \"name\" : \"kMWX4wb\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.232:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.232\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.232:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"master\",\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"jvm\" : {\r\n",
      "        \"timestamp\" : 1491443504886,\r\n",
      "        \"uptime_in_millis\" : 162357760,\r\n",
      "        \"mem\" : {\r\n",
      "          \"heap_used_in_bytes\" : 1214249592,\r\n",
      "          \"heap_used_percent\" : 3,\r\n",
      "          \"heap_committed_in_bytes\" : 33128972288,\r\n",
      "          \"heap_max_in_bytes\" : 33128972288,\r\n",
      "          \"non_heap_used_in_bytes\" : 68588872,\r\n",
      "          \"non_heap_committed_in_bytes\" : 72876032,\r\n",
      "          \"pools\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"used_in_bytes\" : 669358648,\r\n",
      "              \"max_in_bytes\" : 1256259584,\r\n",
      "              \"peak_used_in_bytes\" : 1256259584,\r\n",
      "              \"peak_max_in_bytes\" : 1256259584\r\n",
      "            },\r\n",
      "            \"survivor\" : {\r\n",
      "              \"used_in_bytes\" : 7127584,\r\n",
      "              \"max_in_bytes\" : 157024256,\r\n",
      "              \"peak_used_in_bytes\" : 157024248,\r\n",
      "              \"peak_max_in_bytes\" : 157024256\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"used_in_bytes\" : 537763360,\r\n",
      "              \"max_in_bytes\" : 31715688448,\r\n",
      "              \"peak_used_in_bytes\" : 537763360,\r\n",
      "              \"peak_max_in_bytes\" : 31715688448\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"threads\" : {\r\n",
      "          \"count\" : 115,\r\n",
      "          \"peak_count\" : 120\r\n",
      "        },\r\n",
      "        \"gc\" : {\r\n",
      "          \"collectors\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"collection_count\" : 3,\r\n",
      "              \"collection_time_in_millis\" : 315\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"collection_count\" : 1,\r\n",
      "              \"collection_time_in_millis\" : 381\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"buffer_pools\" : {\r\n",
      "          \"direct\" : {\r\n",
      "            \"count\" : 63,\r\n",
      "            \"used_in_bytes\" : 501761631,\r\n",
      "            \"total_capacity_in_bytes\" : 501761630\r\n",
      "          },\r\n",
      "          \"mapped\" : {\r\n",
      "            \"count\" : 0,\r\n",
      "            \"used_in_bytes\" : 0,\r\n",
      "            \"total_capacity_in_bytes\" : 0\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"classes\" : {\r\n",
      "          \"current_loaded_count\" : 9085,\r\n",
      "          \"total_loaded_count\" : 9085,\r\n",
      "          \"total_unloaded_count\" : 0\r\n",
      "        }\r\n",
      "      }\r\n",
      "    },\r\n",
      "    \"LXj8U7-rR4el_IoiVAa8Cg\" : {\r\n",
      "      \"timestamp\" : 1491443504886,\r\n",
      "      \"name\" : \"LXj8U7-\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.231:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.231\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.231:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"master\",\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"jvm\" : {\r\n",
      "        \"timestamp\" : 1491443504886,\r\n",
      "        \"uptime_in_millis\" : 162357731,\r\n",
      "        \"mem\" : {\r\n",
      "          \"heap_used_in_bytes\" : 1716765864,\r\n",
      "          \"heap_used_percent\" : 5,\r\n",
      "          \"heap_committed_in_bytes\" : 33128972288,\r\n",
      "          \"heap_max_in_bytes\" : 33128972288,\r\n",
      "          \"non_heap_used_in_bytes\" : 70881272,\r\n",
      "          \"non_heap_committed_in_bytes\" : 75497472,\r\n",
      "          \"pools\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"used_in_bytes\" : 872606976,\r\n",
      "              \"max_in_bytes\" : 1256259584,\r\n",
      "              \"peak_used_in_bytes\" : 1256259584,\r\n",
      "              \"peak_max_in_bytes\" : 1256259584\r\n",
      "            },\r\n",
      "            \"survivor\" : {\r\n",
      "              \"used_in_bytes\" : 78800,\r\n",
      "              \"max_in_bytes\" : 157024256,\r\n",
      "              \"peak_used_in_bytes\" : 157024256,\r\n",
      "              \"peak_max_in_bytes\" : 157024256\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"used_in_bytes\" : 844080088,\r\n",
      "              \"max_in_bytes\" : 31715688448,\r\n",
      "              \"peak_used_in_bytes\" : 844080088,\r\n",
      "              \"peak_max_in_bytes\" : 31715688448\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"threads\" : {\r\n",
      "          \"count\" : 119,\r\n",
      "          \"peak_count\" : 122\r\n",
      "        },\r\n",
      "        \"gc\" : {\r\n",
      "          \"collectors\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"collection_count\" : 7,\r\n",
      "              \"collection_time_in_millis\" : 511\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"collection_count\" : 1,\r\n",
      "              \"collection_time_in_millis\" : 400\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"buffer_pools\" : {\r\n",
      "          \"direct\" : {\r\n",
      "            \"count\" : 109,\r\n",
      "            \"used_in_bytes\" : 832603750,\r\n",
      "            \"total_capacity_in_bytes\" : 832603749\r\n",
      "          },\r\n",
      "          \"mapped\" : {\r\n",
      "            \"count\" : 0,\r\n",
      "            \"used_in_bytes\" : 0,\r\n",
      "            \"total_capacity_in_bytes\" : 0\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"classes\" : {\r\n",
      "          \"current_loaded_count\" : 9292,\r\n",
      "          \"total_loaded_count\" : 9292,\r\n",
      "          \"total_unloaded_count\" : 0\r\n",
      "        }\r\n",
      "      }\r\n",
      "    },\r\n",
      "    \"YKUa3dgMSjSnTvC0UgjI3w\" : {\r\n",
      "      \"timestamp\" : 1491443504887,\r\n",
      "      \"name\" : \"YKUa3dg\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.234:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.234\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.234:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"jvm\" : {\r\n",
      "        \"timestamp\" : 1491443504888,\r\n",
      "        \"uptime_in_millis\" : 162357349,\r\n",
      "        \"mem\" : {\r\n",
      "          \"heap_used_in_bytes\" : 1046726648,\r\n",
      "          \"heap_used_percent\" : 3,\r\n",
      "          \"heap_committed_in_bytes\" : 33128972288,\r\n",
      "          \"heap_max_in_bytes\" : 33128972288,\r\n",
      "          \"non_heap_used_in_bytes\" : 68631448,\r\n",
      "          \"non_heap_committed_in_bytes\" : 72945664,\r\n",
      "          \"pools\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"used_in_bytes\" : 637270144,\r\n",
      "              \"max_in_bytes\" : 1256259584,\r\n",
      "              \"peak_used_in_bytes\" : 1256259584,\r\n",
      "              \"peak_max_in_bytes\" : 1256259584\r\n",
      "            },\r\n",
      "            \"survivor\" : {\r\n",
      "              \"used_in_bytes\" : 24829928,\r\n",
      "              \"max_in_bytes\" : 157024256,\r\n",
      "              \"peak_used_in_bytes\" : 156694360,\r\n",
      "              \"peak_max_in_bytes\" : 157024256\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"used_in_bytes\" : 384626576,\r\n",
      "              \"max_in_bytes\" : 31715688448,\r\n",
      "              \"peak_used_in_bytes\" : 384626576,\r\n",
      "              \"peak_max_in_bytes\" : 31715688448\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"threads\" : {\r\n",
      "          \"count\" : 114,\r\n",
      "          \"peak_count\" : 118\r\n",
      "        },\r\n",
      "        \"gc\" : {\r\n",
      "          \"collectors\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"collection_count\" : 3,\r\n",
      "              \"collection_time_in_millis\" : 357\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"collection_count\" : 1,\r\n",
      "              \"collection_time_in_millis\" : 181\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"buffer_pools\" : {\r\n",
      "          \"direct\" : {\r\n",
      "            \"count\" : 45,\r\n",
      "            \"used_in_bytes\" : 363340588,\r\n",
      "            \"total_capacity_in_bytes\" : 363340587\r\n",
      "          },\r\n",
      "          \"mapped\" : {\r\n",
      "            \"count\" : 0,\r\n",
      "            \"used_in_bytes\" : 0,\r\n",
      "            \"total_capacity_in_bytes\" : 0\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"classes\" : {\r\n",
      "          \"current_loaded_count\" : 9106,\r\n",
      "          \"total_loaded_count\" : 9106,\r\n",
      "          \"total_unloaded_count\" : 0\r\n",
      "        }\r\n",
      "      }\r\n",
      "    },\r\n",
      "    \"hy7ZccVjThKC6O2FXquNZA\" : {\r\n",
      "      \"timestamp\" : 1491443504888,\r\n",
      "      \"name\" : \"hy7ZccV\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.233:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.233\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.233:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"master\",\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"jvm\" : {\r\n",
      "        \"timestamp\" : 1491443504888,\r\n",
      "        \"uptime_in_millis\" : 162357766,\r\n",
      "        \"mem\" : {\r\n",
      "          \"heap_used_in_bytes\" : 1204936880,\r\n",
      "          \"heap_used_percent\" : 3,\r\n",
      "          \"heap_committed_in_bytes\" : 33128972288,\r\n",
      "          \"heap_max_in_bytes\" : 33128972288,\r\n",
      "          \"non_heap_used_in_bytes\" : 68259352,\r\n",
      "          \"non_heap_committed_in_bytes\" : 72667136,\r\n",
      "          \"pools\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"used_in_bytes\" : 644871480,\r\n",
      "              \"max_in_bytes\" : 1256259584,\r\n",
      "              \"peak_used_in_bytes\" : 1256259584,\r\n",
      "              \"peak_max_in_bytes\" : 1256259584\r\n",
      "            },\r\n",
      "            \"survivor\" : {\r\n",
      "              \"used_in_bytes\" : 6667776,\r\n",
      "              \"max_in_bytes\" : 157024256,\r\n",
      "              \"peak_used_in_bytes\" : 157024256,\r\n",
      "              \"peak_max_in_bytes\" : 157024256\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"used_in_bytes\" : 553397624,\r\n",
      "              \"max_in_bytes\" : 31715688448,\r\n",
      "              \"peak_used_in_bytes\" : 553397624,\r\n",
      "              \"peak_max_in_bytes\" : 31715688448\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"threads\" : {\r\n",
      "          \"count\" : 114,\r\n",
      "          \"peak_count\" : 119\r\n",
      "        },\r\n",
      "        \"gc\" : {\r\n",
      "          \"collectors\" : {\r\n",
      "            \"young\" : {\r\n",
      "              \"collection_count\" : 3,\r\n",
      "              \"collection_time_in_millis\" : 327\r\n",
      "            },\r\n",
      "            \"old\" : {\r\n",
      "              \"collection_count\" : 1,\r\n",
      "              \"collection_time_in_millis\" : 414\r\n",
      "            }\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"buffer_pools\" : {\r\n",
      "          \"direct\" : {\r\n",
      "            \"count\" : 63,\r\n",
      "            \"used_in_bytes\" : 501761631,\r\n",
      "            \"total_capacity_in_bytes\" : 501761630\r\n",
      "          },\r\n",
      "          \"mapped\" : {\r\n",
      "            \"count\" : 0,\r\n",
      "            \"used_in_bytes\" : 0,\r\n",
      "            \"total_capacity_in_bytes\" : 0\r\n",
      "          }\r\n",
      "        },\r\n",
      "        \"classes\" : {\r\n",
      "          \"current_loaded_count\" : 9093,\r\n",
      "          \"total_loaded_count\" : 9093,\r\n",
      "          \"total_unloaded_count\" : 0\r\n",
      "        }\r\n",
      "      }\r\n",
      "    }\r\n",
      "  }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!curl -XGET \"http://$ES_HOST/_nodes/stats/jvm?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc045e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc02a6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
      },
      {
       "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
      },
      {
       "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "応答の\"nodes\"部分にNodeごとの状態が出力されます。  \n",
    "その下の\"jvm\"部分にJVMの状態が出力されます。次を確認してください。\n",
    "\n",
    "1. mem/heap_max_in_bytes に表示される最大ヒープサイズ（単位はbyte）がインストール時に指定した通りか？\n",
    "2. mem/heap_used_percent に表示されるヒープ使用率（単位はパーセント）が100近くになってしまっていないか？\n",
    "3. gc/collectors/old/collection_count に表示されるGCカウントが、稼働時間に比して大きすぎないか？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fbff22-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fbff22-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc00e4-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fbfd56-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a6fc00e4-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc0616-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### 実行中の重いタスクが無いか？\n",
    "タスク管理用のAPIを発行し、実行中の重いタスクが無いか確認します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc00e4-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc02a6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fbff22-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Task Management APIでクラスタ内のノードで実行中のタスクを確認することができます。  \n",
    "APIの詳細はElasticsearch Referenceの[Task Management API](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html)を参照してください。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc02a6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc045e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc00e4-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"nodes\" : {\r\n",
      "    \"hy7ZccVjThKC6O2FXquNZA\" : {\r\n",
      "      \"name\" : \"hy7ZccV\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.233:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.233\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.233:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"master\",\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"tasks\" : {\r\n",
      "        \"hy7ZccVjThKC6O2FXquNZA:167690\" : {\r\n",
      "          \"node\" : \"hy7ZccVjThKC6O2FXquNZA\",\r\n",
      "          \"id\" : 167690,\r\n",
      "          \"type\" : \"netty\",\r\n",
      "          \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n",
      "          \"start_time_in_millis\" : 1491443505138,\r\n",
      "          \"running_time_in_nanos\" : 3950610,\r\n",
      "          \"cancellable\" : false,\r\n",
      "          \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n",
      "        }\r\n",
      "      }\r\n",
      "    },\r\n",
      "    \"z0-yqIssTTeGPdynW-UEEw\" : {\r\n",
      "      \"name\" : \"z0-yqIs\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.235:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.235\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.235:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"tasks\" : {\r\n",
      "        \"z0-yqIssTTeGPdynW-UEEw:167660\" : {\r\n",
      "          \"node\" : \"z0-yqIssTTeGPdynW-UEEw\",\r\n",
      "          \"id\" : 167660,\r\n",
      "          \"type\" : \"netty\",\r\n",
      "          \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n",
      "          \"start_time_in_millis\" : 1491443505137,\r\n",
      "          \"running_time_in_nanos\" : 3140838,\r\n",
      "          \"cancellable\" : false,\r\n",
      "          \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n",
      "        }\r\n",
      "      }\r\n",
      "    },\r\n",
      "    \"YKUa3dgMSjSnTvC0UgjI3w\" : {\r\n",
      "      \"name\" : \"YKUa3dg\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.234:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.234\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.234:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"tasks\" : {\r\n",
      "        \"YKUa3dgMSjSnTvC0UgjI3w:167664\" : {\r\n",
      "          \"node\" : \"YKUa3dgMSjSnTvC0UgjI3w\",\r\n",
      "          \"id\" : 167664,\r\n",
      "          \"type\" : \"netty\",\r\n",
      "          \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n",
      "          \"start_time_in_millis\" : 1491443505137,\r\n",
      "          \"running_time_in_nanos\" : 3198063,\r\n",
      "          \"cancellable\" : false,\r\n",
      "          \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n",
      "        }\r\n",
      "      }\r\n",
      "    },\r\n",
      "    \"kMWX4wbYTC-jYH-UJHmVmg\" : {\r\n",
      "      \"name\" : \"kMWX4wb\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.232:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.232\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.232:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"master\",\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"tasks\" : {\r\n",
      "        \"kMWX4wbYTC-jYH-UJHmVmg:167687\" : {\r\n",
      "          \"node\" : \"kMWX4wbYTC-jYH-UJHmVmg\",\r\n",
      "          \"id\" : 167687,\r\n",
      "          \"type\" : \"netty\",\r\n",
      "          \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n",
      "          \"start_time_in_millis\" : 1491443505137,\r\n",
      "          \"running_time_in_nanos\" : 4089741,\r\n",
      "          \"cancellable\" : false,\r\n",
      "          \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n",
      "        }\r\n",
      "      }\r\n",
      "    },\r\n",
      "    \"LXj8U7-rR4el_IoiVAa8Cg\" : {\r\n",
      "      \"name\" : \"LXj8U7-\",\r\n",
      "      \"transport_address\" : \"XXX.XXX.XXX.231:9300\",\r\n",
      "      \"host\" : \"XXX.XXX.XXX.231\",\r\n",
      "      \"ip\" : \"XXX.XXX.XXX.231:9300\",\r\n",
      "      \"roles\" : [\r\n",
      "        \"master\",\r\n",
      "        \"data\",\r\n",
      "        \"ingest\"\r\n",
      "      ],\r\n",
      "      \"tasks\" : {\r\n",
      "        \"LXj8U7-rR4el_IoiVAa8Cg:665199\" : {\r\n",
      "          \"node\" : \"LXj8U7-rR4el_IoiVAa8Cg\",\r\n",
      "          \"id\" : 665199,\r\n",
      "          \"type\" : \"transport\",\r\n",
      "          \"action\" : \"cluster:monitor/tasks/lists\",\r\n",
      "          \"start_time_in_millis\" : 1491443505133,\r\n",
      "          \"running_time_in_nanos\" : 4885742,\r\n",
      "          \"cancellable\" : false\r\n",
      "        },\r\n",
      "        \"LXj8U7-rR4el_IoiVAa8Cg:665200\" : {\r\n",
      "          \"node\" : \"LXj8U7-rR4el_IoiVAa8Cg\",\r\n",
      "          \"id\" : 665200,\r\n",
      "          \"type\" : \"direct\",\r\n",
      "          \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n",
      "          \"start_time_in_millis\" : 1491443505135,\r\n",
      "          \"running_time_in_nanos\" : 2679468,\r\n",
      "          \"cancellable\" : false,\r\n",
      "          \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n",
      "        }\r\n",
      "      }\r\n",
      "    }\r\n",
      "  }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!curl -XGET \"http://$ES_HOST/_tasks?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc045e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc02a6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
      },
      {
       "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
      },
      {
       "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
      },
      {
       "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "応答の\"nodes\"部分にNodeごとの状態が出力されます。  \n",
    "その下の\"tasks\"部分に走行しているタスクの種類と状態が出力されます。次を確認してください。\n",
    "\n",
    "1. tasksに現れるタスクに想定外の物が無いか？\n",
    "2. 特にrunning_time_in_nanos（実行時間）の長いものが無いか？\n",
    "3. 大量のタスクが滞留していないか？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc07e2-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc09a4-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc0616-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### インデックスが作成されているか？\n",
    "日付等のインデックスで、投入したドキュメントが適切に分割・管理されているか確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc09a4-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc0b8e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc07e2-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "health status index uuid pri rep docs.count docs.deleted store.size pri.store.size\r\n"
     ]
    }
   ],
   "source": [
    "!curl -XGET http://$ES_HOST/_cat/indices?v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc0b8e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc0d64-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc09a4-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc0d64-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc0f30-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc0b8e-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "出力結果から次の内容が確認できれば正常です。\n",
    "\n",
    "1. indexの列に表示されるインデックス名が想定どおりか？  \n",
    "例えば日付インデックスであれば、日付形式の文字列が含まれるか？\n",
    "2. 各インデックスの\"health\" 列が \"green\" かどうか？\n",
    "  - **green**  \n",
    "      ：正常です。\n",
    "  - **yellow**  \n",
    "    ：Primary Shardは割り当てられていますが、Replica Shardが割り当てられていません。\n",
    "  - **red**  \n",
    "    ：特定のShardがクラスタに割り当てられていません"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc0f30-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc1106-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc0d64-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### Shard数が設定通りか？\n",
    "インデックスごとに設定されるShard数が設定した（はずの）値になっているか確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc1106-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc12d2-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc0f30-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{ }\r\n"
     ]
    }
   ],
   "source": [
    "!curl -XGET http://$ES_HOST/$INDEX/_settings?pretty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc12d2-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc1494-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc1106-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "インデックスごとに次のような内容が出力されます。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc1494-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc1656-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc12d2-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    " \"meteorological-data-2015.02\" : {\n",
    "    \"settings\" : {\n",
    "      \"index\" : {\n",
    "        \"creation_date\" : \"1489559717930\",\n",
    "        \"number_of_shards\" : \"3\",\n",
    "        （略）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc1656-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc1818-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc1494-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■確認内容**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc1818-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc19da-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc1656-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "\"number_of_shards\"の数について次を確認してください。\n",
    "\n",
    "- 最初にセットアップしたShard数と一致しているか？\n",
    "- Shard数を設定したことが無いのであれば、現状のData Node数と比べて小さくないか？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "lc_cell_meme": {
     "current": "a6fc19da-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc1b9c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc1818-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## 症状と対処\n",
    "診断の章で異常があった場合、本章の内容に従い対処します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc1b9c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc1d72-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc19da-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**先頭から順に**症状を確認してください。  \n",
    "症状に対処したら、いったん[診断](#診断)の章に戻り、コマンドを**全て再実行し直してください。**\n",
    "\n",
    "これは、本章の各節にある対処が、その前までの節に症状が出ていないことを前提に記述しているためです。  \n",
    "※例えば「クラスタ状態が\"green\"でない」節の対処は、その前にある「サーバマシンが稼働していない」節では問題が無いことを前提に記述しています。よってサーバが起動しているかの確認はしていません。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc1d72-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc1f3e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc1b9c-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### サーバマシンが稼働していない\n",
    "**■項目**：[サーバマシンが稼働しているか？](#サーバマシンが稼働しているか？)  \n",
    "**■症状**：pingの応答がないマシンが存在する"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc1f3e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc2100-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc1d72-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "ハードウェアの障害や誤操作でサーバが停止している。  \n",
    "または、サーバは稼働しているが、通信経路に問題がありpingが到達しない。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc2100-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc22d6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc1f3e-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- マシンが停止している場合は稼働させてください。  \n",
    "なんらかのハードウェア障害が発生している場合は、それを解決してください。\n",
    "- マシンが停止していない場合は、通信経路の状態を確認し、問題を是正してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc22d6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc24ca-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc2100-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### プロセスが稼働していない\n",
    "**■項目**：[プロセスが稼働しているか？](#プロセスが稼働しているか？)  \n",
    "**■症状**：稼働していないElasticsearchプロセスが存在する"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc24ca-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc268c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc22d6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "なんらかのエラー発生、または誤操作によりプロセスがダウンしている。もしくは、そもそも起動していない。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc268c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc2902-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc24ca-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- 「[エラーログが無いか？](#エラーログが無いか？)」でエラーが発生していない場合は、クラスタを起動してください。  \n",
    "次のコマンドでクラスタを起動できます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc2902-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc2af6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc268c-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\n",
      "PLAY [all] *********************************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.231]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.232]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.233]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.235]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.234]\u001b[0m\n",
      "\n",
      "TASK [start elasticsearch] *****************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.231]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.232]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.233]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.235]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.234]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;32mXXX.XXX.XXX.231\u001b[0m              : \u001b[0;32mok=2   \u001b[0m changed=0    unreachable=0    failed=0   \n",
      "\u001b[0;32mXXX.XXX.XXX.232\u001b[0m              : \u001b[0;32mok=2   \u001b[0m changed=0    unreachable=0    failed=0   \n",
      "\u001b[0;32mXXX.XXX.XXX.233\u001b[0m              : \u001b[0;32mok=2   \u001b[0m changed=0    unreachable=0    failed=0   \n",
      "\u001b[0;32mXXX.XXX.XXX.234\u001b[0m              : \u001b[0;32mok=2   \u001b[0m changed=0    unreachable=0    failed=0   \n",
      "\u001b[0;32mXXX.XXX.XXX.235\u001b[0m              : \u001b[0;32mok=2   \u001b[0m changed=0    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook playbooks/start_cluster.yml -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc2af6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc2ccc-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc2902-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "- エラーログが出力されている場合は「[エラーが出力されている](#エラーが出力されている)」の対処に進んでください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc2ccc-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc2e98-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc2af6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### ディスクの使用量に余裕がない\n",
    "**■項目**：[ディスクの使用量に余裕があるか？](#ディスクの使用量に余裕があるか？)  \n",
    "**■症状**：ディスクの残量が少ない"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc2e98-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc3064-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc2ccc-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "Elasticsearchのプロセスか、それ以外のプロセスによる使用量の増加。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc3064-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc3226-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc2e98-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "どのプロセスがディスクサイズを占有しているのか確認します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc3226-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc33e8-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc3064-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;31mXXX.XXX.XXX.232 | FAILED | rc=1 >>\n",
      "22\t/root\n",
      "1\t/mnt\n",
      "1\t/srv\n",
      "1\t/media\n",
      "0\t/proc\n",
      "23\t/etc\n",
      "6\t/opt\n",
      "104\t/var\n",
      "1983\t/usr\n",
      "1\t/service\n",
      "0\t/dev\n",
      "1\t/lost+found\n",
      "0\t/sys\n",
      "73\t/run\n",
      "223\t/tmp\n",
      "108\t/boot\n",
      "1\t/home\n",
      "2541\t/du: cannot access ‘/proc/13688/task/13688/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/13688/task/13688/fdinfo/4’: No such file or directory\n",
      "du: cannot access ‘/proc/13688/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/13688/fdinfo/4’: No such file or directory\n",
      "\u001b[0m\n",
      "\u001b[0;31mXXX.XXX.XXX.231 | FAILED | rc=1 >>\n",
      "1\t/mnt\n",
      "1\t/home\n",
      "1\t/media\n",
      "104\t/var\n",
      "6\t/opt\n",
      "0\t/proc\n",
      "108\t/boot\n",
      "22\t/root\n",
      "73\t/run\n",
      "223\t/tmp\n",
      "0\t/sys\n",
      "1\t/lost+found\n",
      "1\t/service\n",
      "23\t/etc\n",
      "0\t/dev\n",
      "1983\t/usr\n",
      "1\t/srv\n",
      "2542\t/du: cannot access ‘/proc/16391/task/16391/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/16391/task/16391/fdinfo/4’: No such file or directory\n",
      "du: cannot access ‘/proc/16391/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/16391/fdinfo/4’: No such file or directory\n",
      "\u001b[0m\n",
      "\u001b[0;31mXXX.XXX.XXX.233 | FAILED | rc=1 >>\n",
      "1\t/mnt\n",
      "1983\t/usr\n",
      "108\t/boot\n",
      "22\t/root\n",
      "1\t/media\n",
      "1\t/lost+found\n",
      "0\t/proc\n",
      "223\t/tmp\n",
      "0\t/sys\n",
      "1\t/home\n",
      "0\t/dev\n",
      "73\t/run\n",
      "103\t/var\n",
      "1\t/service\n",
      "6\t/opt\n",
      "23\t/etc\n",
      "1\t/srv\n",
      "2541\t/du: cannot access ‘/proc/12908/task/12908/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/12908/task/12908/fdinfo/4’: No such file or directory\n",
      "du: cannot access ‘/proc/12908/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/12908/fdinfo/4’: No such file or directory\n",
      "\u001b[0m\n",
      "\u001b[0;31mXXX.XXX.XXX.235 | FAILED | rc=1 >>\n",
      "108\t/boot\n",
      "104\t/var\n",
      "1\t/media\n",
      "1983\t/usr\n",
      "65\t/run\n",
      "0\t/proc\n",
      "0\t/dev\n",
      "1\t/service\n",
      "1\t/srv\n",
      "6\t/opt\n",
      "1\t/mnt\n",
      "22\t/root\n",
      "1\t/home\n",
      "23\t/etc\n",
      "1\t/lost+found\n",
      "0\t/sys\n",
      "223\t/tmp\n",
      "2533\t/du: cannot access ‘/proc/2457/task/2457/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/2457/task/2457/fdinfo/4’: No such file or directory\n",
      "du: cannot access ‘/proc/2457/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/2457/fdinfo/4’: No such file or directory\n",
      "\u001b[0m\n",
      "\u001b[0;31mXXX.XXX.XXX.234 | FAILED | rc=1 >>\n",
      "0\t/dev\n",
      "65\t/run\n",
      "105\t/var\n",
      "1\t/srv\n",
      "0\t/sys\n",
      "1\t/media\n",
      "425\t/tmp\n",
      "23\t/etc\n",
      "1\t/mnt\n",
      "1\t/lost+found\n",
      "0\t/proc\n",
      "6\t/opt\n",
      "108\t/boot\n",
      "1\t/service\n",
      "1\t/home\n",
      "22\t/root\n",
      "2208\t/usr\n",
      "2961\t/du: cannot access ‘/proc/3788/task/3788/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/3788/task/3788/fdinfo/4’: No such file or directory\n",
      "du: cannot access ‘/proc/3788/fd/4’: No such file or directory\n",
      "du: cannot access ‘/proc/3788/fdinfo/4’: No such file or directory\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m shell -a \"sudo du -m --max-depth=1 /\" -i ./hosts -u $USER --private-key=$KEYPATH\n",
    "# MB単位で、ルートから1階層のみ計測しています。必要に応じてオプションを変更してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc33e8-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fc33e8-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc35aa-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc3226-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "08ecb21e-182f-11e7-b31a-0242ac12000d",
     "previous": "a6fc3226-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "- Elasticsearchのデフォルトの設定では、ディスク使用率が85%を越えると、そのノードに新たなシャードを割り当てません。そのため、ディスク使用率が85%を越えないようにしてください。\n",
    "- Elasticsearch以外が占有している場合は、そちらのデータ削減を検討してください。\n",
    "- Elasticsearchが占有している場合は、不要になったインデックス等の削除を検討してください。\n",
    "- 削除できない場合は、Data Nodeを追加してください。  \n",
    "[01_50_Add_Data_Node.ipynb](01_50_Add_Data_Node.ipynb)を実施してください。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "08ecb21e-182f-11e7-b31a-0242ac12000d",
     "next": "08ecb66a-182f-11e7-b31a-0242ac12000d",
     "previous": "a6fc33e8-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "- なおElasticsearchには、ディスク使用率が100%にならないようwater markを設け、状態に応じてShardを割り振る機能があります。\n",
    "  - 詳細はElasticsearch Referenceの[Disk-based Shard Allocation](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html)を参照してください。\n",
    "  - 例えばディスク使用率が80%を超えた場合、そのNodeには新しいShardを割り当てなくし、  \n",
    "ディスクの空き領域が50GBあった場合、Shardを再配置するelasticsearch.ymlの設定は次の通りです。\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "08ecb66a-182f-11e7-b31a-0242ac12000d",
     "next": "a6fc35aa-14ed-11e7-b31a-0242ac12000d",
     "previous": "08ecb21e-182f-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "cluster.routing.allocation.disk.watermark.low: \"80%\"\n",
    "cluster.routing.allocation.disk.watermark.high: \"50gb\"\n",
    "cluster.info.update.interval: \"1m\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc35aa-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fc35aa-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc3776-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc33e8-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a6fc3776-14ed-11e7-b31a-0242ac12000d",
     "previous": "08ecb66a-182f-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### クラスタと通信できない\n",
    "**■項目**：[クラスタと通信できるか？](#クラスタと通信できるか？)  \n",
    "**■症状**：APIの応答がない、または異常な内容"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc3776-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc3938-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc35aa-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "稼働しているElasticsearchに外部通信が許可されていない。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc3938-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc3afa-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc3776-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- セキュリティの設定等で、クラスタと外部との通信が阻害されていないか確認してください。  \n",
    "Elasticsearchは外部との通信用に**9200**番ポートを利用します。  \n",
    "9200番ポートが許可されているか確認してください。\n",
    "  - 例えば、AWSの場合はセキュリティグループの設定を確認してください。  \n",
    "  次のコマンドで \"FromPort\"と\"ToPort\"の値域に**9200**を含むブロックを確認してください。  \n",
    "  そのブロックの\"IpRanges\"配下に通信先のサーバが登録されていなければ、外部からの通信が遮断されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc3afa-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc3cbc-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc3938-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"SecurityGroups\": [\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!aws ec2 describe-security-groups\n",
    "\n",
    "#VPCのIDがわかっている場合は次のコマンドで表示を絞り込めます。\n",
    "#VPC_ID='vpc-bd3b81d9'\n",
    "#!aws ec2 describe-security-groups --filters \"Name=vpc-id,Values=$VPC_ID\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc3cbc-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc3e7e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc3afa-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### クラスタにNodeが参加していない\n",
    "**■項目**：[クラスタに全Nodeが参加しているか？](#クラスタに全Nodeが参加しているか？)  \n",
    "**■症状**：\"status\"がgreenでない、または \"number_of_data_nodes\"の数が、起動しているはずのData Node数より少ない"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc3e7e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc4040-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc3cbc-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "クラスタ内部の通信が阻害されており、検出できないNodeがある。  \n",
    "※ここまでの対処で、サーバもプロセスも稼働しており、pingも正常に応答している前提"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc4040-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc4216-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc3e7e-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- セキュリティの設定等で、クラスタ内部の通信が阻害されていないか確認してください。  \n",
    "Elasticsearchはクラスタ内部の通信用に**9300**番ポートを利用します。9300番ポートが許可されているか確認してください。\n",
    "  - 例えばAWSの場合はセキュリティグループの設定を確認してください。  \n",
    "  次のコマンドで \"FromPort\"と\"ToPort\"の値域に**9300**を含むブロックを確認してください。  \n",
    "  そのブロックの\"IpRanges\"配下に通信先のサーバが登録されていなければ、内部の通信が遮断されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc4216-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc43d8-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc4040-14ed-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"SecurityGroups\": [\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!aws ec2 describe-security-groups\n",
    "\n",
    "#VPCのIDがわかっている場合は次のコマンドで表示を絞り込めます。\n",
    "#VPC_ID='vpc-bd3b81d9'\n",
    "#!aws ec2 describe-security-groups --filters \"Name=vpc-id,Values=$VPC_ID\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc43d8-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc459a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc4216-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### エラーログが出力されている（ディスクフル）\n",
    "**■項目**：[エラーログが無いか？](#エラーログが無いか？)  \n",
    "**■症状**：ログに次のようなディスクフルを示すメッセージが出力されている"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc459a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc475c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc43d8-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "[cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc475c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc4928-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc459a-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "Elasticsearchのプロセスか、それ以外のプロセスによる使用量の増加。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc4928-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc4aea-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc475c-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "適切に「[ディスクの使用量に余裕がない](#ディスクの使用量に余裕がない)」の節での対処を実施している場合は、おそらくこの問題は既に解決しています。  \n",
    "「[ディスクの使用量に余裕がない](#ディスクの使用量に余裕がない)」の対処後、同様のログが出力されるか確認してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc4aea-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc4cac-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc4928-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### エラーログが出力されている（OSのメモリマップ設定誤り）\n",
    "**■項目**：[エラーログが無いか？](#エラーログが無いか？)  \n",
    "**■症状**：ログに次のようなOSのメモリマップ設定誤りを示すメッセージが出力されている"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc4cac-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc4e64-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc4aea-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "publish_address {192.168.0.1:9300}, bound_addresses {[::]:9300}\n",
    "Exception in thread \"main\" java.lang.RuntimeException: bootstrap checks failed\n",
    "initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap\n",
    "max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc4e64-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc5026-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc4cac-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "カーネルパラメータvm.max_map_countの設定が誤っています。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc5026-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc51f2-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc4e64-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "[01_04_Install.ipynbの「Elasticsearchの設定」](01_04_Install.ipynb#Elasticsearchの設定)にある「カーネルパラメータを変更」のコマンドを実行し、  \n",
    "カーネルパラメータvm.max_map_countを設定してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc51f2-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc53aa-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc5026-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### Slowログが出力されている\n",
    "**■項目**：[Slowログが無いか？](#Slowログが無いか？)  \n",
    "**■症状**：Slowログが出力されている"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc53aa-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc556c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc51f2-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因/対処**：  \n",
    "サーバのスペック不足、サーバ数の不足、ヒープの枯渇、インデックスの作成漏れ等が考えられます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc556c-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc572e-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc53aa-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "[診断](#診断)結果以外に、**Profile API**でクエリの実行状況を調査できます。  \n",
    "Profile APIは、通常の検索query内に  \n",
    "**\"profile\": true **  \n",
    "のパラメータを追加するだけで動作します。  \n",
    "\n",
    "例えば2015年1月分の東京のデータを検索するクエリでProfileを有効にするには次のようにします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc572e-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc5904-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc556c-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:22 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:23 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:24 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:25 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:26 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:27 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:28 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:29 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:30 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:31 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:32 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:33 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:34 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:35 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:36 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:37 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:38 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:39 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:40 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:41 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:42 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:43 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:44 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:45 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:46 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:47 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:48 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:49 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:50 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:51 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:52 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:53 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:54 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:55 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:56 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:57 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:58 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:00:59 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:00 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:01 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:02 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:03 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:04 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:05 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:06 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:07 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:08 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:09 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:10 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:11 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:12 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:13 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:14 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:15 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:16 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:17 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:18 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:19 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:20 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:21 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:22 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:23 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:24 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:25 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:26 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:27 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:28 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:29 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:30 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:31 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:32 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:33 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:34 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:35 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:36 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:37 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:38 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:39 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:40 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:41 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:42 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:43 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:44 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:45 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:46 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:47 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:48 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:49 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:50 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:51 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:52 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:53 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:54 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:55 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:56 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:57 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:58 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:01:59 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:02:00 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:02:01 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:02:02 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:02:03 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:02:04 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:02:05 --:--:--     0\r",
      "  0     0    0     0    0     0      0      0 --:--:--  0:02:06 --:--:--     0curl: (7) Failed to connect to ec2-54-238-216-212.ap-northeast-1.compute.amazonaws.com port 9200: Connection timed out\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/meteorological-data-2015.01/_search?pretty\" -H 'Content-Type:application/json' -d @- << EOF\n",
    "{\n",
    "  \"profile\": true,\n",
    "  \"query\": {\n",
    "    \"term\" : { \"location\" : \"tokyo\" } \n",
    "  }\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc5904-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc5ac6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc572e-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Profile APIの詳細は、Elasticsearch Referenceの[Profile API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html)を参照してください。\n",
    "\n",
    "ただし、Profileの結果は時間が立つと変化してしまうため注意が必要です。  \n",
    "これは、データがどのshardに入るのかは、データのハッシュ値によって変化するため、  \n",
    "クラスタ状況によってはデータの再配置が起きて、Shard間でデータの移動が行われる可能性があるためです。  \n",
    "（例：ある特定ノードで残容量がwater markを超過したためにshardの移動が発生した。など)  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc5ac6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc5c88-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc5904-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**[診断](#診断)結果とProfile結果を踏まえ、以下の節の内容を確認して対処してください。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc5c88-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc5e4a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc5ac6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "#### クラスタ内のサーバのCPU・メモリが枯渇している\n",
    "**■項目**：[クラスタ内のサーバのCPU・メモリが枯渇していないか？](#クラスタ内のサーバのCPU・メモリが枯渇していないか？)  \n",
    "**■症状**：CPUかメモリのどちらかが枯渇している"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc5e4a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc6002-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc5c88-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "サーバで動作しているElasticsearchかそれ以外のプロセスに対し、サーバのスペックが足りていない。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6002-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc61d8-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc5e4a-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- 他のプロセスによるCPUやメモリの使用率が大きい場合は、プロセスの停止か別サーバへの移動を検討してください。\n",
    "- ElasticsearchのCPU使用率が大きい場合は、サーバへのメモリ追加か、サーバ台数の追加を検討してください。\n",
    "  - AWSのインスタンスサイズを変更する場合は、AWS公式ページの[インスタンスのサイズ変更](http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/ec2-instance-resize.html)を参照してください。\n",
    "  - サーバ台数の追加は[01_50_Add_Data_Node.ipynb](01_50_Add_Data_Node.ipynb)を実施してください。\n",
    "- Elasticsearchのメモリ使用量は、Javaの起動オプションに設定した**\"-Xmx\"**（最大ヒープサイズ）の値を超えることはありません。  \n",
    "この使用量が想定外であった場合は次を確認してください。\n",
    "  1. 実際に稼働しているプロセスのオプションを確認します。  \n",
    "  診断の章の[プロセスが稼働しているか？](#プロセスが稼働しているか？)の節で実施したpsコマンドの出力に、Javaプロセスの引数が出ているはずです。\"-Xmx\" オプションの内容が想定通りか確認してください。\n",
    "  2. 想定と異なる場合、Elasticsearch用のJava起動オプションの設定が正しいか確認してください。次のコマンドで確認できます。  \n",
    "  再設定が必要な場合は[01_04_Install.ipynbの「Elasticsearchの設定」](01_04_Install.ipynb#Elasticsearchの設定)を実施してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc61d8-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc63a4-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6002-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m shell -a \"sudo grep -e '^-Xmx' /etc/elasticsearch/jvm.options\" -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc63a4-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc6566-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc61d8-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "#### ヒープが枯渇している\n",
    "**■項目**：[ヒープが枯渇していないか？](#ヒープが枯渇していないか？)  \n",
    "**■症状**：ヒープの使用量が100%近くに達している、またはGC数が大きい"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6566-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc6746-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc63a4-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "ヒープサイズの設定漏れ、またはクラスタの負荷状況にサーバ台数が及んでいない。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6746-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc6908-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6566-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- ヒープサイズの設定が想定通りか確認してください。\n",
    "  1. 実際に稼働しているプロセスのオプションを確認します。  \n",
    "  診断の章の[プロセスが稼働しているか？](#プロセスが稼働しているか？)の節で実施したpsコマンドの出力に、Javaプロセスの引数が出ているはずです。\"-Xmx\" オプションの内容が想定通りか確認してください。\n",
    "  2. 想定と異なる場合、Elasticsearch用のJava起動オプションの設定が正しいか確認してください。次のコマンドで確認できます。  \n",
    "  再設定が必要な場合は[01_04_Install.ipynbの「Elasticsearchの設定」](01_04_Install.ipynb#Elasticsearchの設定)を実施してください。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6908-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc6b1a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6746-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n",
      "  % self._get_c_name())\n",
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n",
      "-Xmx31g\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible all -m shell -a \"sudo grep -e '^-Xmx' /etc/elasticsearch/jvm.options\" -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6b1a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc6d04-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6908-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "- ヒープサイズが想定通りで、それでもヒープの使用量が100%に近くなってしまっている場合は、Data Nodeの追加を検討してください。  \n",
    "[01_50_Add_Data_Node.ipynb](01_50_Add_Data_Node.ipynb)を実施してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6d04-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6b1a-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "#### 実行中の重いタスクがある\n",
    "**■項目**：[実行中の重いタスクが無いか？](#実行中の重いタスクが無いか？)  \n",
    "**■症状**：想定外のタスクや実行時間の長いタスクがある、または大量のタスクが滞留している"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6d04-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "他のタスクがマシンリソースを圧迫し、クエリの応答を遅くしている。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc724a-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d"
      },
      {
       "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fba2a2-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d"
      },
      {
       "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc724a-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a6fc6d04-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- 呼び出されているタスクの種類や量が想定外の場合、Elasticsearchを呼び出している側のアプリケーションの挙動を確認してください。\n",
    "- 他のタスクがindexingであった場合、マシンへの負荷が高まり応答に時間がかかる場合があります。詳しくはElasticsearch Referenceの[Indexing Performance Tips](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/indexing-performance.html)を参照してください。\n",
    "- タスクがキャンセル処理をサポートしている場合、APIによるキャンセルが可能です。詳しくはElasticsearch Referenceの[Task Cancellation](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html#_task_cancellation)を参照してください。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6d04-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fc6d04-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc6b1a-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc7088-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "#### インデックスが作成されていない\n",
    "**■項目**：[インデックスが作成されているか？](#インデックスが作成されているか？)  \n",
    "**■症状**：インデックスが作成されていない"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6d04-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "インデックスでデータが管理されていないため、不要な範囲の検索が走行しており、遅くなっている。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fc724a-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d"
      },
      {
       "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d",
       "next": "a6fba2a2-14ed-11e7-b31a-0242ac12000d",
       "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a6fc724a-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- インデックスを設定し、インデックス名が想定通りになっているか（例えば日付インデックスであればインデックス名に日付文字列が入るか）を確認してください。  \n",
    "インデックスについては[05_Indexing.ipynb](05_Indexing.ipynb)を参照してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc724a-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc7416-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc7088-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "#### Shard数が足りていない\n",
    "**■項目**：[Shard数が設定通りか？](#Shard数が設定通りか？)  \n",
    "**■症状**：Shard数が設定したはずの数より不足している"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc7416-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc75ce-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc724a-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■原因**：  \n",
    "テンプレートによるShard数の設定漏れ。またはData Nodeを増やしたときにShard数も増やす処理を忘れている。  \n",
    "結果、存在するData Node群での検索処理が十分に並列化できず、遅くなっている。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc75ce-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc7790-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc7416-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■対処**：  \n",
    "- 作成済みのインデックスのシャード数は変更することができません。  \n",
    "シャード数を増やしたインデックスを作成した後、作成したインデックスにデータを移動させてください。(reindex)  \n",
    "次のコマンドを実行して新しいインデックスのシャード数を設定してください。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "collapsed": true,
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc7790-14ed-11e7-b31a-0242ac12000d",
     "next": "a6fc795c-14ed-11e7-b31a-0242ac12000d",
     "previous": "a6fc75ce-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "%%bash\n",
    "curl -XPUT \"http://<ホスト名>:<ポート番号>/_template/<テンプレート名>\" -d @- << EOF\n",
    "{\n",
    "  \"template\" : \"<インデックス名>\",\n",
    "  \"settings\": {\n",
    "    \"number_of_shards\": <シャード数>\n",
    "  }\n",
    "  \"mappings\": {\n",
    "    ...\n",
    "  }\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "lc_cell_meme": {
     "current": "a6fc795c-14ed-11e7-b31a-0242ac12000d",
     "next": null,
     "previous": "a6fc7790-14ed-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "- reindexを実行し、新しいインデックスにデータを移動します。  \n",
    "reindexについては[05_Indexing.ipynb](05_Indexing.ipynb#reindex-APIを使った投入後データの加工方法)を参考にしてください。  \n",
    "このreindexの説明ではデータを加工していますが、\"script\"や\"pipeline\"を指定しないようにすることで加工せずにデータの移動のみ行うことができます。"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  },
  "latex_envs": {
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0
  },
  "lc_notebook_meme": {
   "current": "a6fb9e56-14ed-11e7-b31a-0242ac12000d",
   "lc_server_signature": {
    "history": []
   }
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "117px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_position": {
    "height": "840px",
    "left": "0px",
    "right": "1168px",
    "top": "130px",
    "width": "268px"
   },
   "toc_section_display": "block",
   "toc_window_display": true,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}