03_Faults.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Faults and Disruptions\n",
    "\n",
    "\n",
    "\n",
    "So far we've been experimenting with a *perfect world* assumption. Although this is useful to test your systems, it doesn't suffice as a peer-to-peer network model. In this notebook we will relax this assumption, introducing distortions and faults.\n",
    "\n",
    "In P2PSimpy distortions are modeled as a `Runner`.  \n",
    "\n",
    "`BaseDisruption` is a base class to model periodical incidental failures and slowdowns.  It is modeled as a runner testing once per `interval` a status change. The status change is modeled as uniform random sampling with a mean time between disruption specified with `mtbf` parameter. Finally, the parameter `availability` affects the time each peer is disrupted. For example, if `availability=0.9` the peer will be not disrupted 90 % of the time.\n",
    "\n",
    "Each inherited class must implement two functions: `disruption_start` and `disruption_end`.  \n",
    "Out of the box there two classes available: `Slowdown` and `Downtime`:\n",
    "- `Slowdown` - temporarily reduces bandwidth, which in turn affects the message latencies. The slowdown effect can be specified with `slowdown` parameter (from `0` to `1.0`). This class models [bandwidth throttling](https://en.wikipedia.org/wiki/Bandwidth_throttling).\n",
    "- `Downtime` - temporarily deactivates the peer. This models offline status of a peer with both gracefull exit and crashes. Every connection is restored once the peer is back online.  You can read about [Fault tolerance](https://en.wikipedia.org/wiki/Fault_tolerance).\n",
    "\n",
    "Let's repeat the previous notebook but with crashes and slowdowns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:25.859951Z",
     "start_time": "2020-03-27T12:35:25.793935Z"
    }
   },
   "outputs": [],
   "source": [
    "# Initialize the experiment:\n",
    "from p2psimpy.config import *\n",
    "import networkx as nx\n",
    "\n",
    "from p2psimpy.services.connection_manager import BaseConnectionManager\n",
    "import networkx as nx\n",
    "\n",
    "from p2psimpy.simulation import Simulation\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# Load the previous experiment configurations\n",
    "exper = Simulation.load_experiment(expr_dir='gossip_expr')\n",
    "\n",
    "Locations, topology, peer_services, service_impl = exper\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to assign implementations of following services:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:25.867824Z",
     "start_time": "2020-03-27T12:35:25.861905Z"
    }
   },
   "outputs": [],
   "source": [
    "service_impl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:25.877012Z",
     "start_time": "2020-03-27T12:35:25.870429Z"
    }
   },
   "outputs": [],
   "source": [
    "# We will use default services - but fill free to replace with your services \n",
    "\n",
    "from p2psimpy.services.gossip import GossipService\n",
    "from p2psimpy.services.message_producer import MessageProducer\n",
    "\n",
    "service_impl['BaseConnectionManager'] = BaseConnectionManager\n",
    "service_impl['MessageProducer'] = MessageProducer\n",
    "service_impl['GossipService'] = GossipService\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-21T09:58:41.768297Z",
     "start_time": "2020-03-21T09:58:41.755982Z"
    }
   },
   "source": [
    "## Adding random crashes\n",
    "\n",
    "There are two options on how to model downtime, crashes and disruptions: \n",
    " - Scheduled disruptions \n",
    " - Random disruptions \n",
    "\n",
    "*Scheduled disruptions* is a runner with a list of timeouts. For example, `schedule = [100, 200, 100, 300]`, will run first disruption at the time `100`, and it will be disrupted until time `300`, next disruption starts at time `400` and ends at the time `700`.     \n",
    "\n",
    "*Random disruptions* will insert disruptions modeled with a statistical distribution. \n",
    "There are three parameters that need to be filled, i.e., `start_time, disruption_time, disruption_intervals`. `disruption_time` models time of a disruption, `disruption_invervals` is the time between disruptions, `start_time` is an initial delay to start disruptions. Each of attributes can be one of three types: \n",
    " - Constant value - in this case all peer will get the same value. For example, with `start_time = 100` every peer with this configuration will be delayed by `100 ms`; \n",
    " - `Dist` object: in this case every peer be initialized with a random value with a given distribution. For example, with `start_time=Dist('norm', (100, 20))` peers will sample one value from the normal distribution with an average 100 and the standard deviation of 10 ([documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html#scipy.stats.norm));\n",
    " - `DistAttr` object: every peer will get the distribution itself. In this case service can use value that will change dynamically. For example, `start_time=DistAttr('norm', (100, 20))` every peer will get a value = `Dist('norm', (100, 20))`.  \n",
    "\n",
    "Let's use `DistAttr` object to model `RandomDowntime`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(peer_services['client'].service_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:25.889219Z",
     "start_time": "2020-03-27T12:35:25.879086Z"
    }
   },
   "outputs": [],
   "source": [
    "from p2psimpy.services.disruption import RandomDowntime\n",
    "\n",
    "class DowntimeConfig(Config):\n",
    "    start_time = DistAttr('norm', (1500, 200))\n",
    "    disruption_time = DistAttr('norm', (800, 200))\n",
    "    disruption_intervals = DistAttr('norm', (200, 100))\n",
    "\n",
    "peer_services['peer'].service_map[RandomDowntime] = DowntimeConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing peers connections \n",
    "\n",
    "Let's see how model would execute connections/disconnections with a simple animation. When peer is disconnected it's color will turn red and all edges will not be displayed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pygraphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:48:34.715255Z",
     "start_time": "2020-03-27T12:48:33.064838Z"
    }
   },
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.animation\n",
    "from networkx.drawing.nx_pydot import graphviz_layout\n",
    "import seaborn as sns\n",
    "from IPython.display import HTML\n",
    "\n",
    "sns.set()\n",
    "\n",
    "def visalize_graph(G, ax=None):\n",
    "    master_nodes = [n for (n,ty) in \\\n",
    "    nx.get_node_attributes(G,'type').items() if ty == 'peer']\n",
    "    client_nodes = [n for (n,ty) in \\\n",
    "        nx.get_node_attributes(G,'type').items() if ty == 'client']\n",
    "    online_map = {n:ty for (n,ty) in nx.get_node_attributes(G,'is_online').items()}\n",
    "    \n",
    "    colors = ['blue' if online_map[k] else 'red' for k in master_nodes]\n",
    "    \n",
    "    nx.draw_networkx_nodes(G, pos, nodelist=master_nodes, \\\n",
    "        node_color=colors, node_shape='o', node_size=500, ax=ax)\n",
    "    nx.draw_networkx_nodes(G, pos, nodelist=client_nodes,  \\\n",
    "        node_color='green', node_shape='^', node_size=100, label=1, ax=ax)\n",
    "    nx.draw_networkx_labels(G, pos, labels={k:k for k in master_nodes}, font_color='w', ax=ax)\n",
    "    \n",
    "    nx.draw_networkx_edges(G, pos, edgelist=G.subgraph(master_nodes).edges(), width=1.5, ax=ax)\n",
    "    nx.draw_networkx_edges(G, pos, edgelist=G.edges(nbunch=client_nodes),  style='dotted', ax=ax)\n",
    "\n",
    "    \n",
    "\n",
    "# Init Graph\n",
    "sim = Simulation(Locations, topology, peer_services, service_impl, seed=42)\n",
    "G = sim.get_graph()\n",
    "pos = graphviz_layout(G)\n",
    "\n",
    "# Build plot\n",
    "fig, ax = plt.subplots(figsize=(10,10))\n",
    "\n",
    "def update(num):\n",
    "    if num !=0:\n",
    "        sim.run(num)\n",
    "    G = sim.get_graph()\n",
    "\n",
    "    ax.clear()\n",
    "    visalize_graph(G, ax)\n",
    "    # Scale plot ax\n",
    "    ax.set_title(\"Frame %d: \"%(num+1), fontweight=\"bold\")\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "\n",
    "\n",
    "ani = matplotlib.animation.FuncAnimation(fig, update, frames=range(0, 6100, 100))\n",
    "HTML(ani.to_jshtml())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:42.323209Z",
     "start_time": "2020-03-27T12:35:42.270220Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def total_delay(sim, peer_id, storage_name):\n",
    "    store = sim.peers[peer_id].get_storage(storage_name).txs\n",
    "    for k, peer_time in store.items():\n",
    "        client_id, msg_num = k.split('_')\n",
    "        client_id = int(client_id)\n",
    "        msg_num = int(msg_num)\n",
    "        client_time = sim.peers[client_id].storage[storage_name].txs[k]\n",
    "        yield (int(msg_num), peer_time - client_time)\n",
    "        \n",
    "def get_gossip_table(sim, storage_name):\n",
    "    return pd.DataFrame({k: dict(total_delay(sim, k, storage_name)) \n",
    "                         for k in sim.types_peers['peer']}).sort_index()\n",
    "\n",
    "df = get_gossip_table(sim, 'msg_time')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:43.509231Z",
     "start_time": "2020-03-27T12:35:42.327908Z"
    }
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.set()\n",
    "\n",
    "def show_heat_map(df):\n",
    "    plt.figure(figsize=(12, 7))\n",
    "    ax = plt.axes()\n",
    "\n",
    "    sns.heatmap(df, ax=ax)\n",
    "\n",
    "    plt.xlabel('Peer number', fontsize = 12) # x-axis label with fontsize 15\n",
    "    plt.ylabel('Message number', fontsize = 12) # y-axis label with fontsize 15\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "show_heat_map(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:44.522384Z",
     "start_time": "2020-03-27T12:35:43.516481Z"
    }
   },
   "outputs": [],
   "source": [
    "# Show average overhead on message\n",
    "\n",
    "def calc_overhead(sim, peer_id, storage_name):\n",
    "    store = sim.peers[peer_id].storage[storage_name].times_seen\n",
    "    for k, times in store.items():\n",
    "        msg_num, client_id = k.split('_')\n",
    "        yield (int(msg_num), int(times))\n",
    "    \n",
    "def get_overhead_table(sim, storage_name):\n",
    "    return pd.DataFrame({k: dict(calc_overhead(sim, k, storage_name)) \n",
    "                         for k in sim.types_peers['peer']}).sort_index()\n",
    "\n",
    "def show_overhead_hist(overhead_table):\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.distplot(overhead_table, norm_hist=True)\n",
    "    plt.xlabel('Overhead (message seen by same peer)', fontsize=15)\n",
    "\n",
    "def get_monitor_time_table(sim, data='bytes'):\n",
    "    if data == 'bytes':\n",
    "        data_func = lambda x: x.bytes_load  \n",
    "    elif data == 'msg_count':\n",
    "        data_func = lambda x: x.msg_count_load  \n",
    "\n",
    "    return pd.DataFrame({k: data_func(sim.peers[k])\n",
    "                         for k in sim.types_peers['peer']}).sort_index()\n",
    "\n",
    "def show_bandwidth_time(tt_df):\n",
    "    tt_df2 = tt_df\\\n",
    "        .stack()\\\n",
    "        .reset_index()\\\n",
    "        .rename(columns={'level_0':'time', 'level_1':'peer_id', 0: 'data'})\n",
    "\n",
    "    plt.figure(figsize=(10,6))\n",
    "    ax = sns.lineplot(x='time', y='data', data=tt_df2, \n",
    "                 ci='sd', estimator=\"median\",)\n",
    "\n",
    "    ax.set_title('Bandwidth overhead', fontsize= 20 )\n",
    "    ax.set_xlabel('Time (s)', fontsize=12)\n",
    "    tl = ax.set_ylabel('Overhead per second (bytes/sec)', fontsize=12)\n",
    "    \n",
    "def show_msg_count_time(tt_df):\n",
    "    tt_df2 = tt_df\\\n",
    "        .stack()\\\n",
    "        .reset_index()\\\n",
    "        .rename(columns={'level_0':'time', 'level_1':'peer_id', 0: 'data'})\n",
    "\n",
    "    plt.figure(figsize=(10,6))\n",
    "    ax = sns.lineplot(x='time', y='data', data=tt_df2, \n",
    "                 ci='sd', estimator=\"median\",)\n",
    "\n",
    "    ax.set_title('Msg count overhead', fontsize= 20 )\n",
    "    ax.set_xlabel('Time (s)', fontsize=12)\n",
    "    tl = ax.set_ylabel('Overhead per second (count/sec)', fontsize=12)\n",
    "\n",
    "\n",
    "oh = get_overhead_table(sim, 'msg_time')\n",
    "show_heat_map(oh)            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:44.965607Z",
     "start_time": "2020-03-27T12:35:44.526112Z"
    }
   },
   "outputs": [],
   "source": [
    "show_overhead_hist(oh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workload overhead per second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:45.422713Z",
     "start_time": "2020-03-27T12:35:44.968058Z"
    }
   },
   "outputs": [],
   "source": [
    "bw_oh_tt = get_monitor_time_table(sim)\n",
    "show_bandwidth_time(bw_oh_tt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:45.873668Z",
     "start_time": "2020-03-27T12:35:45.426174Z"
    }
   },
   "outputs": [],
   "source": [
    "mc_oh_tt = get_monitor_time_table(sim, data='msg_count')\n",
    "show_msg_count_time(mc_oh_tt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see not all peers now get the message. This is because some peers might be offline at the moment of disseminating the message. This can lead to the message loss.\n",
    "\n",
    "One of the most critical failures is the failure of a peer directly connected to the client.  \n",
    "\n",
    "Here are some common ways to ensure that client will be connected and message won't be lost:\n",
    "- Client must be connected to multiple peers chosen from a diverse group of peers. This minimize the probability that client is connected to a faulty or malicious peer.\n",
    "- Message should have some meta-data indicators to quickly make sense what is missing. Additionally, if message is depended upon previous messages it includes reference pointers. The simplest index is a *client message counter*. Peer can easily identify if they are missing some messages by going through the range. \n",
    "- To ensure *\"eventual consistency\"* [TODO add link] peer needs to periodically synchronize the data with each other. Example of such a process is sometimes called *set reconciliation*, *anti-entropy*. This commonly used in *AP* databases [TODO add link]. \n",
    "\n",
    "\n",
    "\n",
    "# More client connections!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:45.882965Z",
     "start_time": "2020-03-27T12:35:45.876084Z"
    }
   },
   "outputs": [],
   "source": [
    "from random import sample\n",
    "\n",
    "num_cons = 5 # connect to 5 peers \n",
    "\n",
    "for c in sim.types_peers['client']:\n",
    "    for p in sample(list(sim.types_peers['peer']), num_cons):\n",
    "        topology.add_edge(c, p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gossip with periodical synchronization \n",
    "\n",
    "\n",
    "Let's ensure that all the peers will eventually receive all data from a client. First, we will connect `client` to multiple peers. Second, we will replace simple gossip with a pull based gossip. Pull-based gossip is based on periodical synchronization by exchanging views with their direct neighbors. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:46.477982Z",
     "start_time": "2020-03-27T12:35:45.885361Z"
    }
   },
   "outputs": [],
   "source": [
    "from p2psimpy.services.gossip import PullGossipService, RangedPullGossipService\n",
    "\n",
    "service_impl['GossipService'] = RangedPullGossipService\n",
    "\n",
    "sim_pull = Simulation(Locations, topology, peer_services, service_impl)\n",
    "sim_pull.run(5_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Average receive time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:46.524383Z",
     "start_time": "2020-03-27T12:35:46.479876Z"
    }
   },
   "outputs": [],
   "source": [
    "df_pl = get_gossip_table(sim_pull, 'msg_time')\n",
    "df_pl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see it takes more time on average to reach a peer, namely: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:46.573132Z",
     "start_time": "2020-03-27T12:35:46.529960Z"
    }
   },
   "outputs": [],
   "source": [
    "change = df_pl.mean().mean()/ df.mean().mean()\n",
    "change "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overhead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:47.096719Z",
     "start_time": "2020-03-27T12:35:46.576418Z"
    }
   },
   "outputs": [],
   "source": [
    "oh_pl = get_overhead_table(sim_pull, 'msg_time')\n",
    "show_overhead_hist(oh_pl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:47.797516Z",
     "start_time": "2020-03-27T12:35:47.099190Z"
    }
   },
   "outputs": [],
   "source": [
    "show_heat_map(oh_pl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overhead over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:48.290100Z",
     "start_time": "2020-03-27T12:35:47.799884Z"
    }
   },
   "outputs": [],
   "source": [
    "bw_oh_tt_pull = get_monitor_time_table(sim_pull)\n",
    "show_bandwidth_time(bw_oh_tt_pull)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:48.775056Z",
     "start_time": "2020-03-27T12:35:48.292168Z"
    }
   },
   "outputs": [],
   "source": [
    "bw_oh_tt_pull = get_monitor_time_table(sim_pull)\n",
    "show_bandwidth_time(bw_oh_tt_pull)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:48.785672Z",
     "start_time": "2020-03-27T12:35:48.777913Z"
    }
   },
   "outputs": [],
   "source": [
    "bw_oh_tt_pull.mean(axis=1).mean() / bw_oh_tt.mean(axis=1).mean() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:49.303743Z",
     "start_time": "2020-03-27T12:35:48.791465Z"
    }
   },
   "outputs": [],
   "source": [
    "mc_oh_tt_pull = get_monitor_time_table(sim_pull, data='msg_count')\n",
    "show_msg_count_time(mc_oh_tt_pull)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:35:49.326537Z",
     "start_time": "2020-03-27T12:35:49.318541Z"
    }
   },
   "outputs": [],
   "source": [
    "mc_oh_tt_pull.mean(axis=1).mean() / mc_oh_tt.mean(axis=1).mean() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pull based gossip takes more time to converge on average, but enjoys a lesser overhead. \n",
    "\n",
    "The main ideas of such gossip is to periodically exchange indices of messages instead of messages itself. When peers discover that some messages are missing (by inspecting received indexes) they request missing messages by their indices. This approach saves bandwidth and generally achieves higher throughput (msg per second) with a trade-off of latency.   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-27T12:52:37.474081Z",
     "start_time": "2020-03-27T12:52:37.419875Z"
    }
   },
   "outputs": [],
   "source": [
    "sim_pull.save_experiment(expr_dir='crash_gossip', include_module_classes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More experiments \n",
    "\n",
    "Test your gossip from the previous notebook. How good it can handle periodical outages? \n",
    "\n",
    "Try different topologies? Will the gossip work for all the topologies? \n",
    "\n",
    "Report your findings here:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}