自然语言处理（NLP）基础知识分享系列（12）.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5ad35e0e",
   "metadata": {},
   "source": [
    "## `在Huggingface transformers平台上微调BERT-wwm-ext`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67d432bb",
   "metadata": {},
   "source": [
    "今天是本系列的最后一期。\n",
    "\n",
    "### transformer 与 transformers \n",
    "\n",
    "为了避免命名带来的混淆，我们首先来厘清一下：transformer 与 transformers。 \n",
    "\n",
    "#### • transformer \n",
    "\n",
    "在上一期里，我们已经做过介绍，transformer是一种具有多头自注意力机制的、可以取代RNN/LSTM的神经网络单元结构。 \n",
    "\n",
    "***本质上它是一种深度学习技术。*** \n",
    "\n",
    "#### • transformers\n",
    "\n",
    "今天提到的transformers，是Huggingface🤗公司开发的一套python库包，它提供一个平台框架，使得transformer技术实现的各类模型能通过一致化的接口方式呈现和调用。在其主页上它是这么自我介绍的：\n",
    "\n",
    "🤗 Transformers 提供 API 来轻松下载和训练最先进的预训练模型。使用预训练模型可以降低您的计算成本、碳足迹，并节省您从头开始训练模型的时间。这些模型可用于不同的模式，例如：\n",
    "\n",
    "📝 文本：超过 100 种语言的文本分类、信息提取、问答、摘要、翻译和文本生成。<br>\n",
    "🖼️ 图像：图像分类、对象检测和分割。<br>\n",
    "🗣️ 音频：语音识别和音频分类。<br>\n",
    "🐙 多模态：表格问答、光学字符识别、从扫描文档中提取信息、视频分类和视觉问答。<br>\n",
    "该库支持三个最流行的深度学习库之间的无缝集成：PyTorch、TensorFlow和JAX。在一个框架中用三行代码训练模型，然后加载它以与另一个框架进行推理。\n",
    "\n",
    "每个🤗 Transformers 架构都在独立的 Python 模块中定义，因此可以轻松定制它们以进行研究和实验。\n",
    "\n",
    "***本质上它是一种研发工具产品。*** "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f07801f",
   "metadata": {},
   "source": [
    "### BERT-wwm-ext \n",
    "\n",
    "Google在发布公开论文『Attention Is All You Need』的同时，推出了开源的transformer架构**BERT**（Bidirectional Encoder Representation from Transformers）。可以说BERT开辟了NLP的新时代，产学研界参考BERT推出了很多类似的变种模型，如：**RoBERTa、BART**等。 \n",
    "\n",
    "在中文的类**BERT**研究中，除了Google原生的**bert-base-chinese**之外，**Bert-wwm、MacBert、ChineseBert**等模型创新了不同的机制进行优化，具有比较大的知名度和影响力。 \n",
    "\n",
    "我们今天将基于哈工大与科大讯飞联合实验室（HFL）研发的**Bert-wwm-ext**进行微调，实现我们的场景任务目标——辨别新闻标题A和B的关系分类。\n",
    "与原始的**bert-base-chinese**中文采用单字掩码方式不同，**Bert-wwm-ext**提供了所谓全词掩码（Whole Word Masking）的预训练方式。如下图所示：\n",
    "![这是图片](imgs/masks.png \"Masks\")\n",
    "\n",
    "全词掩码有两个优点：<br>\n",
    "1、部分解决了MLM独立性假设，使得预测token之间拥有了一定的关联性<br>\n",
    "2、提高了MLM任务难度，使得模型需要更多依赖远距离的上下文来判断掩码部分的内容<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c924598",
   "metadata": {},
   "source": [
    "接下来将看到，使用🤗 Transformers 库将使我们的代码非常简化：虽然**Bert-wwm-ext** 的网络结构的复杂程度要远远超过前面的孪生LSTM网络，但实现的代码行数却会大大降低。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e7439139",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\hp\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\huggingface_hub\\utils\\_deprecation.py:92: FutureWarning: Deprecated argument(s) used in 'snapshot_download': ignore_regex. Will not be supported from version '0.12'.\n",
      "\n",
      "Please use `allow_patterns` and `ignore_patterns` instead.\n",
      "  warnings.warn(message, FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\HP/.cache\\\\huggingface\\\\hub\\\\models--hfl--chinese-bert-wwm-ext\\\\snapshots\\\\2a995a880017c60e4683869e817130d8af548486'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from huggingface_hub import snapshot_download\n",
    "#从官方的huggingface_hub下载模型配置、参数、模型词库等信息\n",
    "\n",
    "snapshot_download(repo_id=\"hfl/chinese-bert-wwm-ext\", ignore_regex=[\"*.h5\", \"*.ot\", \"*.msgpack\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9f1ed25d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at C:\\Users\\HP/.cache\\huggingface\\hub\\models--hfl--chinese-Bert-wwm-ext\\snapshots\\2a995a880017c60e4683869e817130d8af548486 were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at C:\\Users\\HP/.cache\\huggingface\\hub\\models--hfl--chinese-Bert-wwm-ext\\snapshots\\2a995a880017c60e4683869e817130d8af548486 and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoConfig,AutoTokenizer,AutoModel,AutoModelForSequenceClassification\n",
    "model_name = 'C:\\\\Users\\\\HP/.cache\\\\huggingface\\\\hub\\\\models--hfl--chinese-Bert-wwm-ext\\\\snapshots\\\\2a995a880017c60e4683869e817130d8af548486' \n",
    "config = AutoConfig.from_pretrained(model_name) \n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name) \n",
    "\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=3) #新闻标题A和B的关系标签有3种类型 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "245d4cae",
   "metadata": {},
   "source": [
    "先通过一个小例子看看🤗 Transformers的 **tokenizer** 是如何工作的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7230dcaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_ab = [\n",
    "    (\"我今天输液了\", \"输什么液?\"),\n",
    "    (\"让我好好爱你行不？\", \"让我陪你一起过日子好不？\")\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "30e565cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[ 101, 2769,  791, 1921, 6783, 3890,  749,  102, 6783,  784,  720, 3890,\n",
       "          136,  102,    0,    0,    0,    0,    0,    0],\n",
       "        [ 101, 6375, 2769, 1962, 1962, 4263,  872, 6121,  679,  102, 6375, 2769,\n",
       "         7373,  872,  671, 6629, 6814, 3189, 2094,  102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],\n",
       "        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],\n",
       "        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_input = tokenizer(list_ab, padding=True, truncation=True, max_length=20, return_tensors='pt')\n",
    "encoded_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "33890666",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[CLS] 让 我 好 好 爱 你 行 不 [SEP] 让 我 陪 你 一 起 过 日 子 [SEP]'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(encoded_input[\"input_ids\"][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc293482",
   "metadata": {},
   "source": [
    "在此label 不需要人工做 **One-hot** 编码，后续模型训练中会自动化进行相关处理。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f5a67323",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_to_index = { 'unrelated' : 0 , 'agreed' : 1 , 'disagreed' : 2 } \n",
    "label_pipeline = lambda x : label_to_index [ x ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "03250960",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 60 \n",
    "TEST_SPLIT = 0.1\n",
    "\n",
    "BATCH_SIZE = 64 \n",
    "LEARNING_RATE = 2e-5 \n",
    "EPOCHS=2 \n",
    "WEIGHT_DECAY=0.01 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0bb4822",
   "metadata": {},
   "source": [
    "**Huggningface transformers** 与**Huggingface** 自己的数据集处理包**datasets** 集成较好。 \n",
    "**Bert-wwm-ext** 将直接处理中文文本语料，因此之前的结巴分词结果也不再需要，我们将使用**pandas** 的**read_csv** 函数 + **datasets** 包里的**load_dataset** 函数加载和处理原始的**.csv** 。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "47e4221c",
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_CSV_PATH = \"./train.csv\" \n",
    "import pandas as pd \n",
    "train = pd.read_csv(TRAIN_CSV_PATH, index_col = 0) \n",
    "cols = ['title1_zh', 'title2_zh', 'label'] \n",
    "\n",
    "train = train.loc[:, cols].fillna('') \n",
    "train.rename(columns={'label':'label_class'},inplace=True) # 重要！字段名称里需要空出transformers规定的「label」保留字 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "594b45ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import Dataset \n",
    "dataset = Dataset.from_pandas(train) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "04df51b1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['title1_zh', 'title2_zh', 'label_class', 'id'],\n",
       "    num_rows: 320552\n",
       "})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "107bb3e4",
   "metadata": {},
   "source": [
    "对数据做些基本的处理，生成成对的新闻标题A和B的文本，以及数字化的关系分类标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "62a00798",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bea0695d32e4fd6a1377be1e5cd2b85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/321 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def tokenize_function(examples): \n",
    "    #print(len(examples[\"label_class\"])) \n",
    "    #print(examples) \n",
    "    labels = [label_pipeline(label) for label in examples[\"label_class\"]] \n",
    "    #print(labels) \n",
    "       \n",
    "    texts = [(examples[\"title1_zh\"][i],examples[\"title2_zh\"][i]) for i in range(len(examples[\"label_class\"]))] \n",
    "    #print(texts) \n",
    "\n",
    "    tokenized = tokenizer(texts, padding='max_length', truncation=True, max_length=MAX_LEN) \n",
    "    tokenized['label'] = labels #transformers的模型通过「label」字段传递分类标签 \n",
    "    return tokenized \n",
    "\n",
    "tokenized_datasets = dataset.map(tokenize_function, batched=True) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78d681b6",
   "metadata": {},
   "source": [
    "手写一个简单的数据集分割功能，因为是cpu训练，为节省训练时间，我们仅使用跟测试验证集同样多的、占总体10%的样本进行模型微调训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3b213f97",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_size = len(tokenized_datasets['label']) \n",
    "test_size = int(TEST_SPLIT * dataset_size) \n",
    "train_size = dataset_size - test_size \n",
    "train_dataset = tokenized_datasets.shuffle(seed=42).select(range(test_size)) #为节省训练时间，仅使用10%的样本训练 \n",
    "test_dataset = tokenized_datasets.shuffle(seed=42).select(range(test_size, 2* test_size)) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "96738609",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['title1_zh', 'title2_zh', 'label_class', 'id', 'input_ids', 'token_type_ids', 'attention_mask', 'label'],\n",
       "    num_rows: 32055\n",
       "})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a40a5614",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "# 将模型和数据转移到cuda, 若无cuda,可更换为cpu \n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\") \n",
    "model = model.to(device) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe52f4a",
   "metadata": {},
   "source": [
    "Trainer在训练期间不会自动评估模型性能。需要向Trainer传递一个函数来计算和报告指标。🤗 Datasets 库提供了一个简单的函数 **load_metric** 加载。 \n",
    "定义一个compute_metric计算预测的准确性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8bbad1c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "from datasets import load_metric \n",
    "\n",
    "metric = load_metric(\"accuracy\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "245861e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(eval_pred): \n",
    "    logits, labels = eval_pred \n",
    "    predictions = np.argmax(logits, axis=-1) \n",
    "    return metric.compute(predictions=predictions, references=labels) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cd7d3bc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments, Trainer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc06ab6f",
   "metadata": {},
   "source": [
    "此时，只剩下三个步骤：\n",
    "\n",
    "在**TrainingArguments** 中定义训练超参数。\n",
    "将训练参数连同模型、数据集、标记器和数据整理器一起传递给**Trainer** 。\n",
    "调用**train()** 来微调模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "da6a1690",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The following columns in the training set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: title2_zh, title1_zh, label_class, id. If title2_zh, title1_zh, label_class, id are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.\n",
      "c:\\users\\hp\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\transformers\\optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  FutureWarning,\n",
      "***** Running training *****\n",
      "  Num examples = 32055\n",
      "  Num Epochs = 2\n",
      "  Instantaneous batch size per device = 64\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 64\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 1002\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='1002' max='1002' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1002/1002 7:32:16, Epoch 2/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.331700</td>\n",
       "      <td>0.286503</td>\n",
       "      <td>0.875838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.214100</td>\n",
       "      <td>0.286773</td>\n",
       "      <td>0.882546</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./results\\checkpoint-500\n",
      "Configuration saved in ./results\\checkpoint-500\\config.json\n",
      "Model weights saved in ./results\\checkpoint-500\\pytorch_model.bin\n",
      "The following columns in the evaluation set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: title2_zh, title1_zh, label_class, id. If title2_zh, title1_zh, label_class, id are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 32055\n",
      "  Batch size = 64\n",
      "Saving model checkpoint to ./results\\checkpoint-1000\n",
      "Configuration saved in ./results\\checkpoint-1000\\config.json\n",
      "Model weights saved in ./results\\checkpoint-1000\\pytorch_model.bin\n",
      "The following columns in the evaluation set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: title2_zh, title1_zh, label_class, id. If title2_zh, title1_zh, label_class, id are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 32055\n",
      "  Batch size = 64\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=1002, training_loss=0.2726947990303744, metrics={'train_runtime': 27155.7832, 'train_samples_per_second': 2.361, 'train_steps_per_second': 0.037, 'total_flos': 1976742329360400.0, 'train_loss': 0.2726947990303744, 'epoch': 2.0})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_args = TrainingArguments( \n",
    "    output_dir=\"./results\", \n",
    "    evaluation_strategy=\"epoch\", \n",
    "    learning_rate=LEARNING_RATE, \n",
    "    per_device_train_batch_size=BATCH_SIZE, \n",
    "    per_device_eval_batch_size=BATCH_SIZE, \n",
    "    num_train_epochs=EPOCHS, \n",
    "    weight_decay=WEIGHT_DECAY, \n",
    ") \n",
    "\n",
    "trainer = Trainer( \n",
    "    model=model, \n",
    "    args=training_args, \n",
    "    train_dataset=train_dataset, \n",
    "    eval_dataset=test_dataset, \n",
    "    compute_metrics=compute_metrics, \n",
    ") \n",
    "\n",
    "trainer.train() "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95fca2a0",
   "metadata": {},
   "source": [
    "经过2轮训练之后的准确率是<font color=\"#dd0000\">88.3%</font>，比之前最佳的孪生LSTM模型<font color=\"#dd0000\">83.1%</font>又提升了超过5个点，而这只是使用了10%的样本进行微调训练，而非LSTM模型使用90%的样本训练！ \n",
    "\n",
    "---\n",
    "\n",
    "至此，本系列分享完毕。 "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}