Jigsaw Multilingual Toxic Comment Classification.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Acknowledgements:\n- https://www.kaggle.com/tarunpaparaju/jigsaw-multilingual-toxicity-eda-models#Introduction\n- https://github.com/dipanjanS/deep_transfer_learning_nlp_dhs2019\n\nRun it on [Kaggle Kernels](https://www.kaggle.com/spsayakpaul/jigsaw-multilingual-toxic-comment-classification). "},{"metadata":{"id":"N0PbF_7bca25"},"cell_type":"markdown","source":"In this notebook, I am going to build a baseline model based on [DistilBERT](https://medium.com/huggingface/distilbert-8cf3380435b5) for the Jigsaw Multilingual Toxic Comment Classification (Kaggle challenge [link](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification)). \n\n**What am I predicting?** (comes from the challenge homepage)\n\nYou are predicting the probability that a comment is toxic. A toxic comment would receive a 1.0. A benign, non-toxic comment would receive a 0.0. In the test set, all comments are classified as either a 1.0 or a 0.0."},{"metadata":{"id":"9npbakvTcSg8","outputId":"11d91e22-b482-4c32-82e2-15c6fa4d1629","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nprint(tf.__version__)","execution_count":1,"outputs":[{"output_type":"stream","text":"2.1.0\n","name":"stdout"}]},{"metadata":{"id":"YKrrMbWAd_fX"},"cell_type":"markdown","source":"An amazing EDA on the dataset in available here: https://www.kaggle.com/tarunpaparaju/jigsaw-multilingual-toxicity-eda-models. "},{"metadata":{"id":"ZzPnhW-ed9Ja"},"cell_type":"markdown","source":"## Load and prepare data"},{"metadata":{"id":"JDmH46e-c4Yz","outputId":"5907bd56-f994-49c8-f4f5-bee810b1bd83","trusted":true},"cell_type":"code","source":"!ls /kaggle/input/jigsaw-multilingual-toxic-comment-classification/","execution_count":2,"outputs":[{"output_type":"stream","text":"jigsaw-toxic-comment-train-processed-seqlen128.csv\r\njigsaw-toxic-comment-train.csv\r\njigsaw-unintended-bias-train-processed-seqlen128.csv\r\njigsaw-unintended-bias-train.csv\r\nsample_submission.csv\r\ntest-processed-seqlen128.csv\r\ntest.csv\r\nvalidation-processed-seqlen128.csv\r\nvalidation.csv\r\n","name":"stdout"}]},{"metadata":{"id":"WOIVOHMxdlGO"},"cell_type":"markdown","source":"Data description is available [here](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/data). "},{"metadata":{"id":"gR66Hwq1dei3","trusted":true},"cell_type":"code","source":"# Load datasets\nimport pandas as pd\nimport os\n\nDATA_PATH = \"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/\"\n\nTEST_PATH = os.path.join(DATA_PATH, \"test.csv\")\nVAL_PATH = os.path.join(DATA_PATH, \"validation.csv\")\nTRAIN_PATH = os.path.join(DATA_PATH, \"jigsaw-toxic-comment-train.csv\")\n\nval_data = pd.read_csv(VAL_PATH)\ntest_data = pd.read_csv(TEST_PATH)\ntrain_data = pd.read_csv(TRAIN_PATH)","execution_count":3,"outputs":[]},{"metadata":{"id":"id1prBeqeVJM","outputId":"2acc9df9-f0c9-4623-e461-754859db4191","trusted":true},"cell_type":"code","source":"# Preview train set\ntrain_data.sample(5)","execution_count":4,"outputs":[{"output_type":"execute_result","execution_count":4,"data":{"text/plain":"                      id                                       comment_text  \\\n202380  aaa1d08c528cb0d8  \" \\n :::::Fine, if you want to add an explanat...   \n84575   e248e1bda4d2998d  \"\\n(1) I'm using \"\"public\"\" in the broadest se...   \n38370   6673b3878fcd6b40  Notice about your edits \\n\\nPlease do not add ...   \n105019  31cd6d6bc4df7f2e  Attacks on editors \\n\\nI strongly suggest you ...   \n147377  3b76cc9b5ee3f91f  Could you also ask Panonian? You have higher s...   \n\n        toxic  severe_toxic  obscene  threat  insult  identity_hate  \n202380      0             0        0       0       0              0  \n84575       0             0        0       0       0              0  \n38370       0             0        0       0       0              0  \n105019      0             0        0       0       0              0  \n147377      0             0        0       0       0              0  ","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>comment_text</th>\n      <th>toxic</th>\n      <th>severe_toxic</th>\n      <th>obscene</th>\n      <th>threat</th>\n      <th>insult</th>\n      <th>identity_hate</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>202380</th>\n      <td>aaa1d08c528cb0d8</td>\n      <td>\" \\n :::::Fine, if you want to add an explanat...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>84575</th>\n      <td>e248e1bda4d2998d</td>\n      <td>\"\\n(1) I'm using \"\"public\"\" in the broadest se...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>38370</th>\n      <td>6673b3878fcd6b40</td>\n      <td>Notice about your edits \\n\\nPlease do not add ...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>105019</th>\n      <td>31cd6d6bc4df7f2e</td>\n      <td>Attacks on editors \\n\\nI strongly suggest you ...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>147377</th>\n      <td>3b76cc9b5ee3f91f</td>\n      <td>Could you also ask Panonian? You have higher s...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"id":"-yyaGcAhehZ6"},"cell_type":"markdown","source":"Columns (comes from [here](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/data)): \n- id - identifier within each file.\n- comment_text - the text of the comment to be classified.\n- toxic:identity_hate - whether or not the comment is classified as toxic. "},{"metadata":{"id":"Ya4qc1NrgI6s","outputId":"c52b6859-70fd-4d9a-9e98-eaf26ed4a6c8","trusted":true},"cell_type":"code","source":"val_data.sample(5)","execution_count":5,"outputs":[{"output_type":"execute_result","execution_count":5,"data":{"text/plain":"        id                                       comment_text lang  toxic\n5738  5738  hz.isa ile ilgili bir takıntısı olduğunu düşün...   tr      1\n4489  4489  E  vero ma in questo caso si tratta di un blog...   it      0\n1908  1908  Şu konuyla ilgili: Kullanıcı gece saatlerinde ...   tr      0\n6689  6689  bilgisayarda çalışırken canım sıkıldı ve biraz...   tr      0\n6105  6105  Devriye olması gerekmiyor muydu? Engeli bittiğ...   tr      0","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>comment_text</th>\n      <th>lang</th>\n      <th>toxic</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>5738</th>\n      <td>5738</td>\n      <td>hz.isa ile ilgili bir takıntısı olduğunu düşün...</td>\n      <td>tr</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4489</th>\n      <td>4489</td>\n      <td>E  vero ma in questo caso si tratta di un blog...</td>\n      <td>it</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1908</th>\n      <td>1908</td>\n      <td>Şu konuyla ilgili: Kullanıcı gece saatlerinde ...</td>\n      <td>tr</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>6689</th>\n      <td>6689</td>\n      <td>bilgisayarda çalışırken canım sıkıldı ve biraz...</td>\n      <td>tr</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>6105</th>\n      <td>6105</td>\n      <td>Devriye olması gerekmiyor muydu? Engeli bittiğ...</td>\n      <td>tr</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"id":"zbkaRDiCgLD9","outputId":"f00bc7e5-a4eb-4ad8-89ed-d7dcd5e2f4f9","trusted":true},"cell_type":"code","source":"test_data.sample(5)","execution_count":6,"outputs":[{"output_type":"execute_result","execution_count":6,"data":{"text/plain":"          id                                            content lang\n21587  21587  Comme il n y a aucune mention du nom  Ayumi Ha...   fr\n4374    4374  surtout obligé les preuves que tu racontes n i...   fr\n21699  21699  , io sono ignorante come un pigna su queste co...   it\n2822    2822  Да, всё, руки не доходят, создать Проект. В те...   ru\n57145  57145  ¿Acaso no se dan cuenta que cualquier mención ...   es","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>content</th>\n      <th>lang</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>21587</th>\n      <td>21587</td>\n      <td>Comme il n y a aucune mention du nom  Ayumi Ha...</td>\n      <td>fr</td>\n    </tr>\n    <tr>\n      <th>4374</th>\n      <td>4374</td>\n      <td>surtout obligé les preuves que tu racontes n i...</td>\n      <td>fr</td>\n    </tr>\n    <tr>\n      <th>21699</th>\n      <td>21699</td>\n      <td>, io sono ignorante come un pigna su queste co...</td>\n      <td>it</td>\n    </tr>\n    <tr>\n      <th>2822</th>\n      <td>2822</td>\n      <td>Да, всё, руки не доходят, создать Проект. В те...</td>\n      <td>ru</td>\n    </tr>\n    <tr>\n      <th>57145</th>\n      <td>57145</td>\n      <td>¿Acaso no se dan cuenta que cualquier mención ...</td>\n      <td>es</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"id":"ErgoMBGIe-Vz"},"cell_type":"markdown","source":"It's a multilingual dataset as you can see. \n\nI am going to borrow the helper functions as shown here: https://www.kaggle.com/tarunpaparaju/jigsaw-multilingual-toxicity-eda-models. "},{"metadata":{"id":"80ouy6L9eZMX","trusted":true},"cell_type":"code","source":"# Remove usernames and links\nimport re\n\nval = val_data\ntrain = train_data\n\ndef clean(text):\n    # fill the missing entries and convert them to lower case\n    text = text.fillna(\"fillna\").str.lower()\n    # replace the newline characters with space \n    text = text.map(lambda x: re.sub('\\\\n',' ',str(x)))\n    text = text.map(lambda x: re.sub(\"\\[\\[User.*\",'',str(x)))\n    # remove usernames and links\n    text = text.map(lambda x: re.sub(\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\",'',str(x)))\n    text = text.map(lambda x: re.sub(\"\\(http://.*?\\s\\(http://.*\\)\",'',str(x)))\n    return text\n\nval[\"comment_text\"] = clean(val[\"comment_text\"])\ntest_data[\"content\"] = clean(test_data[\"content\"])\ntrain[\"comment_text\"] = clean(train[\"comment_text\"])","execution_count":7,"outputs":[]},{"metadata":{"id":"OJatJaA2rEoW","outputId":"0f54b8c6-8289-4848-f2b5-981be52296c2","trusted":true},"cell_type":"code","source":"# Load DistilBERT tokenizer\nimport transformers\n\ntokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')","execution_count":8,"outputs":[{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, description='Downloading', max=995526.0, style=ProgressStyle(descripti…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"db9b189139614058bbda1760d83c15d6"}},"metadata":{}},{"output_type":"stream","text":"\n","name":"stdout"}]},{"metadata":{"id":"v-7uiZJprjaO"},"cell_type":"markdown","source":"The following function comes from [here](https://github.com/dipanjanS/deep_transfer_learning_nlp_dhs2019/blob/master/notebooks/6%20-%20Transformers%20-%20DistilBERT.ipynb)."},{"metadata":{"id":"CRbtTaUvrXxF","trusted":true},"cell_type":"code","source":"import numpy as np\nimport tqdm\n\ndef create_bert_input_features(tokenizer, docs, max_seq_length):\n    \n    all_ids, all_masks = [], []\n    for doc in tqdm.tqdm(docs, desc=\"Converting docs to features\"):\n        tokens = tokenizer.tokenize(doc)\n        if len(tokens) > max_seq_length-2:\n            tokens = tokens[0 : (max_seq_length-2)]\n        tokens = ['[CLS]'] + tokens + ['[SEP]']\n        ids = tokenizer.convert_tokens_to_ids(tokens)\n        masks = [1] * len(ids)\n        # Zero-pad up to the sequence length.\n        while len(ids) < max_seq_length:\n            ids.append(0)\n            masks.append(0)\n        all_ids.append(ids)\n        all_masks.append(masks)\n    encoded = np.array([all_ids, all_masks])\n    return encoded","execution_count":9,"outputs":[]},{"metadata":{"id":"DZh_s_Ecrn-c","trusted":true},"cell_type":"code","source":"# Segregate the comments and their labels (not applicable for test set)\ntrain_comments = train.comment_text.astype(str).values\nval_comments = val_data.comment_text.astype(str).values\ntest_comments = test_data.content.astype(str).values\n\ny_valid = val.toxic.values\ny_train = train.toxic.values","execution_count":10,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect()","execution_count":11,"outputs":[{"output_type":"execute_result","execution_count":11,"data":{"text/plain":"0"},"metadata":{}}]},{"metadata":{"id":"DDCRnH-AsLeZ","outputId":"c7d8e136-a3c6-43c1-8da6-72efdd056522","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Encode the comments\nMAX_SEQ_LENGTH = 500\n\ntrain_features_ids, train_features_masks = create_bert_input_features(tokenizer, train_comments, \n                                                                      max_seq_length=MAX_SEQ_LENGTH)\nval_features_ids, val_features_masks = create_bert_input_features(tokenizer, val_comments, \n                                                                  max_seq_length=MAX_SEQ_LENGTH)\n# test_features = create_bert_input_features(tokenizer, test_comments, \n#                                            max_seq_length=MAX_SEQ_LENGTH)","execution_count":12,"outputs":[{"output_type":"stream","text":"Converting docs to features: 100%|██████████| 223549/223549 [11:01<00:00, 337.90it/s]\nConverting docs to features: 100%|██████████| 8000/8000 [00:23<00:00, 334.54it/s]\n","name":"stderr"}]},{"metadata":{"id":"6lm5Yy88xN2L","outputId":"8934568d-b538-4559-974c-78a6c42855dc","trusted":true},"cell_type":"code","source":"# Verify the shapes\nprint(train_features_ids.shape, train_features_masks.shape, y_train.shape)\nprint(val_features_ids.shape, val_features_masks.shape, y_valid.shape)","execution_count":13,"outputs":[{"output_type":"stream","text":"(223549, 500) (223549, 500) (223549,)\n(8000, 500) (8000, 500) (8000,)\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Configure TPU\nfrom kaggle_datasets import KaggleDatasets\n\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(tpu)\ntf.tpu.experimental.initialize_tpu_system(tpu)\nstrategy = tf.distribute.experimental.TPUStrategy(tpu)\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('jigsaw-multilingual-toxic-comment-classification')\n\nEPOCHS = 2\nBATCH_SIZE = 32 * strategy.num_replicas_in_sync","execution_count":15,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create TensorFlow datasets for better performance\ntrain_ds = (\n    tf.data.Dataset\n    .from_tensor_slices(((train_features_ids, train_features_masks), y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(BATCH_SIZE)\n    .prefetch(tf.data.experimental.AUTOTUNE)\n)\n    \nvalid_ds = (\n    tf.data.Dataset\n    .from_tensor_slices(((val_features_ids, val_features_masks), y_valid))\n    .repeat()\n    .batch(BATCH_SIZE)\n    .prefetch(tf.data.experimental.AUTOTUNE)\n)","execution_count":18,"outputs":[]},{"metadata":{"id":"mMSzIXvAyR27"},"cell_type":"markdown","source":"## Model building and training"},{"metadata":{"id":"mk-WornLuqSx","trusted":true},"cell_type":"code","source":"# Create utility function to get a training ready model on demand\ndef get_training_model():\n    inp_id = tf.keras.layers.Input(shape=(MAX_SEQ_LENGTH,), dtype=tf.int64, name=\"bert_input_ids\")\n    inp_mask = tf.keras.layers.Input(shape=(MAX_SEQ_LENGTH,), dtype=tf.int64, name=\"bert_input_masks\")\n    inputs = [inp_id, inp_mask]\n\n    hidden_state = transformers.TFDistilBertModel.from_pretrained('distilbert-base-multilingual-cased')(inputs)[0]\n    pooled_output = hidden_state[:, 0]    \n    dense1 = tf.keras.layers.Dense(128, activation='relu')(pooled_output)\n    output = tf.keras.layers.Dense(1, activation='sigmoid')(dense1)\n\n    model = tf.keras.Model(inputs=inputs, outputs=output)\n    model.compile(optimizer=tf.optimizers.Adam(learning_rate=2e-5, \n                                            epsilon=1e-08), \n                loss='binary_crossentropy', metrics=['accuracy'])\n\n    return model","execution_count":19,"outputs":[]},{"metadata":{"id":"a1ED6Cyxxpd2","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Authorize wandb\nimport wandb\n\nwandb.login()\nfrom wandb.keras import WandbCallback","execution_count":20,"outputs":[{"output_type":"stream","text":"\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[32m\u001b[41mERROR\u001b[0m Not authenticated.  Copy a key from https://app.wandb.ai/authorize\n","name":"stderr"},{"output_type":"stream","name":"stdout","text":"API Key: ········\n"},{"output_type":"stream","text":"\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n","name":"stderr"}]},{"metadata":{"id":"NTniAlOc2_fj","outputId":"df3f2bf2-ee30-4472-dfe3-658eb848e979","trusted":true},"cell_type":"code","source":"# Initialize wandb\nwandb.init(project=\"jigsaw-toxic\", id=\"distilbert-tpu-kaggle-weighted\")","execution_count":21,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"\n                Logging results to <a href=\"https://wandb.com\" target=\"_blank\">Weights & Biases</a> <a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">(Documentation)</a>.<br/>\n                Project page: <a href=\"https://app.wandb.ai/sayakpaul/jigsaw-toxic\" target=\"_blank\">https://app.wandb.ai/sayakpaul/jigsaw-toxic</a><br/>\n                Run page: <a href=\"https://app.wandb.ai/sayakpaul/jigsaw-toxic/runs/distilbert-tpu-kaggle-weighted\" target=\"_blank\">https://app.wandb.ai/sayakpaul/jigsaw-toxic/runs/distilbert-tpu-kaggle-weighted</a><br/>\n            "},"metadata":{}},{"output_type":"execute_result","execution_count":21,"data":{"text/plain":"W&B Run: https://app.wandb.ai/sayakpaul/jigsaw-toxic/runs/distilbert-tpu-kaggle-weighted"},"metadata":{}}]},{"metadata":{"id":"Z2b4mh00zPQq","outputId":"911a2410-23d1-4889-e533-f2195ca9d4fe","trusted":true},"cell_type":"code","source":"# Create 32 random indices from the English only test comments\nRANDOM_INDICES = np.random.choice(test_comments.shape[0], 32)\nRANDOM_INDICES","execution_count":22,"outputs":[{"output_type":"execute_result","execution_count":22,"data":{"text/plain":"array([45335, 59335, 26736, 16088, 17969, 40881, 11877, 37071, 63772,\n       22713, 22262, 50775,  2368, 34291, 47547, 41171, 28948,  4920,\n       45737,  3029,  1213, 18222, 56638, 41219, 37336, 63313, 36946,\n       26508, 48649,  2890,  7353,  2832])"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"We will be logging some sample predictions on the test dataset to see how our model is doing as it is getting trained. Now, as this is a mulitlingual dataset, we may need to convert a given comment to a language of our choice to make sense of the model's prediction. We will be using the `googletrans` library. "},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q googletrans","execution_count":23,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Demo examples of translations\nfrom googletrans import Translator\n\nsample_comment = test_comments[48649]\nprint(\"Original comment:\", sample_comment)\ntranslated_comment = Translator().translate(sample_comment)\nprint(\"\\n\")\nprint(\"Translated comment:\", translated_comment.text)","execution_count":25,"outputs":[{"output_type":"stream","text":"Original comment:  ¡ah! sí, ya lo sé... pero como que no puedo sacarme ciertos argentinismos de encima a la hora de escribir. —   kved    (discusión)    pd: aunque no sé si lo correcto no es escribir  bloqueé  en lugar de  bloquee . para solucionar ese tema, es más fácil decir   bloquié   y que la rae se vaya a tomar por culo.   ;)\n\n\nTranslated comment: Ah! Yes, I know ... but I can not get me out certain argentinismos off when writing. - kved (discussion) pd: I do not know if right not write blocked instead of blocking. to solve this issue, it is easier to say rae bloquié and that is to take the ass. ;)\n","name":"stdout"}]},{"metadata":{"id":"ydbxVOtV0by0","trusted":true},"cell_type":"code","source":"# Create a sample prediction logger\n# A custom callback to view predictions on the above samples in real-time\nclass TextLogger(tf.keras.callbacks.Callback):\n    def __init__(self):\n        super(TextLogger, self).__init__()\n\n    def on_epoch_end(self, logs, epoch):\n        samples = []\n        for index in RANDOM_INDICES:\n            # Grab the comment and translate it\n            comment = test_comments[index]\n            translated_comment = Translator().translate(comment).text\n            # Create BERT features\n            comment_feature_ids, comment_features_masks = create_bert_input_features(tokenizer,  \n                                    comment, max_seq_length=MAX_SEQ_LENGTH)\n            # Employ the model to get the prediction and parse it\n            predicted_label = self.model.predict([comment_feature_ids, comment_features_masks])\n            predicted_label = np.argmax(predicted_label[0])\n            if predicted_label==0: predicted_label=\"Non-Toxic\"\n            else: predicted_label=\"Toxic\"\n            \n            sample = [comment, translated_comment, predicted_label]\n            \n            samples.append(sample)\n        wandb.log({\"text\": wandb.Table(data=samples, \n                                       columns=[\"Comment\", \"Translated Comment\", \"Predicted Label\"])})","execution_count":26,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Garbage collection\ngc.collect()","execution_count":27,"outputs":[{"output_type":"execute_result","execution_count":27,"data":{"text/plain":"1767"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Account for the class imbalance\nfrom sklearn.utils import class_weight\n\nclass_weights = class_weight.compute_class_weight('balanced',\n                                                 np.unique(y_train),\n                                                 y_train)\nclass_weights","execution_count":34,"outputs":[{"output_type":"execute_result","execution_count":34,"data":{"text/plain":"array([0.55288749, 5.22701553])"},"metadata":{}}]},{"metadata":{"id":"MedenR-oxzhi","outputId":"9bf330f7-4a77-4da3-a42c-b4c17679677f","trusted":true},"cell_type":"code","source":"# Train the model\nimport time\n\nstart = time.time()\n\n# Compile the model with TPU Strategy\nwith strategy.scope():\n    model = get_training_model()\n    \nmodel.fit(train_ds, \n          steps_per_epoch=train_data.shape[0] // BATCH_SIZE,\n          validation_data=valid_ds,\n          validation_steps=val_data.shape[0] // BATCH_SIZE,\n          epochs=EPOCHS,\n          class_weight=class_weights,\n          callbacks=[WandbCallback(), TextLogger()],\n          verbose=1)\nend = time.time() - start\nprint(\"Time taken \",end)\nwandb.log({\"training_time\":end})","execution_count":35,"outputs":[{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, description='Downloading', max=618.0, style=ProgressStyle(description_…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"6c1e4fc2097a4c9f8238cea947a8b185"}},"metadata":{}},{"output_type":"stream","text":"\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, description='Downloading', max=910749124.0, style=ProgressStyle(descri…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"9ffd6587109648ca8ef26774c8817683"}},"metadata":{}},{"output_type":"stream","text":"\nTrain for 873 steps, validate for 31 steps\nEpoch 1/2\n872/873 [============================>.] - ETA: 0s - loss: 0.1161 - accuracy: 0.9536","name":"stdout"},{"output_type":"stream","text":"\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[32m\u001b[41mERROR\u001b[0m Can't save model, h5py returned error: \nConverting docs to features: 100%|██████████| 99/99 [00:00<00:00, 3474.63it/s]\nConverting docs to features: 100%|██████████| 345/345 [00:00<00:00, 3355.53it/s]\nConverting docs to features: 100%|██████████| 169/169 [00:00<00:00, 3387.00it/s]\nConverting docs to features: 100%|██████████| 506/506 [00:00<00:00, 3463.48it/s]\nConverting docs to features: 100%|██████████| 139/139 [00:00<00:00, 3366.29it/s]\nConverting docs to features: 100%|██████████| 312/312 [00:00<00:00, 3323.52it/s]\nConverting docs to features: 100%|██████████| 247/247 [00:00<00:00, 3511.72it/s]\nConverting docs to features: 100%|██████████| 174/174 [00:00<00:00, 3402.15it/s]\nConverting docs to features: 100%|██████████| 612/612 [00:00<00:00, 3367.64it/s]\nConverting docs to features: 100%|██████████| 616/616 [00:00<00:00, 3381.52it/s]\nConverting docs to features: 100%|██████████| 111/111 [00:00<00:00, 3160.89it/s]\nConverting docs to features: 100%|██████████| 122/122 [00:00<00:00, 3464.07it/s]\nConverting docs to features: 100%|██████████| 107/107 [00:00<00:00, 3425.28it/s]\nConverting docs to features: 100%|██████████| 262/262 [00:00<00:00, 3285.92it/s]\nConverting docs to features: 100%|██████████| 668/668 [00:00<00:00, 3335.99it/s]\nConverting docs to features: 100%|██████████| 854/854 [00:00<00:00, 3240.02it/s]\nConverting docs to features: 100%|██████████| 127/127 [00:00<00:00, 3368.05it/s]\nConverting docs to features: 100%|██████████| 218/218 [00:00<00:00, 3377.84it/s]\nConverting docs to features: 100%|██████████| 334/334 [00:00<00:00, 3227.64it/s]\nConverting docs to features: 100%|██████████| 245/245 [00:00<00:00, 3450.14it/s]\nConverting docs to features: 100%|██████████| 176/176 [00:00<00:00, 3486.59it/s]\nConverting docs to features: 100%|██████████| 489/489 [00:00<00:00, 3382.81it/s]\nConverting docs to features: 100%|██████████| 329/329 [00:00<00:00, 3153.29it/s]\nConverting docs to features: 100%|██████████| 280/280 [00:00<00:00, 3456.26it/s]\nConverting docs to features: 100%|██████████| 159/159 [00:00<00:00, 3389.50it/s]\nConverting docs to features: 100%|██████████| 422/422 [00:00<00:00, 3222.83it/s]\nConverting docs to features: 100%|██████████| 90/90 [00:00<00:00, 2918.66it/s]\nConverting docs to features: 100%|██████████| 209/209 [00:00<00:00, 3459.01it/s]\nConverting docs to features: 100%|██████████| 315/315 [00:00<00:00, 3535.34it/s]\nConverting docs to features: 100%|██████████| 439/439 [00:00<00:00, 3269.89it/s]\nConverting docs to features: 100%|██████████| 253/253 [00:00<00:00, 3501.83it/s]\nConverting docs to features: 100%|██████████| 155/155 [00:00<00:00, 3317.09it/s]\n","name":"stderr"},{"output_type":"stream","text":"873/873 [==============================] - 592s 679ms/step - loss: 0.1161 - accuracy: 0.9536 - val_loss: 0.5069 - val_accuracy: 0.8478\nEpoch 2/2\n872/873 [============================>.] - ETA: 0s - loss: 0.0829 - accuracy: 0.9661","name":"stdout"},{"output_type":"stream","text":"Converting docs to features: 100%|██████████| 99/99 [00:00<00:00, 3105.99it/s]\nConverting docs to features: 100%|██████████| 345/345 [00:00<00:00, 3337.37it/s]\nConverting docs to features: 100%|██████████| 169/169 [00:00<00:00, 3222.91it/s]\nConverting docs to features: 100%|██████████| 506/506 [00:00<00:00, 3274.03it/s]\nConverting docs to features: 100%|██████████| 139/139 [00:00<00:00, 3311.02it/s]\nConverting docs to features: 100%|██████████| 312/312 [00:00<00:00, 3270.40it/s]\nConverting docs to features: 100%|██████████| 247/247 [00:00<00:00, 3448.82it/s]\nConverting docs to features: 100%|██████████| 174/174 [00:00<00:00, 3021.22it/s]\nConverting docs to features: 100%|██████████| 612/612 [00:00<00:00, 3367.01it/s]\nConverting docs to features: 100%|██████████| 616/616 [00:00<00:00, 3368.86it/s]\nConverting docs to features: 100%|██████████| 111/111 [00:00<00:00, 3196.04it/s]\nConverting docs to features: 100%|██████████| 122/122 [00:00<00:00, 3186.03it/s]\nConverting docs to features: 100%|██████████| 107/107 [00:00<00:00, 3353.49it/s]\nConverting docs to features: 100%|██████████| 262/262 [00:00<00:00, 3459.66it/s]\nConverting docs to features: 100%|██████████| 668/668 [00:00<00:00, 3111.19it/s]\nConverting docs to features: 100%|██████████| 854/854 [00:00<00:00, 3333.31it/s]\nConverting docs to features: 100%|██████████| 127/127 [00:00<00:00, 2972.79it/s]\nConverting docs to features: 100%|██████████| 218/218 [00:00<00:00, 3529.63it/s]\nConverting docs to features: 100%|██████████| 334/334 [00:00<00:00, 3349.76it/s]\nConverting docs to features: 100%|██████████| 245/245 [00:00<00:00, 3307.61it/s]\nConverting docs to features: 100%|██████████| 176/176 [00:00<00:00, 3512.68it/s]\nConverting docs to features: 100%|██████████| 489/489 [00:00<00:00, 3401.33it/s]\nConverting docs to features: 100%|██████████| 329/329 [00:00<00:00, 3196.40it/s]\nConverting docs to features: 100%|██████████| 280/280 [00:00<00:00, 3383.17it/s]\nConverting docs to features: 100%|██████████| 159/159 [00:00<00:00, 3195.13it/s]\nConverting docs to features: 100%|██████████| 422/422 [00:00<00:00, 3310.87it/s]\nConverting docs to features: 100%|██████████| 90/90 [00:00<00:00, 3227.24it/s]\nConverting docs to features: 100%|██████████| 209/209 [00:00<00:00, 3302.10it/s]\nConverting docs to features: 100%|██████████| 315/315 [00:00<00:00, 3414.87it/s]\nConverting docs to features: 100%|██████████| 439/439 [00:00<00:00, 3153.14it/s]\nConverting docs to features: 100%|██████████| 253/253 [00:00<00:00, 3415.16it/s]\nConverting docs to features: 100%|██████████| 155/155 [00:00<00:00, 3212.15it/s]\n","name":"stderr"},{"output_type":"stream","text":"873/873 [==============================] - 514s 589ms/step - loss: 0.0830 - accuracy: 0.9660 - val_loss: 0.6298 - val_accuracy: 0.8485\nTime taken  1160.0268676280975\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"**As I am logging some demo predictions in between this training time should not be used for any benchmarks. **\n\nLet's try a CNN (with 1D convolutions) now. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create utility function to get a training ready model on demand\ndef get_training_model_cnn():\n    inp_id = tf.keras.layers.Input(shape=(MAX_SEQ_LENGTH,), dtype=tf.int64, name=\"bert_input_ids\")\n    inp_mask = tf.keras.layers.Input(shape=(MAX_SEQ_LENGTH,), dtype=tf.int64, name=\"bert_input_masks\")\n    inputs = [inp_id, inp_mask]\n\n    hidden_state = transformers.TFDistilBertModel.from_pretrained('distilbert-base-multilingual-cased')(inputs)[0]\n    pooled_output = hidden_state[:, 0]    \n    reshaped_pooled = tf.keras.layers.Reshape((768,1), input_shape=(768,))(pooled_output)\n    conv_1 = tf.keras.layers.Conv1D(64, 2, activation='relu')(reshaped_pooled)\n    pooled_2 = tf.keras.layers.GlobalAveragePooling1D()(conv_1)\n    dense_1 = tf.keras.layers.Dense(128, activation='relu')(pooled_2)\n    output = tf.keras.layers.Dense(1, activation='sigmoid')(dense_1)\n\n    model = tf.keras.Model(inputs=inputs, outputs=output)\n    model.compile(optimizer=tf.optimizers.Adam(learning_rate=2e-5, \n                                            epsilon=1e-08), \n                loss='binary_crossentropy', metrics=['accuracy'])\n\n    return model","execution_count":41,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Garbage collection\ngc.collect()\n\n# Reinitialize wandb\nwandb.init(project=\"jigsaw-toxic\", id=\"distilbert-tpu-kaggle-weighted-cnn\")","execution_count":37,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"\n                Logging results to <a href=\"https://wandb.com\" target=\"_blank\">Weights & Biases</a> <a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">(Documentation)</a>.<br/>\n                Project page: <a href=\"https://app.wandb.ai/sayakpaul/jigsaw-toxic\" target=\"_blank\">https://app.wandb.ai/sayakpaul/jigsaw-toxic</a><br/>\n                Run page: <a href=\"https://app.wandb.ai/sayakpaul/jigsaw-toxic/runs/distilbert-tpu-kaggle-weighted-cnn\" target=\"_blank\">https://app.wandb.ai/sayakpaul/jigsaw-toxic/runs/distilbert-tpu-kaggle-weighted-cnn</a><br/>\n            "},"metadata":{}},{"output_type":"execute_result","execution_count":37,"data":{"text/plain":"W&B Run: https://app.wandb.ai/sayakpaul/jigsaw-toxic/runs/distilbert-tpu-kaggle-weighted-cnn"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the CNN-based model\nstart = time.time()\n\n# Compile the model with TPU Strategy\nwith strategy.scope():\n    model = get_training_model_cnn()\n    \nmodel.fit(train_ds, \n          steps_per_epoch=train_data.shape[0] // BATCH_SIZE,\n          validation_data=valid_ds,\n          validation_steps=val_data.shape[0] // BATCH_SIZE,\n          epochs=EPOCHS,\n          class_weight=class_weights,\n          callbacks=[WandbCallback(), TextLogger()],\n          verbose=1)\nend = time.time() - start\nprint(\"Time taken \",end)\nwandb.log({\"training_time\":end})","execution_count":42,"outputs":[{"output_type":"stream","text":"Train for 873 steps, validate for 31 steps\nEpoch 1/2\n872/873 [============================>.] - ETA: 0s - loss: 0.5604 - accuracy: 0.8949","name":"stdout"},{"output_type":"stream","text":"\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[32m\u001b[41mERROR\u001b[0m Can't save model, h5py returned error: \nConverting docs to features: 100%|██████████| 99/99 [00:00<00:00, 3036.11it/s]\nConverting docs to features: 100%|██████████| 345/345 [00:00<00:00, 3261.09it/s]\nConverting docs to features: 100%|██████████| 169/169 [00:00<00:00, 3239.05it/s]\nConverting docs to features: 100%|██████████| 506/506 [00:00<00:00, 3346.99it/s]\nConverting docs to features: 100%|██████████| 139/139 [00:00<00:00, 3184.11it/s]\nConverting docs to features: 100%|██████████| 312/312 [00:00<00:00, 3242.44it/s]\nConverting docs to features: 100%|██████████| 247/247 [00:00<00:00, 3304.00it/s]\nConverting docs to features: 100%|██████████| 174/174 [00:00<00:00, 3211.55it/s]\nConverting docs to features: 100%|██████████| 612/612 [00:00<00:00, 3484.40it/s]\nConverting docs to features: 100%|██████████| 616/616 [00:00<00:00, 3404.12it/s]\nConverting docs to features: 100%|██████████| 111/111 [00:00<00:00, 3290.72it/s]\nConverting docs to features: 100%|██████████| 122/122 [00:00<00:00, 3245.23it/s]\nConverting docs to features: 100%|██████████| 107/107 [00:00<00:00, 3094.64it/s]\nConverting docs to features: 100%|██████████| 262/262 [00:00<00:00, 3367.61it/s]\nConverting docs to features: 100%|██████████| 668/668 [00:00<00:00, 3274.75it/s]\nConverting docs to features: 100%|██████████| 854/854 [00:00<00:00, 3348.55it/s]\nConverting docs to features: 100%|██████████| 127/127 [00:00<00:00, 3469.84it/s]\nConverting docs to features: 100%|██████████| 218/218 [00:00<00:00, 3412.69it/s]\nConverting docs to features: 100%|██████████| 334/334 [00:00<00:00, 3350.42it/s]\nConverting docs to features: 100%|██████████| 245/245 [00:00<00:00, 3311.46it/s]\nConverting docs to features: 100%|██████████| 176/176 [00:00<00:00, 3380.47it/s]\nConverting docs to features: 100%|██████████| 489/489 [00:00<00:00, 3268.14it/s]\nConverting docs to features: 100%|██████████| 329/329 [00:00<00:00, 3537.99it/s]\nConverting docs to features: 100%|██████████| 280/280 [00:00<00:00, 3444.17it/s]\nConverting docs to features: 100%|██████████| 159/159 [00:00<00:00, 2608.43it/s]\nConverting docs to features: 100%|██████████| 422/422 [00:00<00:00, 3327.35it/s]\nConverting docs to features: 100%|██████████| 90/90 [00:00<00:00, 3245.50it/s]\nConverting docs to features: 100%|██████████| 209/209 [00:00<00:00, 3269.31it/s]\nConverting docs to features: 100%|██████████| 315/315 [00:00<00:00, 3259.89it/s]\nConverting docs to features: 100%|██████████| 439/439 [00:00<00:00, 3190.46it/s]\nConverting docs to features: 100%|██████████| 253/253 [00:00<00:00, 3504.19it/s]\nConverting docs to features: 100%|██████████| 155/155 [00:00<00:00, 3066.16it/s]\n","name":"stderr"},{"output_type":"stream","text":"873/873 [==============================] - 583s 668ms/step - loss: 0.5602 - accuracy: 0.8950 - val_loss: 0.5032 - val_accuracy: 0.8459\nEpoch 2/2\n872/873 [============================>.] - ETA: 0s - loss: 0.3544 - accuracy: 0.9043","name":"stdout"},{"output_type":"stream","text":"Converting docs to features: 100%|██████████| 99/99 [00:00<00:00, 3549.48it/s]\nConverting docs to features: 100%|██████████| 345/345 [00:00<00:00, 3274.91it/s]\nConverting docs to features: 100%|██████████| 169/169 [00:00<00:00, 3362.30it/s]\nConverting docs to features: 100%|██████████| 506/506 [00:00<00:00, 3263.48it/s]\nConverting docs to features: 100%|██████████| 139/139 [00:00<00:00, 3437.29it/s]\nConverting docs to features: 100%|██████████| 312/312 [00:00<00:00, 3502.22it/s]\nConverting docs to features: 100%|██████████| 247/247 [00:00<00:00, 3570.63it/s]\nConverting docs to features: 100%|██████████| 174/174 [00:00<00:00, 3517.39it/s]\nConverting docs to features: 100%|██████████| 612/612 [00:00<00:00, 3291.56it/s]\nConverting docs to features: 100%|██████████| 616/616 [00:00<00:00, 3402.21it/s]\nConverting docs to features: 100%|██████████| 111/111 [00:00<00:00, 3236.14it/s]\nConverting docs to features: 100%|██████████| 122/122 [00:00<00:00, 3500.25it/s]\nConverting docs to features: 100%|██████████| 107/107 [00:00<00:00, 3411.35it/s]\nConverting docs to features: 100%|██████████| 262/262 [00:00<00:00, 3092.97it/s]\nConverting docs to features: 100%|██████████| 668/668 [00:00<00:00, 3139.27it/s]\nConverting docs to features: 100%|██████████| 854/854 [00:00<00:00, 3397.20it/s]\nConverting docs to features: 100%|██████████| 127/127 [00:00<00:00, 3183.35it/s]\nConverting docs to features: 100%|██████████| 218/218 [00:00<00:00, 3499.49it/s]\nConverting docs to features: 100%|██████████| 334/334 [00:00<00:00, 3438.13it/s]\nConverting docs to features: 100%|██████████| 245/245 [00:00<00:00, 3540.12it/s]\nConverting docs to features: 100%|██████████| 176/176 [00:00<00:00, 3478.47it/s]\nConverting docs to features: 100%|██████████| 489/489 [00:00<00:00, 3512.96it/s]\nConverting docs to features: 100%|██████████| 329/329 [00:00<00:00, 3471.25it/s]\nConverting docs to features: 100%|██████████| 280/280 [00:00<00:00, 3582.73it/s]\nConverting docs to features: 100%|██████████| 159/159 [00:00<00:00, 3516.71it/s]\nConverting docs to features: 100%|██████████| 422/422 [00:00<00:00, 3569.35it/s]\nConverting docs to features: 100%|██████████| 90/90 [00:00<00:00, 3611.84it/s]\nConverting docs to features: 100%|██████████| 209/209 [00:00<00:00, 3480.34it/s]\nConverting docs to features: 100%|██████████| 315/315 [00:00<00:00, 3550.28it/s]\nConverting docs to features: 100%|██████████| 439/439 [00:00<00:00, 3480.74it/s]\nConverting docs to features: 100%|██████████| 253/253 [00:00<00:00, 3550.27it/s]\nConverting docs to features: 100%|██████████| 155/155 [00:00<00:00, 3435.59it/s]\n","name":"stderr"},{"output_type":"stream","text":"873/873 [==============================] - 495s 567ms/step - loss: 0.3543 - accuracy: 0.9042 - val_loss: 0.4286 - val_accuracy: 0.8459\nTime taken  1096.077669620514\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"The model generalizes better. "}],"metadata":{"colab":{"name":"Jigsaw Multilingual Toxic Comment Classification.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU","widgets":{"application/vnd.jupyter.widget-state+json":{"fbfda2c54a4e4c39a20b308ec4c1dc41":{"model_module":"@jupyter-widgets/controls","model_name":"HBoxModel","state":{"_view_name":"HBoxView","_dom_classes":[],"_model_name":"HBoxModel","_view_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_view_count":null,"_view_module_version":"1.5.0","box_style":"","layout":"IPY_MODEL_da4b537602124ec09c5626b24f32f276","_model_module":"@jupyter-widgets/controls","children":["IPY_MODEL_9720b4c309104fe8affaf46308cac913","IPY_MODEL_401bf9db41354d69a7c795335d9735ef"]}},"da4b537602124ec09c5626b24f32f276":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","state":{"_view_name":"LayoutView","grid_template_rows":null,"right":null,"justify_content":null,"_view_module":"@jupyter-widgets/base","overflow":null,"_model_module_version":"1.2.0","_view_count":null,"flex_flow":null,"width":null,"min_width":null,"border":null,"align_items":null,"bottom":null,"_model_module":"@jupyter-widgets/base","top":null,"grid_column":null,"overflow_y":null,"overflow_x":null,"grid_auto_flow":null,"grid_area":null,"grid_template_columns":null,"flex":null,"_model_name":"LayoutModel","justify_items":null,"grid_row":null,"max_height":null,"align_content":null,"visibility":null,"align_self":null,"height":null,"min_height":null,"padding":null,"grid_auto_rows":null,"grid_gap":null,"max_width":null,"order":null,"_view_module_version":"1.2.0","grid_template_areas":null,"object_position":null,"object_fit":null,"grid_auto_columns":null,"margin":null,"display":null,"left":null}},"9720b4c309104fe8affaf46308cac913":{"model_module":"@jupyter-widgets/controls","model_name":"IntProgressModel","state":{"_view_name":"ProgressView","style":"IPY_MODEL_f75d44d8b8dd45c290d8480426b42cc3","_dom_classes":[],"description":"Downloading: 100%","_model_name":"IntProgressModel","bar_style":"success","max":995526,"_view_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","value":995526,"_view_count":null,"_view_module_version":"1.5.0","orientation":"horizontal","min":0,"description_tooltip":null,"_model_module":"@jupyter-widgets/controls","layout":"IPY_MODEL_949c793594f149d1b903da1e1a025998"}},"401bf9db41354d69a7c795335d9735ef":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","state":{"_view_name":"HTMLView","style":"IPY_MODEL_c79d073616014765b3a59e9b0babe27e","_dom_classes":[],"description":"","_model_name":"HTMLModel","placeholder":"​","_view_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","value":" 996k/996k [00:00&lt;00:00, 2.05MB/s]","_view_count":null,"_view_module_version":"1.5.0","description_tooltip":null,"_model_module":"@jupyter-widgets/controls","layout":"IPY_MODEL_63e7f4dd67e44fd2906c117522aa76cf"}},"f75d44d8b8dd45c290d8480426b42cc3":{"model_module":"@jupyter-widgets/controls","model_name":"ProgressStyleModel","state":{"_view_name":"StyleView","_model_name":"ProgressStyleModel","description_width":"initial","_view_module":"@jupyter-widgets/base","_model_module_version":"1.5.0","_view_count":null,"_view_module_version":"1.2.0","bar_color":null,"_model_module":"@jupyter-widgets/controls"}},"949c793594f149d1b903da1e1a025998":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","state":{"_view_name":"LayoutView","grid_template_rows":null,"right":null,"justify_content":null,"_view_module":"@jupyter-widgets/base","overflow":null,"_model_module_version":"1.2.0","_view_count":null,"flex_flow":null,"width":null,"min_width":null,"border":null,"align_items":null,"bottom":null,"_model_module":"@jupyter-widgets/base","top":null,"grid_column":null,"overflow_y":null,"overflow_x":null,"grid_auto_flow":null,"grid_area":null,"grid_template_columns":null,"flex":null,"_model_name":"LayoutModel","justify_items":null,"grid_row":null,"max_height":null,"align_content":null,"visibility":null,"align_self":null,"height":null,"min_height":null,"padding":null,"grid_auto_rows":null,"grid_gap":null,"max_width":null,"order":null,"_view_module_version":"1.2.0","grid_template_areas":null,"object_position":null,"object_fit":null,"grid_auto_columns":null,"margin":null,"display":null,"left":null}},"c79d073616014765b3a59e9b0babe27e":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","state":{"_view_name":"StyleView","_model_name":"DescriptionStyleModel","description_width":"","_view_module":"@jupyter-widgets/base","_model_module_version":"1.5.0","_view_count":null,"_view_module_version":"1.2.0","_model_module":"@jupyter-widgets/controls"}},"63e7f4dd67e44fd2906c117522aa76cf":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","state":{"_view_name":"LayoutView","grid_template_rows":null,"right":null,"justify_content":null,"_view_module":"@jupyter-widgets/base","overflow":null,"_model_module_version":"1.2.0","_view_count":null,"flex_flow":null,"width":null,"min_width":null,"border":null,"align_items":null,"bottom":null,"_model_module":"@jupyter-widgets/base","top":null,"grid_column":null,"overflow_y":null,"overflow_x":null,"grid_auto_flow":null,"grid_area":null,"grid_template_columns":null,"flex":null,"_model_name":"LayoutModel","justify_items":null,"grid_row":null,"max_height":null,"align_content":null,"visibility":null,"align_self":null,"height":null,"min_height":null,"padding":null,"grid_auto_rows":null,"grid_gap":null,"max_width":null,"order":null,"_view_module_version":"1.2.0","grid_template_areas":null,"object_position":null,"object_fit":null,"grid_auto_columns":null,"margin":null,"display":null,"left":null}}}}},"nbformat":4,"nbformat_minor":4}