Kaggle3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Kaggle3.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [
        "Vm6lBw8ZhdU0",
        "ul9wwYybApeE",
        "RKtqKkKnhg5O",
        "wOfFzQxzhm0i",
        "y_oM46Q3hsF9",
        "1a2_79I5hGHl",
        "Aoorz_zfkMvN",
        "g8u8CCAZkQZE",
        "hJiNBNtbsBKB"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cK6Pz_dQBiY7"
      },
      "source": [
        "# Third Kaggle competition - Visual Question Answering\n",
        "\n",
        "To complete this challenge I have chosen to use two different networks to complete two different complementary task at the same time:\n",
        "- Categorize the question in 7 different classes (numbers, colors, yes or no, weather, actions, left or right, others)\n",
        "- Perform the VQA task\n",
        "\n",
        "To select the final answer is used, as usual, *arg_max* but now between the most probable classes in the category predicted by the first network.\n",
        "\n",
        "This method corrects about 1 to 2% of the answers leading to a proportional gain in the final score.\n",
        "\n",
        "This method is effective in this challenge due to the reduced number of final classes (and also to the fact, for example, the category weather includes only one answer).\n",
        "\n",
        "The VQA network architecture is very similar to the one implemented in [this](http://arxiv.org/pdf/1505.00468.pdf) paper but with some modifications:\n",
        "\n",
        "- Increased number of neurons in the fully connected layers\n",
        "- Xception instead of VGG-16\n",
        "- Different embedding size and other hyperparameters\n",
        "\n",
        "The training set is splitted evenly between categories making it balanced w.r.t. the validation set.\n",
        "\n",
        "During this challenge I used, instead of TensorBoard, Weight and Biases to have statistics about how the model is performing and to perform hyperparameters tuning in a simpler way. Here you can find the entire project with all the runs and sweeps: https://wandb.ai/lrsb/kaggle3\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vm6lBw8ZhdU0"
      },
      "source": [
        "# Download dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GmvzPMcS0Y26"
      },
      "source": [
        "import json\n",
        "\n",
        "!pip install --upgrade --force-reinstall --no-deps kaggle\n",
        "!pip install --upgrade wandb\n",
        "\n",
        "#@markdown Insert here your credentials\n",
        "kaggle_username = ''#@param {type:'string'}\n",
        "kaggle_api_key = ''#@param {type:'string'}\n",
        "wandb_key = ''#@param {type:'string'}\n",
        "\n",
        "!wandb login {wandb_key}\n",
        "\n",
        "api_token = {'username': kaggle_username, 'key': kaggle_api_key}\n",
        "\n",
        "!mkdir ~/.kaggle\n",
        "with open('/root/.kaggle/kaggle.json', 'w') as kaggle_json:\n",
        "  json.dump(api_token, kaggle_json)\n",
        "\n",
        "!chmod 600 ~/.kaggle/kaggle.json\n",
        "!kaggle competitions download -c anndl-2020-vqa\n",
        "!unzip -q anndl-2020-vqa.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ul9wwYybApeE"
      },
      "source": [
        "# Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RKtqKkKnhg5O"
      },
      "source": [
        "### Making results more reproducible and setting params"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MrQ-l7ufA_0B"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "SEED = 1234#@param {type:'number'}\n",
        "tf.random.set_seed(SEED)\n",
        "\n",
        "labels = ['0', '1', '2', '3', '4', '5',\n",
        "              'apple', 'baseball', 'bench', \n",
        "              'bike', 'bird', 'black', 'blanket',\n",
        "              'blue', 'bone', 'book', 'boy', \n",
        "              'brown', 'cat', 'chair', 'couch',\n",
        "              'dog', 'floor', 'food', 'football',\n",
        "              'girl', 'grass', 'gray', 'green',\n",
        "              'left', 'log', 'man', 'monkey bars',\n",
        "              'no', 'nothing', 'orange', 'pie',\n",
        "              'plant', 'playing', 'red', 'right',\n",
        "              'rug', 'sandbox', 'sitting',\n",
        "              'sleeping', 'soccer', 'squirrel', \n",
        "              'standing', 'stool', 'sunny', \n",
        "              'table', 'tree', 'watermelon', 'white', \n",
        "              'wine', 'woman', 'yellow', 'yes']\n",
        "\n",
        "labels_dict = {}\n",
        "for index, label in enumerate(labels):\n",
        "  labels_dict[label] = index\n",
        "\n",
        "categories = {\n",
        "  'numbers': [0, 1, 2, 3, 4, 5],\n",
        "  'colors' : [11, 13, 17, 27, 28, 35, 39, 53, 56],\n",
        "  'yesno': [33, 57],\n",
        "  'weather': [49],\n",
        "  'actions': [43, 44, 47],\n",
        "  'leftright': [29, 40],\n",
        "  'others': [6, 7, 8, 9, 10, 12, 14, 15, 16, 18, 19, 20,\n",
        "             21, 22, 23, 24, 25, 26, 30, 31, 32, 34, 36,\n",
        "             37, 38, 41, 42, 45, 46, 48, 50, 51, 52, 54, 55]\n",
        "}\n",
        "\n",
        "categories_long = {\n",
        "  'numbers': [0, 1, 2, 3, 4, 5],\n",
        "  'colors' : [11, 13, 17, 27, 28, 35, 39, 53, 56],\n",
        "  'yesno': [33, 57],\n",
        "  'weather': [49],\n",
        "  'actions': [38, 43, 44, 47],\n",
        "  'leftright': [29, 40],\n",
        "  'persons': [16, 25, 31, 55],\n",
        "  'objects': [6, 7, 8, 9, 10, 12, 14, 15, 18, 19, 20, 21, 22, 23, 24, 26, 30,\n",
        "              32, 34, 36, 37, 41, 42, 45, 46, 48, 50, 51, 52, 54]\n",
        "}\n",
        "\n",
        "img_w = 299#@param {type:'number'}\n",
        "img_h = 299#@param {type:'number'}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOfFzQxzhm0i"
      },
      "source": [
        "### Code for creating datasets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eTNuqLgITkC4"
      },
      "source": [
        "import os, numpy as np, json\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from tensorflow.keras.preprocessing import image as k_image\n",
        "\n",
        "class VQADataset(tf.keras.utils.Sequence):\n",
        "  def __init__(self, validation_split=0, subset='training', only_text=False, preprocessing_function=None):\n",
        "    self.subset = subset\n",
        "    self.only_text = only_text\n",
        "    self.preprocessing_function = preprocessing_function\n",
        "    self.train_set = []\n",
        "    self.valid_set = []\n",
        "    self.test_set = []\n",
        "    self.tokenizer = tf.keras.preprocessing.text.Tokenizer()\n",
        "\n",
        "    with open('/content/VQA_Dataset/train_questions_annotations.json', 'r') as f:\n",
        "      train_questions = json.load(f)\n",
        "\n",
        "      count = [0 for i in range(0, len(labels_dict))]\n",
        "      self.max_question_len = max(len(q) for q in map(lambda e: e['question'], train_questions.values()))\n",
        "\n",
        "      for answer in map(lambda e: e['answer'], train_questions.values()):\n",
        "        count[labels_dict[answer]] += 1\n",
        "\n",
        "      valid_count = [0 for i in range(0, len(labels_dict))]\n",
        "\n",
        "      for key, value in train_questions.items():\n",
        "        item_index = labels_dict[value['answer']]\n",
        "        if valid_count[item_index] < count[item_index] * validation_split:\n",
        "          self.valid_set.append(value)\n",
        "          valid_count[item_index] += 1\n",
        "        else:\n",
        "          self.train_set.append(value)\n",
        "      \n",
        "      self.tokenizer.fit_on_texts([q for q in map(lambda e: e['question'], train_questions.values())])\n",
        "\n",
        "    with open('/content/VQA_Dataset/test_questions.json', 'r') as f:\n",
        "      self.test_set = [[key, value] for key, value in json.load(f).items()]\n",
        "\n",
        "  def __len__(self):\n",
        "    if self.subset == 'testing':\n",
        "      return len(self.test_set)\n",
        "    if self.subset == 'training':\n",
        "      return len(self.train_set)\n",
        "    return len(self.valid_set)\n",
        "\n",
        "  def __getitem__(self, index):\n",
        "    if self.subset == 'testing':\n",
        "      elem = self.test_set[index][1]\n",
        "    elif self.subset == 'training':\n",
        "      elem = self.train_set[index]\n",
        "    else:\n",
        "      elem = self.valid_set[index]\n",
        "    question = pad_sequences(self.tokenizer.texts_to_sequences([elem['question']]), maxlen=self.max_question_len, padding='post')[0]\n",
        "    if self.only_text:\n",
        "      if self.subset != 'testing':\n",
        "        for key, value in categories.items():\n",
        "          if labels_dict[elem['answer']] in value:\n",
        "            return question, tf.keras.utils.to_categorical(list(categories.keys()).index(key), num_classes=len(categories))\n",
        "      return question, [0 for i in range(0, len(categories))]\n",
        "\n",
        "    image = k_image.load_img(os.path.join('/content/VQA_Dataset/Images', elem['image_id'] + '.png'), target_size=(img_w, img_h)) \n",
        "    image = k_image.img_to_array(image)\n",
        "    if self.preprocessing_function is not None:\n",
        "      image = self.preprocessing_function(image)\n",
        "    \n",
        "    if self.subset == 'testing':\n",
        "      return (question, image), [0 for i in range(0, len(labels_dict))]\n",
        "    return (question, image), tf.keras.utils.to_categorical(labels_dict[elem['answer']], num_classes=len(labels_dict))\n",
        "\n",
        "  def vocabulary_size(self):\n",
        "    return len(self.tokenizer.word_index) + 1\n",
        "\n",
        "  def question_id(self, index):\n",
        "    return self.test_set[index][0]\n",
        "\n",
        "\n",
        "def GetVQADatasets(validation_split=0, batch_size=32, preprocessing_function=None):\n",
        "  train_set = VQADataset(validation_split=validation_split, subset='training', preprocessing_function=preprocessing_function)\n",
        "  valid_set = VQADataset(validation_split=validation_split, subset='validation', preprocessing_function=preprocessing_function)\n",
        "  test_set = VQADataset(subset='testing', preprocessing_function=preprocessing_function)\n",
        "\n",
        "  train_dataset = tf.data.Dataset.from_generator(lambda: train_set, \n",
        "                                                 output_types=((tf.int32, tf.float32), tf.float32),\n",
        "                                                 output_shapes=(([train_set.max_question_len], [img_w, img_h, 3]), [len(labels_dict)]))\n",
        "  train_dataset = train_dataset.batch(batch_size)\n",
        "  train_dataset = train_dataset.repeat()\n",
        "\n",
        "\n",
        "  valid_dataset = tf.data.Dataset.from_generator(lambda: valid_set, \n",
        "                                                 output_types=((tf.int32, tf.float32), tf.float32),\n",
        "                                                 output_shapes=(([valid_set.max_question_len], [img_w, img_h, 3]), [len(labels_dict)]))\n",
        "  valid_dataset = valid_dataset.batch(batch_size)\n",
        "  valid_dataset = valid_dataset.repeat()\n",
        "\n",
        "  test_dataset = tf.data.Dataset.from_generator(lambda: test_set,\n",
        "                                                output_types=((tf.int32, tf.float32), tf.float32),\n",
        "                                                output_shapes=(([valid_set.max_question_len], [img_w, img_h, 3]), [len(labels_dict)]))\n",
        "  test_dataset = test_dataset.batch(1)\n",
        "  test_dataset = test_dataset.repeat()\n",
        "\n",
        "  return train_set, train_dataset, valid_set, valid_dataset, test_set, test_dataset\n",
        "\n",
        "def GetQuestionsDatasets(validation_split=0, batch_size=32):\n",
        "  train_set = VQADataset(validation_split=validation_split, subset='training', only_text=True)\n",
        "  valid_set = VQADataset(validation_split=validation_split, subset='validation', only_text=True)\n",
        "  test_set = VQADataset(validation_split=validation_split, subset='testing', only_text=True)\n",
        "\n",
        "  train_dataset = tf.data.Dataset.from_generator(lambda: train_set, \n",
        "                                                 output_types=(tf.int32, tf.int32),\n",
        "                                                 output_shapes=([train_set.max_question_len], [len(categories)]))\n",
        "  train_dataset = train_dataset.batch(batch_size)\n",
        "  train_dataset = train_dataset.repeat()\n",
        "\n",
        "\n",
        "  valid_dataset = tf.data.Dataset.from_generator(lambda: valid_set, \n",
        "                                                 output_types=(tf.int32, tf.int32),\n",
        "                                                 output_shapes=([valid_set.max_question_len], [len(categories)]))\n",
        "  valid_dataset = valid_dataset.batch(batch_size)\n",
        "  valid_dataset = valid_dataset.repeat()\n",
        "\n",
        "  test_dataset = tf.data.Dataset.from_generator(lambda: test_set, \n",
        "                                                 output_types=(tf.int32, tf.int32),\n",
        "                                                 output_shapes=([valid_set.max_question_len], [len(categories)]))\n",
        "  test_dataset = test_dataset.batch(1)\n",
        "  test_dataset = test_dataset.repeat()\n",
        "\n",
        "  return train_set, train_dataset, valid_set, valid_dataset, test_set, test_dataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_oM46Q3hsF9"
      },
      "source": [
        "### Code for saving testset results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1ba1WBpQawed"
      },
      "source": [
        "import ntpath, numpy as np\n",
        "\n",
        "def PredictTestset(vqa_model, cat_model=None, filename='/content/results.csv', filename_corrected='/content/results-cat.csv', preprocessing_function=None, upload=False):\n",
        "  vqa_datasets = GetVQADatasets(preprocessing_function=preprocess_input)\n",
        "  vqa_predictions = vqa_model.predict(vqa_datasets[5], steps=len(vqa_datasets[4]), verbose=1)\n",
        "  if cat_model is not None:\n",
        "    cat_datasets = GetQuestionsDatasets()\n",
        "    cat_predictions = cat_model.predict(cat_datasets[5], steps=len(cat_datasets[4]), verbose=1)\n",
        "\n",
        "  results = {}\n",
        "  results_corrected = {}\n",
        "  corrected = 0\n",
        "  for i in range(0, len(vqa_predictions)):\n",
        "    key = vqa_datasets[4].question_id(i)\n",
        "    prediction = vqa_predictions[i]\n",
        "    results[key] = np.argmax(prediction)\n",
        "\n",
        "    if cat_model is not None:\n",
        "      question_category = list(categories.keys())[np.argmax(cat_predictions[i])]\n",
        "      possible_answer_indexes = categories[question_category]\n",
        "      final_answer = possible_answer_indexes[np.argmax([prediction[index] for index in possible_answer_indexes])]\n",
        "\n",
        "      if results[key] != final_answer:\n",
        "        corrected += 1\n",
        "      results_corrected[key] = final_answer\n",
        "\n",
        "  if cat_model is not None:\n",
        "    print('Corrected ' + str(corrected) + ' of ' + str(len(results)) + ' (' + str(corrected / len(results) * 100) + '%)')\n",
        "    with open(filename_corrected, 'w') as f:\n",
        "      f.write('Id,Category\\n')\n",
        "      for key, value in results_corrected.items():\n",
        "        f.write(key + ',' + str(value) + '\\n')\n",
        "\n",
        "  with open(filename, 'w') as f:\n",
        "    f.write('Id,Category\\n')\n",
        "    for key, value in results.items():\n",
        "      f.write(key + ',' + str(value) + '\\n')\n",
        "\n",
        "  if upload:\n",
        "    !kaggle competitions submit -c anndl-2020-vqa -f $filename_corrected -m 'Autoupload'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1a2_79I5hGHl"
      },
      "source": [
        "# Implementation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Aoorz_zfkMvN"
      },
      "source": [
        "### Answer category predictor"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8yuUblwXjLEH"
      },
      "source": [
        "import wandb\n",
        "from wandb.keras import WandbCallback\n",
        "\n",
        "def GetCatModel(validation_split, batch_size, epochs, embedding_size, dropout, units_1, units_2):\n",
        "  datasets = GetQuestionsDatasets(validation_split=validation_split, batch_size=batch_size)\n",
        "\n",
        "  model = tf.keras.models.Sequential()\n",
        "  model.add(tf.keras.layers.Embedding(datasets[0].vocabulary_size(), \n",
        "                                      embedding_size, \n",
        "                                      input_length=datasets[0].max_question_len,\n",
        "                                      mask_zero=True))\n",
        "  model.add(tf.keras.layers.LSTM(units_1))\n",
        "  model.add(tf.keras.layers.Dropout(dropout))\n",
        "  model.add(tf.keras.layers.Dense(units_1, activation='relu'))\n",
        "  model.add(tf.keras.layers.Dropout(dropout))\n",
        "  model.add(tf.keras.layers.Dense(units_2, activation='relu'))\n",
        "  model.add(tf.keras.layers.Dense(len(categories), activation='softmax'))\n",
        "  model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "\n",
        "  model.fit(x=datasets[1],\n",
        "            epochs=epochs,\n",
        "            steps_per_epoch=len(datasets[0]) // batch_size,\n",
        "            validation_data=datasets[3],\n",
        "            validation_steps=len(datasets[2]) // batch_size, \n",
        "            callbacks=[WandbCallback()])\n",
        "  return model\n",
        "\n",
        "#@markdown Set hyperparameters used during training\n",
        "\n",
        "validation_split = 0.1#@param {type:'number'}\n",
        "batch_size = 64#@param {type:'number'}\n",
        "epochs = 4#@param {type:'number'}\n",
        "embedding_size = 300#@param {type:'number'}\n",
        "dropout = 0.5#@param {type:'number'}\n",
        "units_1 = 1024#@param {type:'number'}\n",
        "units_2 = 1024#@param {type:'number'}\n",
        "\n",
        "#@markdown Or use hyperparameter optimization\n",
        "\n",
        "use_hyperparameter_optimization = False#@param {type:'boolean'}\n",
        "\n",
        "!nvidia-smi\n",
        "\n",
        "if use_hyperparameter_optimization:\n",
        "  def RunFitWithHypOpt():\n",
        "    wandb.init()\n",
        "    model = GetCatModel(wandb.config.validation_split,\n",
        "                        wandb.config.batch_size,\n",
        "                        wandb.config.epochs,\n",
        "                        wandb.config.embedding_size,\n",
        "                        wandb.config.dropout,\n",
        "                        wandb.config.units_1,\n",
        "                        wandb.config.units_2)\n",
        "\n",
        "  wandb.agent(wandb.sweep({\n",
        "    'method': 'bayes',\n",
        "    'metric': {\n",
        "        'name': 'val_accuracy',\n",
        "        'goal': 'maximize'\n",
        "    },\n",
        "    'early_terminate': {\n",
        "        'type': 'hyperband',\n",
        "        'min_iter': 3\n",
        "    },\n",
        "    'parameters': {\n",
        "        'validation_split': {\n",
        "            'values': [0.1, 0.2, 0.4]\n",
        "        },\n",
        "        'batch_size': {\n",
        "            'values': [4, 32, 64, 256]\n",
        "        },\n",
        "        'epochs': {\n",
        "            'value': 4\n",
        "        },\n",
        "        'embedding_size': {\n",
        "            'values': [64, 128, 300]\n",
        "        },\n",
        "        'dropout': {\n",
        "            'values': [0, 0.5]\n",
        "        },\n",
        "        'units_1': {\n",
        "            'values': [512, 1024, 2048]\n",
        "        },\n",
        "        'units_2': {\n",
        "            'values': [256, 1024]\n",
        "        }\n",
        "      }\n",
        "  }, project='kaggle3'), function=RunFitWithHypOpt)\n",
        "\n",
        "else:\n",
        "  wandb.init(project='kaggle3', config={\n",
        "      'validation_split': validation_split,\n",
        "      'batch_size': batch_size,\n",
        "      'epochs': epochs,\n",
        "      'embedding_size': embedding_size,\n",
        "      'dropout': dropout,\n",
        "      'units_1': units_1,\n",
        "      'units_2': units_2\n",
        "  })\n",
        "\n",
        "  cat_model = GetCatModel(wandb.config.validation_split,\n",
        "                        wandb.config.batch_size,\n",
        "                        wandb.config.epochs,\n",
        "                        wandb.config.embedding_size,\n",
        "                        wandb.config.dropout,\n",
        "                        wandb.config.units_1,\n",
        "                        wandb.config.units_2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ioNLrcXnV247"
      },
      "source": [
        "Best performing model [here](https://wandb.ai/lrsb/kaggle3/runs/muum7bu1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g8u8CCAZkQZE"
      },
      "source": [
        "### VQA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xGFV8XTqEmhc"
      },
      "source": [
        "import wandb\n",
        "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
        "from wandb.keras import WandbCallback\n",
        "from tensorflow.keras.applications.xception import preprocess_input\n",
        "\n",
        "def GetVQAModel(validation_split, batch_size, epochs, use_early_stopping, embedding_size, dropout, lstm_units):\n",
        "  datasets = GetVQADatasets(validation_split=validation_split, batch_size=batch_size, preprocessing_function=preprocess_input)\n",
        "\n",
        "  xception = tf.keras.applications.Xception(weights='imagenet', include_top=True, input_shape=(img_w, img_h, 3))\n",
        "  xception.trainable = False\n",
        "\n",
        "  cnnnorm = tf.keras.layers.Lambda(lambda x: tf.keras.backend.l2_normalize(x, axis=1))(xception.layers[-2].output)\n",
        "  cnndense1 = tf.keras.layers.Dense(units=2048, activation='relu')(cnnnorm)\n",
        "  cnn = tf.keras.layers.Dense(units=2048, activation='tanh')(cnndense1)\n",
        "\n",
        "  embedding_input = tf.keras.Input((datasets[0].max_question_len,))\n",
        "  embedding = tf.keras.layers.Embedding(datasets[0].vocabulary_size(),\n",
        "                                        embedding_size,\n",
        "                                        input_length=datasets[0].max_question_len,\n",
        "                                        mask_zero=True)(embedding_input)\n",
        "  lstm1 = tf.keras.layers.LSTM(units=lstm_units,\n",
        "                               return_sequences=True,\n",
        "                               return_state=True,\n",
        "                               input_shape=(datasets[0].max_question_len, embedding_size))(embedding)\n",
        "  lstm2 = tf.keras.layers.LSTM(units=lstm_units, return_sequences=False, return_state=True)(lstm1)\n",
        "  concat = tf.keras.layers.Concatenate()([lstm1[1], lstm1[2], lstm2[0], lstm2[1]])\n",
        "  lstmdense1 = tf.keras.layers.Dense(units=2048, activation='relu')(concat)\n",
        "  lstmdense2 = tf.keras.layers.Dense(2048, activation='tanh')(lstmdense1)\n",
        "\n",
        "  mul = tf.keras.layers.Multiply()([cnn, lstmdense2])\n",
        "  drop1 = tf.keras.layers.Dropout(dropout)(mul)\n",
        "  dense1 = tf.keras.layers.Dense(2048, activation='tanh')(drop1)\n",
        "  drop2 = tf.keras.layers.Dropout(dropout)(dense1)\n",
        "  dense2 = tf.keras.layers.Dense(2048, activation='tanh')(drop2)\n",
        "  dense3 = tf.keras.layers.Dense(units=len(labels_dict), activation='softmax')(dense2)\n",
        "\n",
        "  model = tf.keras.models.Model(inputs=[embedding_input, xception.input], outputs=dense3)\n",
        "  model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])\n",
        "\n",
        "  callbacks = [WandbCallback(),\n",
        "               ModelCheckpoint(filepath='/tmp/checkpoint',\n",
        "                               save_weights_only=True,\n",
        "                               monitor='val_accuracy',\n",
        "                               mode='max',\n",
        "                               save_best_only=True,\n",
        "                               verbose=1)]\n",
        "  if use_early_stopping:\n",
        "    callbacks.append(EarlyStopping(monitor='val_accuracy', mode='max', patience=5, restore_best_weights=False))\n",
        "\n",
        "  model.fit(x=datasets[1],\n",
        "            epochs=epochs,\n",
        "            steps_per_epoch=len(datasets[0]) // batch_size,\n",
        "            validation_data=datasets[3],\n",
        "            validation_steps=len(datasets[2]) // batch_size,\n",
        "            workers=8,\n",
        "            max_queue_size=200,\n",
        "            use_multiprocessing=True,\n",
        "            callbacks=callbacks)\n",
        "  return model\n",
        "\n",
        "#@markdown Set hyperparameters used during training\n",
        "\n",
        "validation_split = 0.03#@param {type:'number'}\n",
        "batch_size = 256#@param {type:'number'}\n",
        "epochs = 15#@param {type:'number'}\n",
        "use_early_stopping = False#@param {type:'boolean'}\n",
        "embedding_size = 512#@param {type:'number'}\n",
        "dropout = 0.5#@param {type:'number'}\n",
        "lstm_units = 256#@param {type:'number'}\n",
        "\n",
        "#@markdown Or use hyperparameter optimization\n",
        "\n",
        "use_hyperparameter_optimization = False#@param {type:'boolean'}\n",
        "\n",
        "!nvidia-smi\n",
        "\n",
        "if use_hyperparameter_optimization:\n",
        "  def RunFitWithHypOpt():\n",
        "    wandb.init()\n",
        "    model = GetVQAModel(wandb.config.validation_split,\n",
        "                        wandb.config.batch_size,\n",
        "                        wandb.config.epochs,\n",
        "                        wandb.config.use_early_stopping,\n",
        "                        wandb.config.embedding_size,\n",
        "                        wandb.config.dropout,\n",
        "                        wandb.config.lstm_units,\n",
        "                        wandb.config.trainable)\n",
        "    PredictTestset(model,\n",
        "                   cat_model=cat_model, \n",
        "                   filename=os.path.join(wandb.run.dir, 'results.csv'), \n",
        "                   filename_corrected=os.path.join(wandb.run.dir, 'results-cat.csv'), \n",
        "                   preprocessing_function=preprocess_input,\n",
        "                   upload=False)\n",
        "\n",
        "  cat_model = tf.keras.models.load_model('/content/drive/MyDrive/Colab Notebooks/Kaggle3/cat-model-best.h5')\n",
        "  wandb.agent(wandb.sweep({\n",
        "    'method': 'bayes',\n",
        "    'metric': {\n",
        "        'name': 'val_accuracy',\n",
        "        'goal': 'maximize'\n",
        "    },\n",
        "    'parameters': {\n",
        "        'validation_split': {\n",
        "            'value': 0.03\n",
        "        },\n",
        "        'batch_size': {\n",
        "            'value': 256\n",
        "        },\n",
        "        'epochs': {\n",
        "            'value': 13\n",
        "        },\n",
        "        'use_early_stopping': {\n",
        "            'value': True\n",
        "        },\n",
        "        'embedding_size': {\n",
        "            'values': [256, 300, 512]\n",
        "        },\n",
        "        'dropout': {\n",
        "            'values': [0, 0.5]\n",
        "        },\n",
        "        'lstm_units': {\n",
        "            'values': [256, 512, 1024]\n",
        "        }\n",
        "      }\n",
        "  }, project='kaggle3'), function=RunFitWithHypOpt)\n",
        "\n",
        "else:\n",
        "  wandb.init(project='kaggle3', config={\n",
        "      'validation_split': validation_split,\n",
        "      'batch_size': batch_size,\n",
        "      'epochs': epochs,\n",
        "      'use_early_stopping': use_early_stopping,\n",
        "      'embedding_size': embedding_size,\n",
        "      'dropout': dropout,\n",
        "      'lstm_units': lstm_units\n",
        "  })\n",
        "\n",
        "  model = GetVQAModel(wandb.config.validation_split,\n",
        "                      wandb.config.batch_size,\n",
        "                      wandb.config.epochs,\n",
        "                      wandb.config.use_early_stopping,\n",
        "                      wandb.config.embedding_size,\n",
        "                      wandb.config.dropout,\n",
        "                      wandb.config.lstm_units)\n",
        "  \n",
        "  cat_model = tf.keras.models.load_model('/content/drive/MyDrive/Colab Notebooks/Kaggle3/cat-model-best.h5')\n",
        "  PredictTestset(model, cat_model=cat_model, \n",
        "                 filename=os.path.join(wandb.run.dir, 'results.csv'), \n",
        "                 filename_corrected=os.path.join(wandb.run.dir, 'results-cat.csv'),\n",
        "                 preprocessing_function=preprocess_input,\n",
        "                 upload=False)\n",
        "  model.load_weights('/tmp/checkpoint')\n",
        "  PredictTestset(model, cat_model=cat_model, \n",
        "                 filename=os.path.join(wandb.run.dir, 'results-restored.csv'), \n",
        "                 filename_corrected=os.path.join(wandb.run.dir, 'results-cat-restored.csv'),\n",
        "                 preprocessing_function=preprocess_input,\n",
        "                 upload=False)\n",
        "  wandb.init()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BwzeceDaNcT0"
      },
      "source": [
        "Best performing model [here](https://wandb.ai/lrsb/kaggle3/runs/1710a4ge)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hJiNBNtbsBKB"
      },
      "source": [
        "# Utilities"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KgnQ54_fnfTI"
      },
      "source": [
        "### Predict dataset using a saved model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gj8w73qdi9Md"
      },
      "source": [
        "from tensorflow.keras.applications.xception import preprocess_input\n",
        "\n",
        "model = tf.keras.models.load_model('/content/drive/MyDrive/Colab Notebooks/Kaggle3/model-best.h5')\n",
        "model_last = tf.keras.models.load_model('/content/drive/MyDrive/Colab Notebooks/Kaggle3/model-best-last.h5')\n",
        "cat_model = tf.keras.models.load_model('/content/drive/MyDrive/Colab Notebooks/Kaggle3/cat-model-best.h5')\n",
        "\n",
        "PredictTestset(model, cat_model=cat_model, filename='/content/results.csv', filename_corrected='/content/results-cat.csv', preprocessing_function=preprocess_input, upload=False)\n",
        "PredictTestset(model_last, cat_model=cat_model, filename='/content/results-last.csv', filename_corrected='/content/results-cat-last.csv', preprocessing_function=preprocess_input, upload=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h2n2x6rWUxSC"
      },
      "source": [
        "model_last.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d2nn3AsXRUIM"
      },
      "source": [
        "print(model_last.get_layer('dense_2').get_config())"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}