fuji-sfm-and-pfuji-size-data-patch-generator.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Data Patch Generation for Fuji-SfM dataset and PFuji-Size PCL Base Files\n",
        "\n",
        "Our task with this notebook is to essentially generate PCL patches with apple and tree class data points with approximately similar distributions of data points from each class in those patches.\n",
        "\n",
        "#### 1. Generation of 3D Data Patch Dataset\n",
        "\n",
        "First, we normalize _(z-transform)_ each of the base numpy point cloud datasets to re-center the datasets.\n",
        "Second, we compare the data distribution of each datasets and make sure that the generated data patches have similar distribution during the generation stage. \n",
        "Also, we drop the color RGB information for 30 % of the data points for each class to have more robust model training.\n",
        "Finally, we save this generated dataset for future segmentation model training usage.\n",
        "\n",
        "#### 2. Generation of 3D Data Patch Dataset with Extra Features\n",
        "\n",
        "Post this, we also generate a dataset variant with calculated normal information to append extra features into the generated point cloud dataset. And, also randomly drop 30 % of these calculated normals to have more robust segmentation model training.\n",
        "\n",
        "#### 3. Visualization of Data Patches in the 3D Space\n",
        "\n",
        "This visualization helps in verifying the upsampling logic used to upsample the apple point cloud data and further look into the generated data patches qualitatively."
      ],
      "metadata": {
        "id": "c6HyWJ2IwPHy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# installing open3d package\n",
        "!pip install open3d"
      ],
      "metadata": {
        "id": "OK-8kwCc6okZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0CyA1iEpKFMc",
        "outputId": "450b191b-fd8f-48ff-96f4-ad93529d0700"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ],
      "source": [
        "# for loading the dataset into the runtime\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LKEdWvZhKNII",
        "outputId": "2399b3e9-0be8-4be5-95bb-7974c7dbbfb0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "drive/MyDrive/point-cloud-prototyping/datasets/fuji-sfm-pcl-processed.zip\n",
            "drive/MyDrive/point-cloud-prototyping/datasets/pfuji-size-pcl-processed-2018.zip\n",
            "drive/MyDrive/point-cloud-prototyping/datasets/pfuji-size-pcl-processed-2020.zip\n"
          ]
        }
      ],
      "source": [
        "# processed pcl numpy arrays of fuji apple datasets\n",
        "!ls drive/MyDrive/point-cloud-prototyping/datasets/*processed*.zip"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# unzip these files into the current runtime location for further processing\n",
        "!unzip drive/MyDrive/point-cloud-prototyping/datasets/fuji-sfm-pcl-processed.zip -d .\n",
        "!unzip drive/MyDrive/point-cloud-prototyping/datasets/pfuji-size-pcl-processed-2018.zip -d .\n",
        "!unzip drive/MyDrive/point-cloud-prototyping/datasets/pfuji-size-pcl-processed-2020.zip -d ."
      ],
      "metadata": {
        "id": "yZ-_jjMMoMe1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# import statements for this script\n",
        "import os\n",
        "import random\n",
        "import numpy as np\n",
        "import open3d as o3d\n",
        "from tqdm import tqdm\n",
        "from copy import deepcopy"
      ],
      "metadata": {
        "id": "rSciTelP6sC8"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# copy files to other directory\n",
        "import shutil\n",
        "# list all files in the directory\n",
        "from os import listdir\n",
        "from os.path import isfile, join"
      ],
      "metadata": {
        "id": "Co2b1A5o3Za4"
      },
      "execution_count": 163,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# fixing randomization seed values\n",
        "random.seed(16)\n",
        "np.random.seed(16)"
      ],
      "metadata": {
        "id": "3cII1PMSCuVE"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# utility functions for patch generation task\n",
        "# PC max size for visualization for plotly\n",
        "MAX_PC_SIZE = 40960 # RandLA-Net's input dimension size\n",
        "\n",
        "# procedural logic for the upsampling function\n",
        "\n",
        "# separate out apple class sub array with rgb values\n",
        "# build a KDTree of the separated out sub array\n",
        "# loop over upsampling_factor number of points\n",
        "# then random point selection based on index\n",
        "# post that, query the nearest neighbor to that random point\n",
        "# preserve rgb information, separate array for actual upsampling operation\n",
        "\n",
        "# also, update the whole PC to create new upsampling temporary PC\n",
        "# after certain number of upsampled new PC size chunk is created\n",
        "# for better upsampling consistency.\n",
        "\n",
        "def upsample_minor_class(pcl_arr, upsampling_factor = 1.75):\n",
        "    assert upsampling_factor >= 1.0 and upsampling_factor <= 3.0\n",
        "    pcl_arr_cls = pcl_arr[pcl_arr[:, -1] == 1]\n",
        "    pcl_arr_bg = pcl_arr[pcl_arr[:, -1] == 0]\n",
        "    # open3d pcl processing\n",
        "    pcd = o3d.geometry.PointCloud()\n",
        "    pcd.points = o3d.utility.Vector3dVector(pcl_arr_cls[:,0:3])\n",
        "    # building KD-Tree for NN query\n",
        "    pcd_tree = o3d.geometry.KDTreeFlann(pcd)\n",
        "    upsample_count = int((upsampling_factor - 1.0)*pcl_arr_cls.shape[0])\n",
        "    # upsampled points nested list\n",
        "    upsampled_points = []\n",
        "    umsample_limit_flg = 0\n",
        "    for i in tqdm(range(upsample_count)):\n",
        "        rand_ind = random.randint(0,pcl_arr_cls.shape[0]-1) \n",
        "        [k, idx_vals, _] = pcd_tree.search_knn_vector_3d(pcd.points[rand_ind], 4)\n",
        "        idx_vals = idx_vals[1:]\n",
        "        rand_neighbor = random.choice(idx_vals)\n",
        "        # creating new upsampled points\n",
        "        arr_vals = (pcl_arr_cls[rand_ind] + pcl_arr_cls[rand_neighbor]) / 2\n",
        "        upsampled_points.append(list(arr_vals))\n",
        "        if len(upsampled_points) > 0.8 * pcl_arr_cls.shape[0] and umsample_limit_flg == 0:\n",
        "            pcl_arr_cls =  np.concatenate((pcl_arr_cls, np.array(upsampled_points)), axis=0)\n",
        "            pcd = o3d.geometry.PointCloud()\n",
        "            pcd.points = o3d.utility.Vector3dVector(pcl_arr_cls[:,0:3])\n",
        "            # building KD-Tree for NN query\n",
        "            pcd_tree = o3d.geometry.KDTreeFlann(pcd)\n",
        "            umsample_limit_flg = 1\n",
        "            upsampled_points = []\n",
        "    if len(upsampled_points) != 0:\n",
        "        pcl_arr_cls =  np.concatenate((pcl_arr_cls, np.array(upsampled_points)), axis=0)\n",
        "    pcl_arr_final = np.concatenate((pcl_arr_cls, pcl_arr_bg), axis=0)\n",
        "    del pcd\n",
        "    del pcd_tree\n",
        "    return pcl_arr_final\n",
        "\n",
        "# downsampling the PC size for standardization of visualized 'cls' & 'bg' PC size\n",
        "def downsample_pcl(pcl_arr, downsampling_factor = 0.75):\n",
        "    assert downsampling_factor >= 0.2 and downsampling_factor < 1.0\n",
        "    downsampled_pc_count = int(downsampling_factor * pcl_arr.shape[0])\n",
        "    if downsampled_pc_count > MAX_PC_SIZE:\n",
        "        downsampled_pc_count = MAX_PC_SIZE\n",
        "    idx_cls = np.random.randint((pcl_arr.shape[0] - 1),\n",
        "                                size = downsampled_pc_count)\n",
        "    pcl_arr = pcl_arr[idx_cls,:] \n",
        "    return pcl_arr"
      ],
      "metadata": {
        "id": "BabziGkf1xX-"
      },
      "execution_count": 124,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the location of the unzipped numpy arrays\n",
        "!ls *.npy"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jGzW6YobtYmE",
        "outputId": "cc41ce36-daac-4b83-cfd6-62456318b27a"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "fuji-sfm-pcl-processed.npy\t   pfuji-size-pcl-processed-2020.npy\n",
            "pfuji-size-pcl-processed-2018.npy\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "b5A_cKbtKNJo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a673c33c-3323-4729-b329-58947972f068"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(7153845, 7) (16473962, 7) (14213504, 7)\n"
          ]
        }
      ],
      "source": [
        "# loading arrays for basic exploratory quantitative analysis\n",
        "fuji_sfm_arr = np.load('fuji-sfm-pcl-processed.npy')\n",
        "pfuji_size_2018_arr = np.load('pfuji-size-pcl-processed-2018.npy')\n",
        "pfuji_size_2020_arr = np.load('pfuji-size-pcl-processed-2020.npy')\n",
        "print(fuji_sfm_arr.shape, pfuji_size_2018_arr.shape, pfuji_size_2020_arr.shape)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# checking ratio and fraction of total pc occupied by the apple class point clouds\n",
        "print(fuji_sfm_arr[fuji_sfm_arr[:,6] == 1][:, 1].shape[0] / fuji_sfm_arr[fuji_sfm_arr[:,6] == 0][:, 0].shape[0], \\\n",
        "    fuji_sfm_arr[fuji_sfm_arr[:,6] == 1][:, 1].shape[0] / (fuji_sfm_arr[fuji_sfm_arr[:,6] == 0][:, 0].shape[0] + fuji_sfm_arr[fuji_sfm_arr[:,6] == 1][:, 1].shape[0]))\n",
        "\n",
        "print(pfuji_size_2018_arr[pfuji_size_2018_arr[:,6] == 1][:, 1].shape[0] / pfuji_size_2018_arr[pfuji_size_2018_arr[:,6] == 0][:, 0].shape[0], \\\n",
        "    pfuji_size_2018_arr[pfuji_size_2018_arr[:,6] == 1][:, 1].shape[0] / (pfuji_size_2018_arr[pfuji_size_2018_arr[:,6] == 0][:, 0].shape[0] + pfuji_size_2018_arr[pfuji_size_2018_arr[:,6] == 1][:, 1].shape[0]))\n",
        "\n",
        "print(pfuji_size_2020_arr[pfuji_size_2020_arr[:,6] == 1][:, 1].shape[0] / pfuji_size_2020_arr[pfuji_size_2020_arr[:,6] == 0][:, 0].shape[0], \\\n",
        "    pfuji_size_2020_arr[pfuji_size_2020_arr[:,6] == 1][:, 1].shape[0] / (pfuji_size_2020_arr[pfuji_size_2020_arr[:,6] == 0][:, 0].shape[0] + pfuji_size_2020_arr[pfuji_size_2020_arr[:,6] == 1][:, 1].shape[0]))\n",
        "\n",
        "# twice the upsampling added for each patch with more than 10K data points\n",
        "# and apple class objects being present"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N2yaj932yNhr",
        "outputId": "e4b98124-cc47-46fc-aa18-c7603c590773"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.5389213119620704 0.3501941962678811\n",
            "0.4086897735419506 0.29012049438987414\n",
            "0.29237205647537456 0.22622901432328016\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the pcl data limit points to create window paritions for patch generation\n",
        "# x-axis coordinate limits\n",
        "print(np.min(fuji_sfm_arr[:,0]), np.max(fuji_sfm_arr[:,0]), (np.max(fuji_sfm_arr[:,0]) - np.min(fuji_sfm_arr[:,0])) )   \n",
        "# y-axis coordinate limits\n",
        "print(np.min(fuji_sfm_arr[:,1]), np.max(fuji_sfm_arr[:,1]), (np.max(fuji_sfm_arr[:,1]) - np.min(fuji_sfm_arr[:,1])) )\n",
        "# z-axis coordinate limits\n",
        "print(np.min(fuji_sfm_arr[:,2]), np.max(fuji_sfm_arr[:,2]), (np.max(fuji_sfm_arr[:,2]) - np.min(fuji_sfm_arr[:,2])) )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ms5q3C9ou_M1",
        "outputId": "1a0ec1a1-ed09-43e4-ae32-c7e3ffad6872"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "11.4000082 13.14999771 1.749989509999999\n",
            "67.25000763 75.0 7.749992370000001\n",
            "308.75 312.49996948 3.7499694800000043\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "1eVXPFL5KNN6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7ee24a96-e7dd-4d05-81a3-484e171e970d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-0.75 0.74999917 1.4999991700000002\n",
            "-0.24999963 2.9999995200000003 3.2499991500000003\n",
            "8.73e-06 3.24999905 3.24999032\n"
          ]
        }
      ],
      "source": [
        "# checking the pcl data limit points to create window paritions for patch generation\n",
        "# x-axis coordinate limits\n",
        "print(np.min(pfuji_size_2018_arr[:,0]), np.max(pfuji_size_2018_arr[:,0]), (np.max(pfuji_size_2018_arr[:,0]) - np.min(pfuji_size_2018_arr[:,0])) )   \n",
        "# y-axis coordinate limits\n",
        "print(np.min(pfuji_size_2018_arr[:,1]), np.max(pfuji_size_2018_arr[:,1]), (np.max(pfuji_size_2018_arr[:,1]) - np.min(pfuji_size_2018_arr[:,1])) )\n",
        "# z-axis coordinate limits\n",
        "print(np.min(pfuji_size_2018_arr[:,2]), np.max(pfuji_size_2018_arr[:,2]), (np.max(pfuji_size_2018_arr[:,2]) - np.min(pfuji_size_2018_arr[:,2])) )"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# checking the pcl data limit points to create window paritions for patch generation\n",
        "# x-axis coordinate limits\n",
        "print(np.min(pfuji_size_2020_arr[:,0]), np.max(pfuji_size_2020_arr[:,0]), (np.max(pfuji_size_2020_arr[:,0]) - np.min(pfuji_size_2020_arr[:,0])) )   \n",
        "# y-axis coordinate limits\n",
        "print(np.min(pfuji_size_2020_arr[:,1]), np.max(pfuji_size_2020_arr[:,1]), (np.max(pfuji_size_2020_arr[:,1]) - np.min(pfuji_size_2020_arr[:,1])) )\n",
        "# z-axis coordinate limits\n",
        "print(np.min(pfuji_size_2020_arr[:,2]), np.max(pfuji_size_2020_arr[:,2]), (np.max(pfuji_size_2020_arr[:,2]) - np.min(pfuji_size_2020_arr[:,2])) )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "prn99WhhvGOn",
        "outputId": "159eff7e-d542-4ac9-c1df-883296e5eba8"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-0.99996859 0.99989963 1.9998682200000002\n",
            "-0.5 2.75 3.25\n",
            "3.37e-06 2.99999905 2.99999568\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# create a 30 % rgb info dropper function\n",
        "def random_rgb_drop(pcl_arr, drop_frac = 0.3):\n",
        "    pcl_arr_cls = pcl_arr[pcl_arr[:,6] == 1]\n",
        "    pcl_arr_bg = pcl_arr[pcl_arr[:,6] == 0]\n",
        "\n",
        "    idx_cls = np.random.choice(np.arange(pcl_arr_cls.shape[0]), replace=False,\n",
        "                           size=int(pcl_arr_cls.shape[0] * drop_frac))\n",
        "    idx_bg = np.random.choice(np.arange(pcl_arr_bg.shape[0]), replace=False,\n",
        "                           size=int(pcl_arr_bg.shape[0] * drop_frac))\n",
        "    \n",
        "    pcl_arr_cls[idx_cls, 3:6] = 0.0\n",
        "    pcl_arr_bg[idx_bg, 3:6] = 0.0\n",
        "\n",
        "    pcl_arr =  np.concatenate((pcl_arr_cls, pcl_arr_bg), axis=0)\n",
        "    return pcl_arr"
      ],
      "metadata": {
        "id": "KdjQZcub4T2j"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# drop rgb information for the pcl base arrays in proper to their class sizes\n",
        "fuji_sfm_arr[:, 3:6] = fuji_sfm_arr[:, 3:6]/255\n",
        "fuji_sfm_arr = random_rgb_drop(fuji_sfm_arr)\n",
        "pfuji_size_2018_arr = random_rgb_drop(pfuji_size_2018_arr)\n",
        "pfuji_size_2020_arr = random_rgb_drop(pfuji_size_2020_arr)"
      ],
      "metadata": {
        "id": "kh0o7c5yCWSs"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create normal estimation function and \n",
        "# create a 30 % normal info dropper function\n",
        "def norm_estimate_and_drop(pcl_arr, drop_frac = 0.3):\n",
        "    pcd = o3d.geometry.PointCloud()\n",
        "    pcd.points = o3d.utility.Vector3dVector(pcl_arr[:,0:3])\n",
        "    pcd.estimate_normals() # computationally intensive step\n",
        "    pcd_arr = np.column_stack((np.asarray(pcd.points), pcl_arr[:,3:6], np.asarray(pcd.normals),\n",
        "                               pcl_arr[:, 6]))\n",
        "    del pcd # clear object from the memory\n",
        "\n",
        "    pcd_arr_cls = pcd_arr[pcd_arr[:,9] == 1]\n",
        "    pcd_arr_bg = pcd_arr[pcd_arr[:,9] == 0]\n",
        "\n",
        "    idx_cls = np.random.choice(np.arange(pcd_arr_cls.shape[0]), replace=False,\n",
        "                           size=int(pcd_arr_cls.shape[0] * drop_frac))\n",
        "    idx_bg = np.random.choice(np.arange(pcd_arr_bg.shape[0]), replace=False,\n",
        "                           size=int(pcd_arr_bg.shape[0] * drop_frac))\n",
        "    \n",
        "    pcd_arr_cls[idx_cls, 6:9] = 0.0\n",
        "    pcd_arr_bg[idx_bg, 6:9] = 0.0\n",
        "\n",
        "    pcd_arr =  np.concatenate((pcd_arr_cls, pcd_arr_bg), axis=0)\n",
        "\n",
        "    del pcd_arr_cls # clear object from the memory\n",
        "    del pcd_arr_bg # clear object from the memory\n",
        "\n",
        "    return pcd_arr"
      ],
      "metadata": {
        "id": "jhZDyg0FDQBn"
      },
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "fuji_sfm_norm_arr = norm_estimate_and_drop(fuji_sfm_arr)"
      ],
      "metadata": {
        "id": "e9JTMt0YHzUV"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pfuji_size_2018_norm_arr = norm_estimate_and_drop(pfuji_size_2018_arr)\n",
        "pfuji_size_2020_norm_arr = norm_estimate_and_drop(pfuji_size_2020_arr)"
      ],
      "metadata": {
        "id": "EqYBeeThP1cy"
      },
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(fuji_sfm_norm_arr.shape, fuji_sfm_arr.shape)\n",
        "print(pfuji_size_2018_norm_arr.shape, pfuji_size_2018_arr.shape)\n",
        "print(pfuji_size_2020_norm_arr.shape, pfuji_size_2020_arr.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JTy7sAC4MhlV",
        "outputId": "ddf759db-c5e4-4992-c992-a5cd093c3174"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(7153845, 10) (7153845, 7)\n",
            "(16473962, 10) (16473962, 7)\n",
            "(14213504, 10) (14213504, 7)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# if data patch directories not created, create those directories\n",
        "# both for normal estimate and only with rgb information\n",
        "FUJI_SFM_TEMP_DIR = 'fuji-data-patches'\n",
        "NFUJI_SFM_TEMP_DIR = 'fuji-norm-patches'\n",
        "\n",
        "\n",
        "if not os.path.exists(FUJI_SFM_TEMP_DIR):\n",
        "    os.makedirs(FUJI_SFM_TEMP_DIR)\n",
        "if not os.path.exists(NFUJI_SFM_TEMP_DIR):\n",
        "    os.makedirs(NFUJI_SFM_TEMP_DIR)"
      ],
      "metadata": {
        "id": "gUsZXyZe2QRS"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# patch window size generation for the fuji sfm dataset\n",
        "wind_sfm_x = ( 13.15 - 11.4 ) / 3\n",
        "wind_sfm_y = ( 75.00 - 67.25 ) / 12\n",
        "wind_sfm_z = ( 312.50 - 308.75 ) / 6\n",
        "print(wind_sfm_x, wind_sfm_y, wind_sfm_z)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3wU5S-fz1iwi",
        "outputId": "1b13b7cf-90fe-48ea-8cc3-707e809689ae"
      },
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.5833333333333334 0.6458333333333334 0.625\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# patch window size generation for the pfuji size 2018 dataset\n",
        "wind_pfuji_2018_x = abs(( - 0.75 - 0.75 ) / 4)\n",
        "wind_pfuji_2018_y = abs(( -0.25 - 3.0 ) / 9)\n",
        "wind_pfuji_2018_z = abs(( 0 - 3.25 ) / 9)\n",
        "print(wind_pfuji_2018_x, wind_pfuji_2018_y, wind_pfuji_2018_z)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wCZ4m-xhRiz_",
        "outputId": "70469906-ba4e-4545-8689-d8e799e49639"
      },
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.375 0.3611111111111111 0.3611111111111111\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# patch window size generation for the pfuji size 2020 dataset\n",
        "wind_pfuji_2020_x = abs(( - 1.0 - 1.0 ) / 4)\n",
        "wind_pfuji_2020_y = abs(( -0.5 - 2.75 ) / 7)\n",
        "wind_pfuji_2020_z = abs(( 0 - 3.0 ) / 6)\n",
        "print(wind_pfuji_2020_x, wind_pfuji_2020_y, wind_pfuji_2020_z)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lt3QiHrZRi8U",
        "outputId": "fa9bac8c-6c29-47ec-af2b-a13899bb2944"
      },
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.5 0.4642857142857143 0.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# global data patch counter for fuji datasets\n",
        "DATA_PATCH_COUNTER = 0\n",
        "NORM_DATA_PATCH_COUNTER = 0"
      ],
      "metadata": {
        "id": "w6Nqc13QaHNp"
      },
      "execution_count": 96,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# patch generator function to generate the pcl data patches for model training\n",
        "def patch_generator(pcl_arr, x_rng, y_rng, z_rng, \\\n",
        "                    wind_x, wind_y, wind_z, patch_path, patch_counter):\n",
        "    for i in range(0, x_rng):\n",
        "        for j in range(0, y_rng):\n",
        "            for k in range(0, z_rng):\n",
        "                pc_temp = pcl_arr[\n",
        "                                ( pcl_arr[:,0] > np.min(pcl_arr[:,0]) + wind_x*i ) &\n",
        "                                ( pcl_arr[:,0] <= np.min(pcl_arr[:,0]) + wind_x*(i+1) ) &\n",
        "\n",
        "                                ( pcl_arr[:,1] > np.min(pcl_arr[:,1]) + wind_y*j ) &\n",
        "                                ( pcl_arr[:,1] <= np.min(pcl_arr[:,1]) + wind_y*(j+1) ) &\n",
        "\n",
        "                                ( pcl_arr[:,2] > np.min(pcl_arr[:,2]) + wind_z*k ) &\n",
        "                                ( pcl_arr[:,2] <= np.min(pcl_arr[:,2]) + wind_z*(k+1) )\n",
        "                        ]\n",
        "\n",
        "                if pc_temp.shape[0] > 8192 and np.max(pc_temp[:,-1]) == 1.0:\n",
        "                    try:\n",
        "                        pc_temp = downsample_pcl(pc_temp)\n",
        "                        pc_temp = upsample_minor_class(pc_temp)\n",
        "                        patch_counter = patch_counter + 1\n",
        "                        file_name = patch_path + '/data_patch_'+str(patch_counter)+'.npy'\n",
        "                        np.save(file_name, pc_temp)\n",
        "                    except:\n",
        "                        pass\n",
        "\n",
        "    return (patch_counter + 1)"
      ],
      "metadata": {
        "id": "e689_DReURZp"
      },
      "execution_count": 104,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "DATA_PATCH_COUNTER = patch_generator(fuji_sfm_arr, 3, 12, 6,\n",
        "                                     wind_sfm_x, wind_sfm_y, wind_sfm_z,\n",
        "                                     FUJI_SFM_TEMP_DIR, DATA_PATCH_COUNTER)"
      ],
      "metadata": {
        "id": "ifeTk0myoHRo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "DATA_PATCH_COUNTER = patch_generator(pfuji_size_2018_arr, 4, 9, 9,\n",
        "                                     wind_pfuji_2018_x, wind_pfuji_2018_y, wind_pfuji_2018_z,\n",
        "                                     FUJI_SFM_TEMP_DIR, DATA_PATCH_COUNTER + 1)"
      ],
      "metadata": {
        "id": "ApJ4muLMrePa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "DATA_PATCH_COUNTER = patch_generator(pfuji_size_2020_arr, 4, 7, 6,\n",
        "                                     wind_pfuji_2020_x, wind_pfuji_2020_y, wind_pfuji_2020_z,\n",
        "                                     FUJI_SFM_TEMP_DIR, DATA_PATCH_COUNTER)"
      ],
      "metadata": {
        "id": "CesX9GLfuVdr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!ls fuji-norm-patches"
      ],
      "metadata": {
        "id": "sZ2b3X4NxKF1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "NORM_DATA_PATCH_COUNTER = patch_generator(fuji_sfm_norm_arr, 3, 12, 6,\n",
        "                                     wind_sfm_x, wind_sfm_y, wind_sfm_z,\n",
        "                                     NFUJI_SFM_TEMP_DIR, NORM_DATA_PATCH_COUNTER)\n",
        "                                     \n",
        "NORM_DATA_PATCH_COUNTER = patch_generator(pfuji_size_2018_norm_arr, 4, 9, 9,\n",
        "                                     wind_pfuji_2018_x, wind_pfuji_2018_y, wind_pfuji_2018_z,\n",
        "                                     NFUJI_SFM_TEMP_DIR, NORM_DATA_PATCH_COUNTER)\n",
        "                                     \n",
        "NORM_DATA_PATCH_COUNTER = patch_generator(pfuji_size_2020_norm_arr, 4, 7, 6,\n",
        "                                     wind_pfuji_2020_x, wind_pfuji_2020_y, wind_pfuji_2020_z,\n",
        "                                     NFUJI_SFM_TEMP_DIR, NORM_DATA_PATCH_COUNTER)"
      ],
      "metadata": {
        "id": "vEHB4L2Dsd2Z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# list all the created data patch files\n",
        "data_patches_file_lst = [f for f in listdir(FUJI_SFM_TEMP_DIR) if isfile(join(FUJI_SFM_TEMP_DIR, f))]\n",
        "norm_data_patches_file_lst = [f for f in listdir(NFUJI_SFM_TEMP_DIR) if isfile(join(NFUJI_SFM_TEMP_DIR, f))]"
      ],
      "metadata": {
        "id": "XuTLvgC0selm"
      },
      "execution_count": 138,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create final dataset directories for train, valid, test split\n",
        "# both for normal estimate and only with rgb information\n",
        "FUJI_SFM_DIR = 'fuji-dataset'\n",
        "FUJI_SFM_TRN_DIR = 'fuji-dataset/train'\n",
        "FUJI_SFM_VLD_DIR = 'fuji-dataset/valid'\n",
        "FUJI_SFM_TST_DIR = 'fuji-dataset/test'\n",
        "\n",
        "if not os.path.exists(FUJI_SFM_DIR):\n",
        "    os.makedirs(FUJI_SFM_TRN_DIR)\n",
        "    os.makedirs(FUJI_SFM_VLD_DIR)\n",
        "    os.makedirs(FUJI_SFM_TST_DIR)\n",
        "\n",
        "NFUJI_SFM_DIR = 'fuji-norm-dataset'\n",
        "NFUJI_SFM_TRN_DIR = 'fuji-norm-dataset/train'\n",
        "NFUJI_SFM_VLD_DIR = 'fuji-norm-dataset/valid'\n",
        "NFUJI_SFM_TST_DIR = 'fuji-norm-dataset/test'\n",
        "\n",
        "if not os.path.exists(NFUJI_SFM_DIR):\n",
        "    os.makedirs(NFUJI_SFM_DIR)\n",
        "    os.makedirs(NFUJI_SFM_TRN_DIR)\n",
        "    os.makedirs(NFUJI_SFM_VLD_DIR)\n",
        "    os.makedirs(NFUJI_SFM_TST_DIR)"
      ],
      "metadata": {
        "id": "yqtcc2sIseoa"
      },
      "execution_count": 187,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# random split of the lists to prepare training splits\n",
        "dp_fuji_val =  set(random.sample(data_patches_file_lst, 20))\n",
        "data_patches_file_lst = set(data_patches_file_lst) - dp_fuji_val\n",
        "dp_fuji_test =  set(random.sample(data_patches_file_lst, 20))\n",
        "data_patches_file_lst = set(data_patches_file_lst) - dp_fuji_test\n",
        "\n",
        "dp_fuji_val = list(dp_fuji_val)\n",
        "dp_fuji_test = list(dp_fuji_test)\n",
        "data_patches_file_lst = list(data_patches_file_lst)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F2ICRB_54IWJ",
        "outputId": "b0f9d831-9d9b-43b1-98bc-9a47a4bb6f97"
      },
      "execution_count": 159,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-159-16c41c14f336>:2: DeprecationWarning: Sampling from a set deprecated\n",
            "since Python 3.9 and will be removed in a subsequent version.\n",
            "  dp_fuji_val =  set(random.sample(data_patches_file_lst, 20))\n",
            "<ipython-input-159-16c41c14f336>:4: DeprecationWarning: Sampling from a set deprecated\n",
            "since Python 3.9 and will be removed in a subsequent version.\n",
            "  dp_fuji_test =  set(random.sample(data_patches_file_lst, 20))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(dp_fuji_val)\n",
        "print(dp_fuji_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ukH_HMhY4IYL",
        "outputId": "c6a4b867-77b6-40f0-ece0-b02c20a9fc06"
      },
      "execution_count": 162,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['data_patch_234.npy', 'data_patch_150.npy', 'data_patch_188.npy', 'data_patch_361.npy', 'data_patch_398.npy', 'data_patch_217.npy', 'data_patch_338.npy', 'data_patch_162.npy', 'data_patch_53.npy', 'data_patch_196.npy', 'data_patch_164.npy', 'data_patch_3.npy', 'data_patch_276.npy', 'data_patch_45.npy', 'data_patch_110.npy', 'data_patch_156.npy', 'data_patch_103.npy', 'data_patch_23.npy', 'data_patch_181.npy', 'data_patch_84.npy']\n",
            "['data_patch_123.npy', 'data_patch_97.npy', 'data_patch_362.npy', 'data_patch_344.npy', 'data_patch_51.npy', 'data_patch_261.npy', 'data_patch_264.npy', 'data_patch_357.npy', 'data_patch_209.npy', 'data_patch_154.npy', 'data_patch_363.npy', 'data_patch_350.npy', 'data_patch_377.npy', 'data_patch_393.npy', 'data_patch_171.npy', 'data_patch_121.npy', 'data_patch_95.npy', 'data_patch_345.npy', 'data_patch_369.npy', 'data_patch_232.npy']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# random split of the lists to prepare training splits\n",
        "norm_dp_fuji_val =  set(random.sample(norm_data_patches_file_lst, 20))\n",
        "norm_data_patches_file_lst = set(norm_data_patches_file_lst) - norm_dp_fuji_val\n",
        "norm_dp_fuji_test =  set(random.sample(norm_data_patches_file_lst, 20))\n",
        "norm_data_patches_file_lst = set(norm_data_patches_file_lst) - norm_dp_fuji_test\n",
        "\n",
        "norm_dp_fuji_val = list(norm_dp_fuji_val)\n",
        "norm_dp_fuji_test = list(norm_dp_fuji_test)\n",
        "norm_data_patches_file_lst = list(norm_data_patches_file_lst)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fjqUUW7E4IbB",
        "outputId": "e7f0445f-2466-4f6a-d639-67a8937a0073"
      },
      "execution_count": 164,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-164-f93e5e476586>:4: DeprecationWarning: Sampling from a set deprecated\n",
            "since Python 3.9 and will be removed in a subsequent version.\n",
            "  norm_dp_fuji_test =  set(random.sample(norm_data_patches_file_lst, 20))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(norm_dp_fuji_val)\n",
        "print(norm_dp_fuji_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nVk3OPn96rGc",
        "outputId": "efd38b06-e716-407d-f963-a0a5d940fb92"
      },
      "execution_count": 165,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['data_patch_6.npy', 'data_patch_256.npy', 'data_patch_132.npy', 'data_patch_65.npy', 'data_patch_109.npy', 'data_patch_327.npy', 'data_patch_54.npy', 'data_patch_298.npy', 'data_patch_297.npy', 'data_patch_343.npy', 'data_patch_368.npy', 'data_patch_379.npy', 'data_patch_45.npy', 'data_patch_249.npy', 'data_patch_131.npy', 'data_patch_236.npy', 'data_patch_352.npy', 'data_patch_40.npy', 'data_patch_347.npy', 'data_patch_58.npy']\n",
            "['data_patch_165.npy', 'data_patch_143.npy', 'data_patch_36.npy', 'data_patch_161.npy', 'data_patch_277.npy', 'data_patch_64.npy', 'data_patch_235.npy', 'data_patch_334.npy', 'data_patch_363.npy', 'data_patch_375.npy', 'data_patch_248.npy', 'data_patch_393.npy', 'data_patch_121.npy', 'data_patch_296.npy', 'data_patch_330.npy', 'data_patch_336.npy', 'data_patch_185.npy', 'data_patch_24.npy', 'data_patch_372.npy', 'data_patch_84.npy']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# three separate copy loops for the three data splits\n",
        "for file_name in data_patches_file_lst:\n",
        "    full_file_name = os.path.join(FUJI_SFM_TEMP_DIR, file_name)\n",
        "    if os.path.isfile(full_file_name):\n",
        "        shutil.copy(full_file_name, FUJI_SFM_TRN_DIR)\n",
        "\n",
        "for file_name in dp_fuji_val:\n",
        "    full_file_name = os.path.join(FUJI_SFM_TEMP_DIR, file_name)\n",
        "    if os.path.isfile(full_file_name):\n",
        "        shutil.copy(full_file_name, FUJI_SFM_VLD_DIR)\n",
        "\n",
        "for file_name in dp_fuji_test:\n",
        "    full_file_name = os.path.join(FUJI_SFM_TEMP_DIR, file_name)\n",
        "    if os.path.isfile(full_file_name):\n",
        "        shutil.copy(full_file_name, FUJI_SFM_TST_DIR)"
      ],
      "metadata": {
        "id": "PPmW-Ldk6rIN"
      },
      "execution_count": 178,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!ls fuji-dataset/valid"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rJ4h3KHC6rK7",
        "outputId": "935f671b-b5fa-4d81-c92e-e759744786e5"
      },
      "execution_count": 213,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "data_patch_103.npy  data_patch_164.npy\tdata_patch_234.npy  data_patch_398.npy\n",
            "data_patch_110.npy  data_patch_181.npy\tdata_patch_23.npy   data_patch_3.npy\n",
            "data_patch_150.npy  data_patch_188.npy\tdata_patch_276.npy  data_patch_45.npy\n",
            "data_patch_156.npy  data_patch_196.npy\tdata_patch_338.npy  data_patch_53.npy\n",
            "data_patch_162.npy  data_patch_217.npy\tdata_patch_361.npy  data_patch_84.npy\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# three separate copy loops for the three data splits\n",
        "for file_name in norm_data_patches_file_lst:\n",
        "    full_file_name = os.path.join(NFUJI_SFM_TEMP_DIR, file_name)\n",
        "    if os.path.isfile(full_file_name):\n",
        "        shutil.copy(full_file_name, NFUJI_SFM_TRN_DIR)\n",
        "\n",
        "for file_name in norm_dp_fuji_val:\n",
        "    full_file_name = os.path.join(NFUJI_SFM_TEMP_DIR, file_name)\n",
        "    if os.path.isfile(full_file_name):\n",
        "        shutil.copy(full_file_name, NFUJI_SFM_VLD_DIR)\n",
        "\n",
        "for file_name in norm_dp_fuji_test:\n",
        "    full_file_name = os.path.join(NFUJI_SFM_TEMP_DIR, file_name)\n",
        "    if os.path.isfile(full_file_name):\n",
        "        shutil.copy(full_file_name, NFUJI_SFM_TST_DIR)"
      ],
      "metadata": {
        "id": "-6YRvK5l9Iog"
      },
      "execution_count": 195,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# moving the processed point cloud datasets into the drive\n",
        "# TODO: update the below paths based on your project setup\n",
        "!zip -r drive/MyDrive/point-cloud-prototyping/datasets/fuji-complete-dataset.zip  fuji-dataset/"
      ],
      "metadata": {
        "id": "SwV6Sgf59IqQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO: update the below paths based on your project setup\n",
        "!zip -r drive/MyDrive/point-cloud-prototyping/datasets/fuji-norm-complete-dataset.zip  fuji-norm-dataset/"
      ],
      "metadata": {
        "id": "6cW7VQy29IsE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "NsLcULmL_Q3q"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}