notebooks/hps_demo.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "73d98b19-caa5-4574-8caa-a6e9472cfa77",
   "metadata": {},
   "source": [
    "# Copyright 2021 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\n",
    "# Each user is responsible for checking the content of datasets and the\n",
    "# applicable licenses and determining if suitable for the intended use."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2961ee55",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_hugectr_hps-demo/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# Hierarchical Parameter Server Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25c1747e",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In HugeCTR version 3.5, we provide Python APIs for embedding table lookup with [HugeCTR Hierarchical Parameter Server (HPS)](https://nvidia-merlin.github.io/HugeCTR/master/hugectr_core_features.html#hierarchical-parameter-server)\n",
    "HPS supports different database backends and GPU embedding caches.\n",
    "\n",
    "This notebook demonstrates how to use HPS with HugeCTR Python APIs. Without loss of generality, the HPS APIs are utilized together with the ONNX Runtime APIs to create an ensemble inference model, where HPS is responsible for embedding table lookup while the ONNX model takes charge of feed forward of dense neural networks.\n",
    "\n",
    "1. [Inference with HPS & ONNX](#section-1)\n",
    "2. [Lookup the Embedding Vector from DLPacke](#section-2)\n",
    "3. [Multi-process inferenceon](#section-3)\n",
    "4. [Redis Cluster deployment (without TLS/SSL)](#section-4)\n",
    "5. [Redis Cluster deployment (with TLS/SSL)](#section-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c420aed2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Setup\n",
    "\n",
    "To setup the environment, refer to [HugeCTR Example Notebooks](../notebooks) and follow the instructions there before running the following."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6ca5759",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Data Generation\n",
    "\n",
    "HugeCTR provides a tool to generate synthetic datasets. The [Data Generator](https://nvidia-merlin.github.io/HugeCTR/master/api/python_interface.html#data-generator-api) is capable of generating datasets of different file formats and different distributions. We will generate one-hot Parquet datasets with power-law distribution for this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ba5c7207",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[HCTR][06:31:47.413][INFO][RK0][main]: Generate Parquet dataset\n",
      "[HCTR][06:31:47.413][INFO][RK0][main]: train data folder: ./data_parquet, eval data folder: ./data_parquet, slot_size_array: 10000, 10000, 10000, 10000, nnz array: 1, 1, 1, 1, #files for train: 16, #files for eval: 4, #samples per file: 40960, Use power law distribution: 1, alpha of power law: 1.3\n",
      "[HCTR][06:31:47.416][INFO][RK0][main]: ./data_parquet exist\n",
      "[HCTR][06:31:47.423][INFO][RK0][main]: ./data_parquet/train/gen_0.parquet\n",
      "[HCTR][06:31:50.739][INFO][RK0][main]: ./data_parquet/train/gen_1.parquet\n",
      "[HCTR][06:31:50.846][INFO][RK0][main]: ./data_parquet/train/gen_2.parquet\n",
      "[HCTR][06:31:50.929][INFO][RK0][main]: ./data_parquet/train/gen_3.parquet\n",
      "[HCTR][06:31:51.011][INFO][RK0][main]: ./data_parquet/train/gen_4.parquet\n",
      "[HCTR][06:31:51.092][INFO][RK0][main]: ./data_parquet/train/gen_5.parquet\n",
      "[HCTR][06:31:51.171][INFO][RK0][main]: ./data_parquet/train/gen_6.parquet\n",
      "[HCTR][06:31:51.250][INFO][RK0][main]: ./data_parquet/train/gen_7.parquet\n",
      "[HCTR][06:31:51.329][INFO][RK0][main]: ./data_parquet/train/gen_8.parquet\n",
      "[HCTR][06:31:51.407][INFO][RK0][main]: ./data_parquet/train/gen_9.parquet\n",
      "[HCTR][06:31:51.485][INFO][RK0][main]: ./data_parquet/train/gen_10.parquet\n",
      "[HCTR][06:31:51.562][INFO][RK0][main]: ./data_parquet/train/gen_11.parquet\n",
      "[HCTR][06:31:51.638][INFO][RK0][main]: ./data_parquet/train/gen_12.parquet\n",
      "[HCTR][06:31:51.715][INFO][RK0][main]: ./data_parquet/train/gen_13.parquet\n",
      "[HCTR][06:31:51.792][INFO][RK0][main]: ./data_parquet/train/gen_14.parquet\n",
      "[HCTR][06:31:51.868][INFO][RK0][main]: ./data_parquet/train/gen_15.parquet\n",
      "[HCTR][06:31:51.962][INFO][RK0][main]: ./data_parquet/file_list.txt done!\n",
      "[HCTR][06:31:51.986][INFO][RK0][main]: ./data_parquet/val/gen_0.parquet\n",
      "[HCTR][06:31:52.064][INFO][RK0][main]: ./data_parquet/val/gen_1.parquet\n",
      "[HCTR][06:31:52.142][INFO][RK0][main]: ./data_parquet/val/gen_2.parquet\n",
      "[HCTR][06:31:52.218][INFO][RK0][main]: ./data_parquet/val/gen_3.parquet\n",
      "[HCTR][06:31:52.296][INFO][RK0][main]: ./data_parquet/file_list_test.txt done!\n"
     ]
    }
   ],
   "source": [
    "import hugectr\n",
    "from hugectr.tools import DataGeneratorParams, DataGenerator\n",
    "\n",
    "data_generator_params = DataGeneratorParams(\n",
    "  format = hugectr.DataReaderType_t.Parquet,\n",
    "  label_dim = 1,\n",
    "  dense_dim = 10,\n",
    "  num_slot = 4,\n",
    "  i64_input_key = True,\n",
    "  nnz_array = [1, 1, 1, 1],\n",
    "  source = \"./data_parquet/file_list.txt\",\n",
    "  eval_source = \"./data_parquet/file_list_test.txt\",\n",
    "  slot_size_array = [10000, 10000, 10000, 10000],\n",
    "  check_type = hugectr.Check_t.Non,\n",
    "  dist_type = hugectr.Distribution_t.PowerLaw,\n",
    "  power_law_type = hugectr.PowerLaw_t.Short,\n",
    "  num_files = 16,\n",
    "  eval_num_files = 4,\n",
    "  num_samples_per_file = 40960)\n",
    "data_generator = DataGenerator(data_generator_params)\n",
    "data_generator.generate()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc51dd73",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Train from Scratch\n",
    "\n",
    "We can train from scratch by performing the following steps with Python APIs:\n",
    "\n",
    "1. Create the solver, reader and optimizer, then initialize the model.\n",
    "2. Construct the model graph by adding input, sparse embedding and dense layers in order.\n",
    "3. Compile the model and have an overview of the model graph.\n",
    "4. Dump the model graph to the JSON file.\n",
    "5. Fit the model, save the model weights and optimizer states implicitly.\n",
    "6. Dump one batch of evaluation results to files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "de4fd9aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing train.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile train.py\n",
    "import os\n",
    "import hugectr\n",
    "from mpi4py import MPI\n",
    "import numpy as np\n",
    "solver = hugectr.CreateSolver(model_name = \"hps_demo\",\n",
    "                              max_eval_batches = 1,\n",
    "                              batchsize_eval = 1024,\n",
    "                              batchsize = 1024,\n",
    "                              lr = 0.001,\n",
    "                              vvgpu = [[0]],\n",
    "                              i64_input_key = True,\n",
    "                              repeat_dataset = True,\n",
    "                              use_cuda_graph = True)\n",
    "reader = hugectr.DataReaderParams(data_reader_type = hugectr.DataReaderType_t.Parquet,\n",
    "                                  source = [\"./data_parquet/file_list.txt\"],\n",
    "                                  eval_source = \"./data_parquet/file_list_test.txt\",\n",
    "                                  check_type = hugectr.Check_t.Non,\n",
    "                                  slot_size_array = [10000, 10000, 10000, 10000])\n",
    "optimizer = hugectr.CreateOptimizer(optimizer_type = hugectr.Optimizer_t.Adam)\n",
    "model = hugectr.Model(solver, reader, optimizer)\n",
    "model.add(hugectr.Input(label_dim = 1, label_name = \"label\",\n",
    "                        dense_dim = 10, dense_name = \"dense\",\n",
    "                        data_reader_sparse_param_array = \n",
    "                        [hugectr.DataReaderSparseParam(\"data1\", [1, 1], True, 2),\n",
    "                        hugectr.DataReaderSparseParam(\"data2\", [1, 1], True, 2)]))\n",
    "model.add(hugectr.SparseEmbedding(embedding_type = hugectr.Embedding_t.DistributedSlotSparseEmbeddingHash, \n",
    "                            workspace_size_per_gpu_in_mb = 4,\n",
    "                            embedding_vec_size = 16,\n",
    "                            combiner = \"sum\",\n",
    "                            sparse_embedding_name = \"sparse_embedding1\",\n",
    "                            bottom_name = \"data1\",\n",
    "                            optimizer = optimizer))\n",
    "model.add(hugectr.SparseEmbedding(embedding_type = hugectr.Embedding_t.DistributedSlotSparseEmbeddingHash, \n",
    "                            workspace_size_per_gpu_in_mb = 8,\n",
    "                            embedding_vec_size = 32,\n",
    "                            combiner = \"sum\",\n",
    "                            sparse_embedding_name = \"sparse_embedding2\",\n",
    "                            bottom_name = \"data2\",\n",
    "                            optimizer = optimizer))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Reshape,\n",
    "                            bottom_names = [\"sparse_embedding1\"],\n",
    "                            top_names = [\"reshape1\"],\n",
    "                            leading_dim=32))                            \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Reshape,\n",
    "                            bottom_names = [\"sparse_embedding2\"],\n",
    "                            top_names = [\"reshape2\"],\n",
    "                            leading_dim=64))                            \n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Concat,\n",
    "                            bottom_names = [\"reshape1\", \"reshape2\", \"dense\"], top_names = [\"concat1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"concat1\"],\n",
    "                            top_names = [\"fc1\"],\n",
    "                            num_output=1024))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc1\"],\n",
    "                            top_names = [\"relu1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"relu1\"],\n",
    "                            top_names = [\"fc2\"],\n",
    "                            num_output=1))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.BinaryCrossEntropyLoss,\n",
    "                            bottom_names = [\"fc2\", \"label\"],\n",
    "                            top_names = [\"loss\"]))\n",
    "model.compile()\n",
    "model.summary()\n",
    "model.graph_to_json(\"hps_demo.json\")\n",
    "model.fit(max_iter = 1100, display = 200, eval_interval = 1000, snapshot = 1000, snapshot_prefix = \"hps_demo\")\n",
    "\n",
    "ground_truth = model.check_out_tensor(\"fc2\", hugectr.Tensor_t.Evaluate)\n",
    "np.save(\"ground_truth.npy\", ground_truth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cd15bdae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HugeCTR Version: 23.8\n",
      "====================================================Model Init=====================================================\n",
      "[HCTR][06:32:11.556][INFO][RK0][main]: Initialize model: hps_demo\n",
      "[HCTR][06:32:11.556][INFO][RK0][main]: Global seed is 2598678435\n",
      "[HCTR][06:32:11.561][INFO][RK0][main]: Device to NUMA mapping:\n",
      "[HCTR][06:32:11.642][INFO][RK0][main]:   GPU 0 ->  node 0\n",
      "[HCTR][06:32:15.564][WARNING][RK0][main]: Peer-to-peer access cannot be fully enabled.\n",
      "[HCTR][06:32:15.564][DEBUG][RK0][main]: [device 0] allocating 0.0000 GB, available 30.0886 \n",
      "[HCTR][06:32:15.564][INFO][RK0][main]: Start all2all warmup\n",
      "[HCTR][06:32:15.565][INFO][RK0][main]: End all2all warmup\n",
      "[HCTR][06:32:15.566][INFO][RK0][main]: Using All-reduce algorithm: NCCL\n",
      "[HCTR][06:32:15.567][INFO][RK0][main]: Device 0: Tesla V100-SXM2-32GB\n",
      "[HCTR][06:32:15.636][INFO][RK0][main]: eval source ./data_parquet/file_list_test.txt max_row_group_size 40960\n",
      "[HCTR][06:32:15.808][INFO][RK0][main]: train source ./data_parquet/file_list.txt max_row_group_size 40960\n",
      "[HCTR][06:32:15.810][INFO][RK0][main]: num of DataReader workers for train: 1\n",
      "[HCTR][06:32:15.810][INFO][RK0][main]: num of DataReader workers for eval: 1\n",
      "[HCTR][06:32:15.937][INFO][RK0][main]: max_vocabulary_size_per_gpu_=21845\n",
      "[HCTR][06:32:15.938][DEBUG][RK0][main]: [device 0] allocating 0.0047 GB, available 29.6921 \n",
      "[HCTR][06:32:15.939][INFO][RK0][main]: max_vocabulary_size_per_gpu_=21845\n",
      "[HCTR][06:32:15.940][DEBUG][RK0][main]: [device 0] allocating 0.0092 GB, available 29.6824 \n",
      "[HCTR][06:32:15.940][INFO][RK0][main]: Graph analysis to resolve tensor dependency\n",
      "[HCTR][06:32:15.940][WARNING][RK0][main]: You are using reshape layer with parameter leading_dim. This will be deprecated in the future. Please switch to parameter shape.[HCTR][06:32:15.940][WARNING][RK0][main]: You are using reshape layer with parameter leading_dim. This will be deprecated in the future. Please switch to parameter shape.[HCTR][06:32:15.946][WARNING][RK0][main]: You are using reshape layer with parameter leading_dim. This will be deprecated in the future. Please switch to parameter shape.[HCTR][06:32:15.946][WARNING][RK0][main]: You are using reshape layer with parameter leading_dim. This will be deprecated in the future. Please switch to parameter shape.===================================================Model Compile===================================================\n",
      "[HCTR][06:32:17.205][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][06:32:17.205][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][06:32:17.205][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][06:32:17.206][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][06:32:17.207][INFO][RK0][main]: Starting AUC NCCL warm-up\n",
      "[HCTR][06:32:17.208][INFO][RK0][main]: Warm-up done\n",
      "===================================================Model Summary===================================================\n",
      "[HCTR][06:32:17.208][INFO][RK0][main]: Model structure on each GPU\n",
      "Label                                   Dense                         Sparse                        \n",
      "label                                   dense                          data1,data2                   \n",
      "(1024, 1)                               (1024, 10)                              \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "Layer Type                              Input Name                    Output Name                   Output Shape                  \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "DistributedSlotSparseEmbeddingHash      data1                         sparse_embedding1             (1024, 2, 16)                 \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "DistributedSlotSparseEmbeddingHash      data2                         sparse_embedding2             (1024, 2, 32)                 \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Reshape                                 sparse_embedding1             reshape1                      (1024, 32)                    \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Reshape                                 sparse_embedding2             reshape2                      (1024, 64)                    \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Concat                                  reshape1                      concat1                       (1024, 106)                   \n",
      "                                        reshape2                                                                                  \n",
      "                                        dense                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            concat1                       fc1                           (1024, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc1                           relu1                         (1024, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            relu1                         fc2                           (1024, 1)                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "BinaryCrossEntropyLoss                  fc2                           loss                                                        \n",
      "                                        label                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "[HCTR][06:32:17.212][INFO][RK0][main]: Save the model graph to hps_demo.json successfully\n",
      "=====================================================Model Fit=====================================================\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Use non-epoch mode with number of iterations: 1100\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Training batchsize: 1024, evaluation batchsize: 1024\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Evaluation interval: 1000, snapshot interval: 1000\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Dense network trainable: True\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Sparse embedding sparse_embedding1 trainable: True\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Sparse embedding sparse_embedding2 trainable: True\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Use mixed precision: False, scaler: 1.000000, use cuda graph: True\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: lr: 0.001000, warmup_steps: 1, end_lr: 0.000000\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: decay_start: 0, decay_steps: 1, decay_power: 2.000000\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Training source file: ./data_parquet/file_list.txt\n",
      "[HCTR][06:32:17.213][INFO][RK0][main]: Evaluation source file: ./data_parquet/file_list_test.txt\n",
      "[HCTR][06:32:17.658][INFO][RK0][main]: Iter: 200 Time(200 iters): 0.444961s Loss: 0.693355 lr:0.001\n",
      "[HCTR][06:32:18.167][INFO][RK0][main]: Iter: 400 Time(200 iters): 0.508793s Loss: 0.694358 lr:0.001\n",
      "[HCTR][06:32:18.589][INFO][RK0][main]: Iter: 600 Time(200 iters): 0.422282s Loss: 0.695494 lr:0.001\n",
      "[HCTR][06:32:18.764][INFO][RK0][main]: Iter: 800 Time(200 iters): 0.175263s Loss: 0.691037 lr:0.001\n",
      "[HCTR][06:32:18.939][INFO][RK0][main]: Iter: 1000 Time(200 iters): 0.174492s Loss: 0.688767 lr:0.001\n",
      "[HCTR][06:32:18.940][INFO][RK0][main]: Evaluation, AUC: 0.503806\n",
      "[HCTR][06:32:18.940][INFO][RK0][main]: Eval Time for 1 iters: 0.000913s\n",
      "[HCTR][06:32:18.941][INFO][RK0][main]: Rank0: Write hash table to file\n",
      "[HCTR][06:32:19.024][INFO][RK0][main]: Rank0: Write hash table to file\n",
      "[HCTR][06:32:19.092][INFO][RK0][main]: Dumping sparse weights to files, successful\n",
      "[HCTR][06:32:19.093][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HCTR][06:32:19.123][INFO][RK0][main]: Done\n",
      "[HCTR][06:32:19.123][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HCTR][06:32:19.148][INFO][RK0][main]: Done\n",
      "[HCTR][06:32:19.150][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HCTR][06:32:19.203][INFO][RK0][main]: Done\n",
      "[HCTR][06:32:19.203][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HCTR][06:32:19.252][INFO][RK0][main]: Done\n",
      "[HCTR][06:32:19.252][INFO][RK0][main]: Dumping sparse optimzer states to files, successful\n",
      "[HCTR][06:32:19.262][INFO][RK0][main]: Dumping dense weights to file, successful\n",
      "[HCTR][06:32:19.279][INFO][RK0][main]: Dumping dense optimizer states to file, successful\n",
      "[HCTR][06:32:19.368][INFO][RK0][main]: Finish 1100 iterations with batchsize: 1024 in 2.16s.\n"
     ]
    }
   ],
   "source": [
    "!python3 train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07ab2648",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Convert HugeCTR to ONNX\n",
    "\n",
    "We will convert the saved HugeCTR models to ONNX using the HugeCTR to ONNX Converter. For more information about the converter, refer to the README in the [onnx_converter](https://github.com/NVIDIA-Merlin/HugeCTR/tree/master/onnx_converter) directory of the repository.\n",
    "\n",
    "For the sake of double checking the correctness, we will investigate both cases of conversion depending on whether or not to convert the sparse embedding models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "859c99fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[HUGECTR2ONNX][INFO]: Converting Data layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting DistributedSlotSparseEmbeddingHash layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting DistributedSlotSparseEmbeddingHash layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting Reshape layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting Reshape layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting Concat layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting InnerProduct layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting ReLU layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting InnerProduct layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting Sigmoid layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: The model is checked!\n",
      "[HUGECTR2ONNX][INFO]: The model is saved at hps_demo_with_embedding.onnx\n",
      "[HUGECTR2ONNX][INFO]: Converting Data layer to ONNX\n",
      "Skip sparse embedding layers in converted ONNX model\n",
      "[HUGECTR2ONNX][INFO]: Converting DistributedSlotSparseEmbeddingHash layer to ONNX\n",
      "Skip sparse embedding layers in converted ONNX model\n",
      "[HUGECTR2ONNX][INFO]: Converting DistributedSlotSparseEmbeddingHash layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting Reshape layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting Reshape layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting Concat layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting InnerProduct layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting ReLU layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting InnerProduct layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: Converting Sigmoid layer to ONNX\n",
      "[HUGECTR2ONNX][INFO]: The model is checked!\n",
      "[HUGECTR2ONNX][INFO]: The model is saved at hps_demo_without_embedding.onnx\n"
     ]
    }
   ],
   "source": [
    "import hugectr2onnx\n",
    "hugectr2onnx.converter.convert(onnx_model_path = \"hps_demo_with_embedding.onnx\",\n",
    "                            graph_config = \"hps_demo.json\",\n",
    "                            dense_model = \"hps_demo_dense_1000.model\",\n",
    "                            convert_embedding = True,\n",
    "                            sparse_models = [\"hps_demo0_sparse_1000.model\", \"hps_demo1_sparse_1000.model\"])\n",
    "\n",
    "hugectr2onnx.converter.convert(onnx_model_path = \"hps_demo_without_embedding.onnx\",\n",
    "                            graph_config = \"hps_demo.json\",\n",
    "                            dense_model = \"hps_demo_dense_1000.model\",\n",
    "                            convert_embedding = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "604b8923",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "<a id=\"section-1\"></a>\n",
    "## 1. Inference with HPS & ONNX\n",
    "\n",
    "We will make inference by performing the following steps with Python APIs:\n",
    "\n",
    "1. Configure the HPS hyperparameters. Please refer to [hps configuration](https://nvidia-merlin.github.io/HugeCTR/main/hugectr_parameter_server.html#inference-parameters-and-embedding-cache-configuration) for detailed configurations.\n",
    "2. Initialize the HPS object, which is responsible for embedding table lookup.\n",
    "3. Loading the Parquet data.\n",
    "4. Make inference with the HPS object and the ONNX inference session of `hps_demo_without_embedding.onnx`.\n",
    "5. Check the correctness by comparing with dumped evaluation results.\n",
    "6. Make inference with the ONNX inference session of `hps_demo_with_embedding.onnx` (double check)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f1650d32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[HCTR][06:32:40.791][WARNING][RK0][main]: default_value_for_each_table.size() is not equal to the number of embedding tables\n",
      "====================================================HPS Create====================================================\n",
      "[HCTR][06:32:40.791][INFO][RK0][main]: Creating HashMap CPU database backend...\n",
      "[HCTR][06:32:40.791][DEBUG][RK0][main]: Created blank database backend in local memory!\n",
      "[HCTR][06:32:40.791][INFO][RK0][main]: Volatile DB: initial cache rate = 1\n",
      "[HCTR][06:32:40.791][INFO][RK0][main]: Volatile DB: cache missed embeddings = 0\n",
      "[HCTR][06:32:40.791][DEBUG][RK0][main]: Created raw model loader in local memory!\n",
      "[HCTR][06:32:41.123][INFO][RK0][main]: Table: hps_et.hps_demo.sparse_embedding1; cached 18488 / 18488 embeddings in volatile database (HashMapBackend); load: 18488 / 18446744073709551615 (0.00%).\n",
      "[HCTR][06:32:41.431][INFO][RK0][main]: Table: hps_et.hps_demo.sparse_embedding2; cached 18470 / 18470 embeddings in volatile database (HashMapBackend); load: 18470 / 18446744073709551615 (0.00%).\n",
      "[HCTR][06:32:41.431][DEBUG][RK0][main]: Real-time subscribers created!\n",
      "[HCTR][06:32:41.431][INFO][RK0][main]: Creating embedding cache in device 0.\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: Model name: hps_demo\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: Max batch size: 1024\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: Fuse embedding tables: False\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: Number of embedding tables: 2\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: Use GPU embedding cache: True, cache size percentage: 0.500000\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: Embedding cache type: dynamic\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: Use I64 input key: True\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: Configured cache hit rate threshold: 1.000000\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: The size of thread pool: 80\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: The size of worker memory pool: 2\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: The size of refresh memory pool: 1\n",
      "[HCTR][06:32:41.437][INFO][RK0][main]: The refresh percentage : 0.000000\n",
      "[HCTR][06:32:41.453][INFO][RK0][main]: LookupSession i64_input_key: True\n",
      "[HCTR][06:32:41.453][INFO][RK0][main]: Creating lookup session for hps_demo on device: 0\n",
      "ground_truth:  [0.4895492  0.509022   0.38192913 ... 0.5264926  0.50650454 0.47927693]\n",
      "pred:  [[0.48954916]\n",
      " [0.50902206]\n",
      " [0.38192907]\n",
      " ...\n",
      " [0.52649266]\n",
      " [0.5065045 ]\n",
      " [0.4792769 ]]\n",
      "mse between pred and ground_truth:  2.3887142e-15\n",
      "pred_ref:  [[0.48954916]\n",
      " [0.50902206]\n",
      " [0.38192907]\n",
      " ...\n",
      " [0.52649266]\n",
      " [0.5065045 ]\n",
      " [0.4792769 ]]\n",
      "mse between pred_ref and ground_truth:  2.3887142e-15\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-09-20 06:32:41.566238532 [W:onnxruntime:, graph.cc:3543 CleanUnusedInitializersAndNodeArgs] Removing initializer 'key_to_indice_hash_all_tables'. It is not used by any node and should be removed from the model.\n"
     ]
    }
   ],
   "source": [
    "from hugectr.inference import HPS, ParameterServerConfig, InferenceParams\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import onnxruntime as ort\n",
    "\n",
    "slot_size_array = [10000, 10000, 10000, 10000]\n",
    "key_offset = np.insert(np.cumsum(slot_size_array), 0, 0)[:-1]\n",
    "batch_size = 1024\n",
    "\n",
    "# 1. Configure the HPS hyperparameters\n",
    "ps_config = ParameterServerConfig(\n",
    "           emb_table_name = {\"hps_demo\": [\"sparse_embedding1\", \"sparse_embedding2\"]},\n",
    "           embedding_vec_size = {\"hps_demo\": [16, 32]},\n",
    "           max_feature_num_per_sample_per_emb_table = {\"hps_demo\": [2, 2]},\n",
    "           inference_params_array = [\n",
    "              InferenceParams(\n",
    "                model_name = \"hps_demo\",\n",
    "                max_batchsize = batch_size,\n",
    "                hit_rate_threshold = 1.0,\n",
    "                dense_model_file = \"\",\n",
    "                sparse_model_files = [\"hps_demo0_sparse_1000.model\", \"hps_demo1_sparse_1000.model\"],\n",
    "                deployed_devices = [0],\n",
    "                use_gpu_embedding_cache = True,\n",
    "                cache_size_percentage = 0.5,\n",
    "                i64_input_key = True)\n",
    "           ])\n",
    "\n",
    "# 2. Initialize the HPS object\n",
    "hps = HPS(ps_config)\n",
    "\n",
    "# 3. Loading the Parquet data.\n",
    "df = pd.read_parquet(\"data_parquet/val/gen_0.parquet\")\n",
    "dense_input_columns = df.columns[1:11]\n",
    "cat_input1_columns = df.columns[11:13]\n",
    "cat_input2_columns = df.columns[13:15]\n",
    "dense_input = df[dense_input_columns].loc[0:batch_size-1].to_numpy(dtype=np.float32)\n",
    "cat_input1 = (df[cat_input1_columns].loc[0:batch_size-1].to_numpy(dtype=np.int64) + key_offset[0:2]).reshape((batch_size, 2, 1))\n",
    "cat_input2 = (df[cat_input2_columns].loc[0:batch_size-1].to_numpy(dtype=np.int64) + key_offset[2:4]).reshape((batch_size, 2, 1))\n",
    "\n",
    "# 4. Make inference from the HPS object and the ONNX inference session of `hps_demo_without_embedding.onnx`.\n",
    "embedding1 = hps.lookup(cat_input1.flatten(), \"hps_demo\", 0).reshape(batch_size, 2, 16)\n",
    "embedding2 = hps.lookup(cat_input2.flatten(), \"hps_demo\", 1).reshape(batch_size, 2, 32)\n",
    "sess = ort.InferenceSession(\"hps_demo_without_embedding.onnx\")\n",
    "res = sess.run(output_names=[sess.get_outputs()[0].name],\n",
    "               input_feed={sess.get_inputs()[0].name: dense_input,\n",
    "               sess.get_inputs()[1].name: embedding1,\n",
    "               sess.get_inputs()[2].name: embedding2})\n",
    "pred = res[0]\n",
    "\n",
    "# 5. Check the correctness by comparing with dumped evaluation results.\n",
    "ground_truth = np.load(\"ground_truth.npy\").flatten()\n",
    "print(\"ground_truth: \", ground_truth)\n",
    "\n",
    "diff = pred.flatten()-ground_truth\n",
    "mse = np.mean(diff*diff)\n",
    "print(\"pred: \", pred)\n",
    "print(\"mse between pred and ground_truth: \", mse)\n",
    "\n",
    "# 6. Make inference with the ONNX inference session of `hps_demo_with_embedding.onnx` (double check).\n",
    "sess_ref = ort.InferenceSession(\"hps_demo_with_embedding.onnx\")\n",
    "res_ref = sess_ref.run(output_names=[sess_ref.get_outputs()[0].name],\n",
    "                   input_feed={sess_ref.get_inputs()[0].name: dense_input,\n",
    "                   sess_ref.get_inputs()[1].name: cat_input1,\n",
    "                   sess_ref.get_inputs()[2].name: cat_input2})\n",
    "pred_ref = res_ref[0]\n",
    "diff_ref = pred_ref.flatten()-ground_truth\n",
    "mse_ref = np.mean(diff_ref*diff_ref)\n",
    "print(\"pred_ref: \", pred_ref)\n",
    "print(\"mse between pred_ref and ground_truth: \", mse_ref)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a169887",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "<a id=\"section-2\"></a>\n",
    "## 2. Lookup the Embedding Vector from DLPack\n",
    "\n",
    "We also provide a `lookup_fromdlpack` interface that could query embedding keys on the `CPU` and return the embedding vectors on the `GPU/CPU`.\n",
    "\n",
    "1. Suppose you have created a Pytorch/Tensorflow tensor that stores the embedded keys.\n",
    "2. Convert the embedding key tensor to DLPack capsule through the corresponding platform's `to_dlpack` function.\n",
    "3. Creates an empty tensor as a buffer to store embedding vectors. \n",
    "4. Convert a buffer tensor to DLPack capsule.\n",
    "5. Lookup the embedding vector of the corresponding embedding key directly through `lookup_fromdlpack` interface, and output it to the embedding vector buffer tensor\n",
    "6. If the output capsule is allocated on the GPU, then a  `device_id` needs to be specified in `lookup_fromdlpack` interface for corresponding embedding cache. If not specified, the default value is device 0\n",
    "\n",
    "Note: Please make sure that tensorflow or pytorch have been installed correctly in the `merlin-hugectr` container:\n",
    "\n",
    "```bash\n",
    "pip install tensorflow\n",
    "pip install torch\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "37e4fd27",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Look up from dlpack for Pytorch tensor\n",
      "************Look up from pytorch dlpack on CPU\n",
      "The device type of embedding keys that lookup dlpack from hps interface for embedding table 0 of hps_demo: cpu, the keys: tensor([   85, 10028,     0,  ..., 10004,    10, 10000])\n",
      "[The device type of embedding vectors that lookup dlpack from hps interface for embedding table 0 of hps_demo: cpu, the vectors: tensor([[-0.0307,  0.0264, -0.0294,  ...,  0.0151, -0.0281,  0.0088]])\n",
      "\n",
      "Pytorch dlpack on cpu  results are consistent with native HPS lookup api, mse: 0.0\n",
      "************Look up from pytorch dlpack on GPU\n",
      "The device type of embedding vectors that lookup dlpack from hps interface for embedding table 0 of hps_demo: cuda:0, the vectors: tensor([-0.0307,  0.0264, -0.0294,  ...,  0.0151, -0.0281,  0.0088],\n",
      "       device='cuda:0')\n",
      "\n",
      "\n",
      "Pytorch dlpack on GPU results are consistent with native HPS lookup api, mse: 0.0\n"
     ]
    }
   ],
   "source": [
    "embedding1 = hps.lookup(cat_input1.flatten(), \"hps_demo\", 0).reshape(batch_size, 2, 16)\n",
    "embedding2 = hps.lookup(cat_input2.flatten(), \"hps_demo\", 1).reshape(batch_size, 2, 32)\n",
    "\n",
    "# 1. Look up from dlpack for Pytorch tensor on CPU\n",
    "print(\" Look up from dlpack for Pytorch tensor\")\n",
    "import torch.utils.dlpack\n",
    "import os\n",
    "print(\"************Look up from pytorch dlpack on CPU\")\n",
    "device = torch.device(\"cpu\")\n",
    "key = torch.tensor(cat_input1.flatten(),dtype=torch.int64, device=device)\n",
    "out = torch.empty((1,cat_input1.flatten().shape[0]*16), dtype=torch.float32, device=device)\n",
    "key_capsule = torch.utils.dlpack.to_dlpack(key)\n",
    "print(\"The device type of embedding keys that lookup dlpack from hps interface for embedding table 0 of hps_demo: {}, the keys: {}\".format(key.device, key))\n",
    "out_capsule = torch.utils.dlpack.to_dlpack(out)\n",
    "# Lookup the embedding vectors from dlpack\n",
    "hps.lookup_fromdlpack(key_capsule, out_capsule,\"hps_demo\", 0)\n",
    "out_put = torch.utils.dlpack.from_dlpack(out_capsule)\n",
    "print(\"[The device type of embedding vectors that lookup dlpack from hps interface for embedding table 0 of hps_demo: {}, the vectors: {}\\n\".format(out_put.device, out_put))\n",
    "diff = out_put-embedding1.reshape(1,cat_input1.flatten().shape[0]*16)\n",
    "if diff.mean() > 1e-4:\n",
    "    raise RuntimeError(\"Too large mse between pytorch dlpack on cpu and native HPS lookup api: {}\".format(diff.mean()))\n",
    "    sys.exit(1)\n",
    "else:\n",
    "    print(\"Pytorch dlpack on cpu  results are consistent with native HPS lookup api, mse: {}\".format(diff.mean()))\n",
    "    \n",
    "\n",
    "# 2. Look up from dlpack for Pytorch tensor on GPU\n",
    "print(\"************Look up from pytorch dlpack on GPU\")\n",
    "cuda_device = torch.device(\"cuda:0\" if torch.cuda.is_available else \"cpu\")\n",
    "key = torch.tensor(cat_input1.flatten(),dtype=torch.int64, device=device)\n",
    "key_capsule = torch.utils.dlpack.to_dlpack(key)\n",
    "out = torch.empty((cat_input1.flatten().shape[0]*16), dtype=torch.float32, device=cuda_device)\n",
    "out_capsule = torch.utils.dlpack.to_dlpack(out)\n",
    "hps.lookup_fromdlpack(key_capsule, out_capsule,\"hps_demo\", 0)\n",
    "out_put = torch.utils.dlpack.from_dlpack(out_capsule)\n",
    "print(\"The device type of embedding vectors that lookup dlpack from hps interface for embedding table 0 of hps_demo: {}, the vectors: {}\\n\\n\".format(out_put.device, out_put))\n",
    "diff = out_put.cpu()-embedding1.reshape(1,cat_input1.flatten().shape[0]*16)\n",
    "if diff.mean() > 1e-3:\n",
    "    raise RuntimeError(\"Too large mse between pytorch dlpack on cpu and native HPS lookup api: {}\".format(diff.mean()))\n",
    "    sys.exit(1)\n",
    "else:\n",
    "    print(\"Pytorch dlpack on GPU results are consistent with native HPS lookup api, mse: {}\".format(diff.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "45d8e30d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Look up from dlpack for Tensorflow tensor\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-09-20 06:34:21.729218: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***************Look up from tensorflow dlpack on CPU**********\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-09-20 06:34:44.168630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 30048 MB memory:  -> device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:06:00.0, compute capability: 7.0\n",
      "2023-09-20 06:34:44.170043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 30184 MB memory:  -> device: 1, name: Tesla V100-SXM2-32GB, pci bus id: 0000:07:00.0, compute capability: 7.0\n",
      "2023-09-20 06:34:44.171618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 30184 MB memory:  -> device: 2, name: Tesla V100-SXM2-32GB, pci bus id: 0000:0a:00.0, compute capability: 7.0\n",
      "2023-09-20 06:34:44.173095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 30184 MB memory:  -> device: 3, name: Tesla V100-SXM2-32GB, pci bus id: 0000:0b:00.0, compute capability: 7.0\n",
      "2023-09-20 06:34:44.174795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:4 with 30184 MB memory:  -> device: 4, name: Tesla V100-SXM2-32GB, pci bus id: 0000:85:00.0, compute capability: 7.0\n",
      "2023-09-20 06:34:44.176299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:5 with 30184 MB memory:  -> device: 5, name: Tesla V100-SXM2-32GB, pci bus id: 0000:86:00.0, compute capability: 7.0\n",
      "2023-09-20 06:34:44.177782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:6 with 30184 MB memory:  -> device: 6, name: Tesla V100-SXM2-32GB, pci bus id: 0000:89:00.0, compute capability: 7.0\n",
      "2023-09-20 06:34:44.179411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:7 with 30184 MB memory:  -> device: 7, name: Tesla V100-SXM2-32GB, pci bus id: 0000:8a:00.0, compute capability: 7.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The device type of embedding keys that lookup dlpack from hps interface for embedding table 1 of hps_demo: /job:localhost/replica:0/task:0/device:CPU:0, the keys: [20005 30047 20004 ... 30001 20037 30001]\n",
      "The device type of embedding vectors that lookup dlpack from hps interface for embedding table 1 of hps_demo: /job:localhost/replica:0/task:0/device:CPU:0, the vectors: [[ 0.02182689  0.01806355  0.01985828 ...  0.0136845  -0.01738386\n",
      "  -0.00323257]]\n",
      "\n",
      "tensorflow dlpack on CPU results are consistent with native HPS lookup api, mse: 0.0\n",
      "***************Look up from tensorflow dlpack on GPU**********\n",
      "[HUGECTR][INFO] The device type of embedding vectors that lookup dlpack from hps interface for embedding table 1 of wdl: /job:localhost/replica:0/task:0/device:GPU:0, the vectors: [ 0.02182689  0.01806355  0.01985828 ...  0.0136845  -0.01738386\n",
      " -0.00323257]\n",
      "\n",
      "tensorflow dlpack on GPU results are consistent with native HPS lookup api, mse: 0.0\n"
     ]
    }
   ],
   "source": [
    "# 3. Look up from dlpack for tensorflow tensor on CPU\n",
    "print(\"Look up from dlpack for Tensorflow tensor\")\n",
    "from tensorflow.python.dlpack import dlpack  \n",
    "import tensorflow as tf\n",
    "from tensorflow.python.eager import context\n",
    "from tensorflow.python.framework import dtypes\n",
    "print(\"***************Look up from tensorflow dlpack on CPU**********\")\n",
    "with tf.device('/CPU:0'):\n",
    "    key_tensor = tf.constant(cat_input2.flatten(),dtype=tf.int64)\n",
    "    out_tensor = tf.zeros([1, cat_input2.flatten().shape[0]*32],dtype=tf.float32)\n",
    "    print(\"The device type of embedding keys that lookup dlpack from hps interface for embedding table 1 of hps_demo: {}, the keys: {}\".format(key_tensor.device, key_tensor))\n",
    "    key_capsule = tf.experimental.dlpack.to_dlpack(key_tensor)\n",
    "    out_dlcapsule = tf.experimental.dlpack.to_dlpack(out_tensor)\n",
    "hps.lookup_fromdlpack(key_capsule,out_dlcapsule, \"hps_demo\", 1)\n",
    "out = tf.experimental.dlpack.from_dlpack(out_dlcapsule)\n",
    "print(\"The device type of embedding vectors that lookup dlpack from hps interface for embedding table 1 of hps_demo: {}, the vectors: {}\\n\".format(out.device, out))\n",
    "diff = out-embedding2.reshape(1,cat_input2.flatten().shape[0]*32)\n",
    "mse = tf.reduce_mean(diff)\n",
    "if mse> 1e-3:\n",
    "    raise RuntimeError(\"Too large mse between tensorflow dlpack on cpu and native HPS lookup api: {}\".format(mse))\n",
    "    sys.exit(1)\n",
    "else:\n",
    "    print(\"tensorflow dlpack on CPU results are consistent with native HPS lookup api, mse: {}\".format(mse))\n",
    "    \n",
    "# 4. Look up from dlpack for tensorflow tensor on GPU\n",
    "print(\"***************Look up from tensorflow dlpack on GPU**********\")\n",
    "with tf.device('/GPU:0'):\n",
    "    key_tensor = tf.constant(cat_input2.flatten(),dtype=tf.int64)\n",
    "    out_tensor = tf.zeros([cat_input2.flatten().shape[0]*32],dtype=tf.float32)\n",
    "    key_capsule = tf.experimental.dlpack.to_dlpack(key_tensor)\n",
    "    out_dlcapsule = tf.experimental.dlpack.to_dlpack(out_tensor)\n",
    "hps.lookup_fromdlpack(key_capsule,out_dlcapsule, \"hps_demo\", 1)\n",
    "out= tf.experimental.dlpack.from_dlpack(out_dlcapsule)\n",
    "print(\"[HUGECTR][INFO] The device type of embedding vectors that lookup dlpack from hps interface for embedding table 1 of wdl: {}, the vectors: {}\\n\".format(out.device, out))\n",
    "diff = out-embedding2.reshape(1,cat_input2.flatten().shape[0]*32)\n",
    "mse = tf.reduce_mean(diff)\n",
    "if mse> 1e-3:\n",
    "    raise RuntimeError(\"Too large mse between tensorflow dlpack on cpu and native HPS lookup api: {}\".format(mse))\n",
    "    sys.exit(1)\n",
    "else:\n",
    "    print(\"tensorflow dlpack on GPU results are consistent with native HPS lookup api, mse: {}\".format(mse))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7977023-0b0b-4be3-84f8-090acdbc01b3",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "<a id=\"section-3\"></a>\n",
    "## 3. Multi-process inference\n",
    "\n",
    "It is possible to share the a hashmap database between multiple processes. The following example launches 3 processes which achieve this using the operating system's shared memory, which is located at `/dev/shm` in most unix systems. In this example, we separate processes into a primary and multiple secondary processes, and only the primary process initializes the shared memory database. The secondary processes wait until the shared memory has been fully initialized. However, note that inter-process database access is guaranteed to be thread-safe. Therefore, it is also possible to implement more complicated initialization/refresh mechanisms for your use-case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f2750d05-1aeb-4c88-83ef-2f44e06a1957",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing multi_process_hps.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile multi_process_hps.py\n",
    "import os\n",
    "import time\n",
    "import multiprocessing as mp\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import onnxruntime as ort\n",
    "from hugectr import DatabaseType_t\n",
    "from hugectr.inference import HPS, ParameterServerConfig, InferenceParams, VolatileDatabaseParams\n",
    "\n",
    "slot_size_array = [10000, 10000, 10000, 10000]\n",
    "key_offset = np.insert(np.cumsum(slot_size_array), 0, 0)[:-1]\n",
    "batch_size = 1024\n",
    "\n",
    "def create_hps(name, initialized, device_id, num_max_processes):\n",
    "    print(f'subprocess：{name}（{os.getpid()}）launch...')\n",
    "    \n",
    "    # 1. Let secondary processes wait until shared memory is initialized.\n",
    "    while name != 'primary' and initialized.value == 0:\n",
    "        print(f'Subprocess {name} awaiting SHM initialization...')\n",
    "        time.sleep(1)\n",
    "\n",
    "    # 2. Configure the HPS hyperparameters\n",
    "    ps_config = ParameterServerConfig(\n",
    "           emb_table_name = {\"hps_demo\": [\"sparse_embedding1\", \"sparse_embedding2\"]},\n",
    "           embedding_vec_size = {\"hps_demo\": [16, 32]},\n",
    "           max_feature_num_per_sample_per_emb_table = {\"hps_demo\": [2, 2]},\n",
    "           inference_params_array = [\n",
    "              InferenceParams(\n",
    "                model_name = \"hps_demo\",\n",
    "                max_batchsize = batch_size,\n",
    "                hit_rate_threshold = 1.0,\n",
    "                dense_model_file = \"\",\n",
    "                sparse_model_files = [\"hps_demo0_sparse_1000.model\", \"hps_demo1_sparse_1000.model\"],\n",
    "                device_id=device_id,\n",
    "                deployed_devices = [device_id],\n",
    "                use_gpu_embedding_cache = True,\n",
    "                cache_size_percentage = 0.5,\n",
    "                i64_input_key = True)\n",
    "           ],\n",
    "           volatile_db = VolatileDatabaseParams(\n",
    "                DatabaseType_t.multi_process_hash_map,  # Use /dev/shm instead of normal memory for storage.\n",
    "                # Skips initializing model. If we run HPS in multiple processes, only one needs to initialize.\n",
    "                initialize_after_startup = name == 'primary',\n",
    "           ))\n",
    "\n",
    "    # 3. Initialize the HPS object\n",
    "    hps = HPS(ps_config)\n",
    "    initialized.value += 1\n",
    "    print(f'Subprocess {name} initialized')\n",
    "    \n",
    "    # 4. In (1) the secondary processes wait until the primary process has completed initializing\n",
    "    #    the shared memory. If the last process disconnects, the shared memory is erased.\n",
    "    #    Therefore, if threads that currently have attached to the shared memory manage to complete\n",
    "    #    their program before another process has attached, the contents of the shared memory are\n",
    "    #    lost and the new process will instead construct an empty shared memory. To avoid this\n",
    "    #    situation, we have multiple options.\n",
    "    #\n",
    "    #   a) Setting `shared_memory_auto_remove = False` in the `VolatileDatabaseParams`\n",
    "    #      configuration [default: True]. This will prevent the deletion of the shared memory when\n",
    "    #      the last process disconnects. In other words, revoking this flag allows you to preserve\n",
    "    #      and use the state of a shared memory across multiple program restarts. However, while\n",
    "    #      desirable in some situations, this is not the behavior we need here, because this\n",
    "    #      notebook cell should be allowed to be executed repeatedly without relying on risidual\n",
    "    #      state.\n",
    "    #\n",
    "    #   b) Another approach is to ensure that the all other processes that should attach have\n",
    "    #      attached. Here we achieve this by simply monitoring the `initialized` cross process\n",
    "    #      counter variable that we used in (1). Once it hits `num_max_processes` we can be sure\n",
    "    #      that each subprocess has properly connected.\n",
    "    while initialized.value != num_max_processes:\n",
    "        print(f'Subprocess {name} await other processes...')\n",
    "        time.sleep(1)\n",
    "    \n",
    "    # 5. Load query data.\n",
    "    df = pd.read_parquet(\"data_parquet/val/gen_0.parquet\")\n",
    "    dense_input_columns = df.columns[1:11]\n",
    "    cat_input1_columns = df.columns[11:13]\n",
    "    cat_input2_columns = df.columns[13:15]\n",
    "    dense_input = df[dense_input_columns].loc[0:batch_size-1].to_numpy(dtype=np.float32)\n",
    "    cat_input1 = (df[cat_input1_columns].loc[0:batch_size-1].to_numpy(dtype=np.int64) + key_offset[0:2]).reshape((batch_size, 2, 1))\n",
    "    cat_input2 = (df[cat_input2_columns].loc[0:batch_size-1].to_numpy(dtype=np.int64) + key_offset[2:4]).reshape((batch_size, 2, 1))\n",
    "\n",
    "    # 6. Make inference from the HPS object and the ONNX inference session of `hps_demo_without_embedding.onnx`.\n",
    "    embedding1 = hps.lookup(cat_input1.flatten(), \"hps_demo\", 0,device_id).reshape(batch_size, 2, 16)\n",
    "    embedding2 = hps.lookup(cat_input2.flatten(), \"hps_demo\", 1,device_id).reshape(batch_size, 2, 32)\n",
    "    sess = ort.InferenceSession(\"hps_demo_without_embedding.onnx\")\n",
    "    res = sess.run(output_names=[sess.get_outputs()[0].name],\n",
    "                   input_feed={sess.get_inputs()[0].name: dense_input,\n",
    "                   sess.get_inputs()[1].name: embedding1,\n",
    "                   sess.get_inputs()[2].name: embedding2})\n",
    "    pred = res[0]\n",
    "\n",
    "    # 7. Check the correctness by comparing with dumped evaluation results.\n",
    "    ground_truth = np.load(\"ground_truth.npy\").flatten()\n",
    "    print(f'Subprocess {name}; ground_truth: {ground_truth}')\n",
    "    diff = pred.flatten()-ground_truth\n",
    "    mse = np.mean(diff*diff)\n",
    "    print(f'Subprocess {name}; pred: {pred}')\n",
    "    print(f'Subprocess {name}; mse between pred and ground_truth: {mse}')\n",
    "\n",
    "    # 8. Make inference with the ONNX inference session of `hps_demo_with_embedding.onnx` (double check).\n",
    "    sess_ref = ort.InferenceSession(\"hps_demo_with_embedding.onnx\")\n",
    "    res_ref = sess_ref.run(output_names=[sess_ref.get_outputs()[0].name],\n",
    "                   input_feed={sess_ref.get_inputs()[0].name: dense_input,\n",
    "                   sess_ref.get_inputs()[1].name: cat_input1,\n",
    "                   sess_ref.get_inputs()[2].name: cat_input2})\n",
    "    pred_ref = res_ref[0]\n",
    "    diff_ref = pred_ref.flatten()-ground_truth\n",
    "    mse_ref = np.mean(diff_ref*diff_ref)\n",
    "    print(f'Subprocess {name}; pred_ref: {pred_ref}')\n",
    "    print(f'Subprocess {name}; mse between pred_ref and ground_truth: {mse_ref}')\n",
    "\n",
    "    print(f'Subprocess {name} exiting...')\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    # Destroy shared memory.\n",
    "    try:\n",
    "        os.remove('/dev/shm/hctr_mp_hash_map_database')\n",
    "    except:\n",
    "        pass\n",
    "    \n",
    "    initialized = mp.Value('i', 0)\n",
    "\n",
    "    # Create sub processes.\n",
    "    processes = [\n",
    "        mp.Process(target=create_hps, args=('primary', initialized, 0, 3)),\n",
    "        mp.Process(target=create_hps, args=('secondary', initialized, 1, 3)),\n",
    "        mp.Process(target=create_hps, args=('secondary', initialized, 2, 3)),\n",
    "    ]\n",
    "    for p in processes:\n",
    "        p.start()\n",
    "\n",
    "    # Go to sleep until subprocesses are initialized.\n",
    "    while initialized.value < len(processes):\n",
    "        print(f'Main process; awaiting subprocess initialization... So far {initialized.value} initialized...')\n",
    "        time.sleep(1)\n",
    "        \n",
    "    # Wait for subprocesses to exit.\n",
    "    for i, p in enumerate(processes):\n",
    "        print(f'Main process; awaiting subprocess {i} to exit...')\n",
    "        p.join()\n",
    "    print(f'Main process; exiting...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ba71b2c5-187a-4429-bd81-f4addb2b1c76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "subprocess：primary（1394）launch...\n",
      "[HCTR][06:48:37.272][WARNING][RK0][main]: default_value_for_each_table.size() is not equal to the number of embedding tables\n",
      "====================================================HPS Create====================================================\n",
      "[HCTR][06:48:37.272][INFO][RK0][main]: Creating Multi-Process HashMap CPU database backend...\n",
      "[HCTR][06:48:37.272][INFO][RK0][main]: Connecting to shared memory 'hctr_mp_hash_map_database'...\n",
      "subprocess：secondary（1396）launch...\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "Main process; awaiting subprocess initialization... So far 0 initialized...\n",
      "subprocess：secondary（1397）launch...\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "[HCTR][06:48:37.772][INFO][RK0][main]: Connected to shared memory 'hctr_mp_hash_map_database'; OS total = 270453215232 bytes, OS available = 269706559488 bytes, HCTR allocated = 17179869184 bytes, HCTR free = 17179868672 bytes; other processes connected = 0\n",
      "[HCTR][06:48:37.773][INFO][RK0][main]: Volatile DB: initial cache rate = 1\n",
      "[HCTR][06:48:37.773][INFO][RK0][main]: Volatile DB: cache missed embeddings = 0\n",
      "[HCTR][06:48:37.773][DEBUG][RK0][main]: Created raw model loader in local memory!\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "Main process; awaiting subprocess initialization... So far 0 initialized...\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "[HCTR][06:48:38.313][INFO][RK0][main]: Table: hps_et.hps_demo.sparse_embedding1; cached 18488 / 18488 embeddings in volatile database (MultiProcessHashMapBackend); load: 18488 / 18446744073709551615 (0.00%).\n",
      "[HCTR][06:48:38.947][INFO][RK0][main]: Table: hps_et.hps_demo.sparse_embedding2; cached 18470 / 18470 embeddings in volatile database (MultiProcessHashMapBackend); load: 18470 / 18446744073709551615 (0.00%).\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "Main process; awaiting subprocess initialization... So far 0 initialized...\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "Main process; awaiting subprocess initialization... So far 0 initialized...\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "Main process; awaiting subprocess initialization... So far 0 initialized...\n",
      "Subprocess secondary awaiting SHM initialization...\n",
      "[HCTR][06:48:41.289][DEBUG][RK0][main]: Real-time subscribers created!\n",
      "[HCTR][06:48:41.289][INFO][RK0][main]: Creating embedding cache in device 0.\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: Model name: hps_demo\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: Max batch size: 1024\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: Fuse embedding tables: False\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: Number of embedding tables: 2\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: Use GPU embedding cache: True, cache size percentage: 0.500000\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: Embedding cache type: dynamic\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: Use I64 input key: True\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: Configured cache hit rate threshold: 1.000000\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: The size of thread pool: 80\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: The size of worker memory pool: 2\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: The size of refresh memory pool: 1\n",
      "[HCTR][06:48:41.295][INFO][RK0][main]: The refresh percentage : 0.000000\n",
      "[HCTR][06:48:41.311][INFO][RK0][main]: LookupSession i64_input_key: True\n",
      "[HCTR][06:48:41.311][INFO][RK0][main]: Creating lookup session for hps_demo on device: 0\n",
      "Subprocess primary initialized\n",
      "Subprocess primary await other processes...\n",
      "Main process; awaiting subprocess initialization... So far 1 initialized...\n",
      "[HCTR][06:48:42.279][WARNING][RK0][main]: default_value_for_each_table.size() is not equal to the number of embedding tables\n",
      "====================================================HPS Create====================================================\n",
      "[HCTR][06:48:42.280][INFO][RK0][main]: Creating Multi-Process HashMap CPU database backend...\n",
      "[HCTR][06:48:42.281][INFO][RK0][main]: Connecting to shared memory 'hctr_mp_hash_map_database'...\n",
      "[HCTR][06:48:42.281][WARNING][RK0][main]: default_value_for_each_table.size() is not equal to the number of embedding tables\n",
      "====================================================HPS Create====================================================\n",
      "[HCTR][06:48:42.282][INFO][RK0][main]: Creating Multi-Process HashMap CPU database backend...\n",
      "[HCTR][06:48:42.282][INFO][RK0][main]: Connecting to shared memory 'hctr_mp_hash_map_database'...\n",
      "Subprocess primary await other processes...\n",
      "[HCTR][06:48:42.781][INFO][RK0][main]: Connected to shared memory 'hctr_mp_hash_map_database'; OS total = 270453215232 bytes, OS available = 260310085632 bytes, HCTR allocated = 17179869184 bytes, HCTR free = 7783505728 bytes; other processes connected = 1\n",
      "[HCTR][06:48:42.781][INFO][RK0][main]: Volatile DB: initial cache rate = 1\n",
      "[HCTR][06:48:42.781][INFO][RK0][main]: Volatile DB: cache missed embeddings = 0\n",
      "[HCTR][06:48:42.781][DEBUG][RK0][main]: Created raw model loader in local memory!\n",
      "Main process; awaiting subprocess initialization... So far 1 initialized...\n",
      "[HCTR][06:48:43.281][INFO][RK0][main]: Connected to shared memory 'hctr_mp_hash_map_database'; OS total = 270453215232 bytes, OS available = 260310085632 bytes, HCTR allocated = 17179869184 bytes, HCTR free = 7783505728 bytes; other processes connected = 1\n",
      "[HCTR][06:48:43.281][INFO][RK0][main]: Volatile DB: initial cache rate = 1\n",
      "[HCTR][06:48:43.281][INFO][RK0][main]: Volatile DB: cache missed embeddings = 0\n",
      "[HCTR][06:48:43.281][DEBUG][RK0][main]: Created raw model loader in local memory!\n",
      "Subprocess primary await other processes...\n",
      "Main process; awaiting subprocess initialization... So far 1 initialized...\n",
      "Subprocess primary await other processes...\n",
      "Main process; awaiting subprocess initialization... So far 1 initialized...\n",
      "Subprocess primary await other processes...\n",
      "[HCTR][06:48:45.440][DEBUG][RK0][main]: Real-time subscribers created!\n",
      "[HCTR][06:48:45.441][INFO][RK0][main]: Creating embedding cache in device 1.\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: Model name: hps_demo\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: Max batch size: 1024\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: Fuse embedding tables: False\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: Number of embedding tables: 2\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: Use GPU embedding cache: True, cache size percentage: 0.500000\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: Embedding cache type: dynamic\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: Use I64 input key: True\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: Configured cache hit rate threshold: 1.000000\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: The size of thread pool: 80\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: The size of worker memory pool: 2\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: The size of refresh memory pool: 1\n",
      "[HCTR][06:48:45.463][INFO][RK0][main]: The refresh percentage : 0.000000\n",
      "[HCTR][06:48:45.706][DEBUG][RK0][main]: Real-time subscribers created!\n",
      "[HCTR][06:48:45.706][INFO][RK0][main]: Creating embedding cache in device 2.\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: Model name: hps_demo\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: Max batch size: 1024\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: Fuse embedding tables: False\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: Number of embedding tables: 2\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: Use GPU embedding cache: True, cache size percentage: 0.500000\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: Embedding cache type: dynamic\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: Use I64 input key: True\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: Configured cache hit rate threshold: 1.000000\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: The size of thread pool: 80\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: The size of worker memory pool: 2\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: The size of refresh memory pool: 1\n",
      "[HCTR][06:48:45.711][INFO][RK0][main]: The refresh percentage : 0.000000\n",
      "Main process; awaiting subprocess initialization... So far 1 initialized...\n",
      "Subprocess primary await other processes...\n",
      "[HCTR][06:48:46.699][INFO][RK0][main]: LookupSession i64_input_key: True\n",
      "[HCTR][06:48:46.699][INFO][RK0][main]: Creating lookup session for hps_demo on device: 1\n",
      "Subprocess secondary initialized\n",
      "Subprocess secondary await other processes...\n",
      "[HCTR][06:48:46.764][INFO][RK0][main]: LookupSession i64_input_key: True\n",
      "[HCTR][06:48:46.764][INFO][RK0][main]: Creating lookup session for hps_demo on device: 2\n",
      "Subprocess secondary initialized\n",
      "2023-09-20 06:48:46.842594773 [W:onnxruntime:, graph.cc:3543 CleanUnusedInitializersAndNodeArgs] Removing initializer 'key_to_indice_hash_all_tables'. It is not used by any node and should be removed from the model.\n",
      "Subprocess secondary; ground_truth: [0.4895492  0.509022   0.38192913 ... 0.5264926  0.50650454 0.47927693]\n",
      "Subprocess secondary; pred: [[0.48954916]\n",
      " [0.50902206]\n",
      " [0.38192907]\n",
      " ...\n",
      " [0.52649266]\n",
      " [0.5065045 ]\n",
      " [0.4792769 ]]\n",
      "Subprocess secondary; mse between pred and ground_truth: 2.3887142264200634e-15\n",
      "Subprocess secondary; pred_ref: [[0.48954916]\n",
      " [0.50902206]\n",
      " [0.38192907]\n",
      " ...\n",
      " [0.52649266]\n",
      " [0.5065045 ]\n",
      " [0.4792769 ]]\n",
      "Subprocess secondary; mse between pred_ref and ground_truth: 2.3887142264200634e-15\n",
      "Subprocess secondary exiting...\n",
      "[HCTR][06:48:46.900][INFO][RK0][main]: Disconnecting from shared memory 'hctr_mp_hash_map_database'.\n",
      "Main process; awaiting subprocess 0 to exit...\n",
      "2023-09-20 06:48:47.497305659 [W:onnxruntime:, graph.cc:3543 CleanUnusedInitializersAndNodeArgs] Removing initializer 'key_to_indice_hash_all_tables'. It is not used by any node and should be removed from the model.\n",
      "Subprocess primary; ground_truth: [0.4895492  0.509022   0.38192913 ... 0.5264926  0.50650454 0.47927693]\n",
      "Subprocess primary; pred: [[0.48954916]\n",
      " [0.50902206]\n",
      " [0.38192907]\n",
      " ...\n",
      " [0.52649266]\n",
      " [0.5065045 ]\n",
      " [0.4792769 ]]\n",
      "Subprocess primary; mse between pred and ground_truth: 2.3887142264200634e-15\n",
      "Subprocess primary; pred_ref: [[0.48954916]\n",
      " [0.50902206]\n",
      " [0.38192907]\n",
      " ...\n",
      " [0.52649266]\n",
      " [0.5065045 ]\n",
      " [0.4792769 ]]\n",
      "Subprocess primary; mse between pred_ref and ground_truth: 2.3887142264200634e-15\n",
      "Subprocess primary exiting...\n",
      "[HCTR][06:48:47.568][INFO][RK0][main]: Disconnecting from shared memory 'hctr_mp_hash_map_database'.\n",
      "2023-09-20 06:48:48.101124718 [W:onnxruntime:, graph.cc:3543 CleanUnusedInitializersAndNodeArgs] Removing initializer 'key_to_indice_hash_all_tables'. It is not used by any node and should be removed from the model.\n",
      "Subprocess secondary; ground_truth: [0.4895492  0.509022   0.38192913 ... 0.5264926  0.50650454 0.47927693]\n",
      "Subprocess secondary; pred: [[0.48954916]\n",
      " [0.50902206]\n",
      " [0.38192907]\n",
      " ...\n",
      " [0.52649266]\n",
      " [0.5065045 ]\n",
      " [0.4792769 ]]\n",
      "Subprocess secondary; mse between pred and ground_truth: 2.3887142264200634e-15\n",
      "Subprocess secondary; pred_ref: [[0.48954916]\n",
      " [0.50902206]\n",
      " [0.38192907]\n",
      " ...\n",
      " [0.52649266]\n",
      " [0.5065045 ]\n",
      " [0.4792769 ]]\n",
      "Subprocess secondary; mse between pred_ref and ground_truth: 2.3887142264200634e-15\n",
      "Subprocess secondary exiting...\n",
      "[HCTR][06:48:48.176][INFO][RK0][main]: Disconnecting from shared memory 'hctr_mp_hash_map_database'.\n",
      "Main process; awaiting subprocess 1 to exit...\n",
      "[HCTR][06:48:48.687][INFO][RK0][main]: Detached last process from shared memory 'hctr_mp_hash_map_database'. Auto remove in progress...\n",
      "Main process; awaiting subprocess 2 to exit...\n",
      "Main process; exiting...\n"
     ]
    }
   ],
   "source": [
    "!python3 multi_process_hps.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f01d2b-af38-4e1e-8adc-9e041ff09f6a",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "<a id=\"section-4\"></a>\n",
    "## 4. Redis Cluster deployment (without TLS/SSL)\n",
    "HugeCTR can use Redis clusters as backing storage. In the following steps we show how to setup a mock Redis / HugeCTR deployment in a single machine. We assume that you have started this notebook in a HugeCTR docker container."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e5cf419-ca12-4bc8-aef2-de116666e9b4",
   "metadata": {},
   "source": [
    "**Step 1: Get + build Redis**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cf926d3d-e718-4a1a-955b-4d673bf26b11",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-09-20 06:49:01--  https://github.com/redis/redis/archive/7.0.8.tar.gz\n",
      "Resolving github.com (github.com)... 192.30.255.112\n",
      "Connecting to github.com (github.com)|192.30.255.112|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://codeload.github.com/redis/redis/tar.gz/refs/tags/7.0.8 [following]\n",
      "--2023-09-20 06:49:01--  https://codeload.github.com/redis/redis/tar.gz/refs/tags/7.0.8\n",
      "Resolving codeload.github.com (codeload.github.com)... 192.30.255.120\n",
      "Connecting to codeload.github.com (codeload.github.com)|192.30.255.120|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: unspecified [application/x-gzip]\n",
      "Saving to: ‘7.0.8.tar.gz’\n",
      "\n",
      "7.0.8.tar.gz            [   <=>              ]   2.87M  5.50MB/s    in 0.5s    \n",
      "\n",
      "2023-09-20 06:49:02 (5.50 MB/s) - ‘7.0.8.tar.gz’ saved [3011655]\n",
      "\n",
      "cd src && make all\n",
      "make[1]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/src'\n",
      "./mkreleasehdr.sh: line 2: echo: write error: Broken pipe\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mMakefile.dep\u001b[0m\n",
      "./mkreleasehdr.sh: line 2: echo: write error: Broken pipe\n",
      "rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-rdb redis-check-aof *.o *.gcda *.gcno *.gcov redis.info lcov-html Makefile.dep\n",
      "rm -f adlist.d quicklist.d ae.d anet.d dict.d server.d sds.d zmalloc.d lzf_c.d lzf_d.d pqsort.d zipmap.d sha1.d ziplist.d release.d networking.d util.d object.d db.d replication.d rdb.d t_string.d t_list.d t_set.d t_zset.d t_hash.d config.d aof.d pubsub.d multi.d debug.d sort.d intset.d syncio.d cluster.d crc16.d endianconv.d slowlog.d eval.d bio.d rio.d rand.d memtest.d syscheck.d crcspeed.d crc64.d bitops.d sentinel.d notify.d setproctitle.d blocked.d hyperloglog.d latency.d sparkline.d redis-check-rdb.d redis-check-aof.d geo.d lazyfree.d module.d evict.d expire.d geohash.d geohash_helper.d childinfo.d defrag.d siphash.d rax.d t_stream.d listpack.d localtime.d lolwut.d lolwut5.d lolwut6.d acl.d tracking.d connection.d tls.d sha256.d timeout.d setcpuaffinity.d monotonic.d mt19937-64.d resp_parser.d call_reply.d script_lua.d script.d functions.d function_lua.d commands.d anet.d adlist.d dict.d redis-cli.d zmalloc.d release.d ae.d redisassert.d crcspeed.d crc64.d siphash.d crc16.d monotonic.d cli_common.d mt19937-64.d ae.d anet.d redis-benchmark.d adlist.d dict.d zmalloc.d redisassert.d release.d crcspeed.d crc64.d siphash.d crc16.d monotonic.d cli_common.d mt19937-64.d\n",
      "(cd ../deps && make distclean)\n",
      "make[2]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps'\n",
      "(cd hiredis && make clean) > /dev/null || true\n",
      "(cd linenoise && make clean) > /dev/null || true\n",
      "(cd lua && make clean) > /dev/null || true\n",
      "(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true\n",
      "(cd hdr_histogram && make clean) > /dev/null || true\n",
      "(rm -f .make-*)\n",
      "make[2]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps'\n",
      "(cd modules && make clean)\n",
      "make[2]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/src/modules'\n",
      "rm -rf *.xo *.so\n",
      "make[2]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/src/modules'\n",
      "(cd ../tests/modules && make clean)\n",
      "make[2]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/tests/modules'\n",
      "rm -f commandfilter.so basics.so testrdb.so fork.so infotest.so propagate.so misc.so hooks.so blockonkeys.so blockonbackground.so scan.so datatype.so datatype2.so auth.so keyspace_events.so blockedclient.so getkeys.so getchannels.so test_lazyfree.so timer.so defragtest.so keyspecs.so hash.so zset.so stream.so mallocsize.so aclcheck.so list.so subcommands.so reply.so cmdintrospection.so eventloop.so moduleconfigs.so moduleconfigstwo.so publish.so usercall.so commandfilter.xo basics.xo testrdb.xo fork.xo infotest.xo propagate.xo misc.xo hooks.xo blockonkeys.xo blockonbackground.xo scan.xo datatype.xo datatype2.xo auth.xo keyspace_events.xo blockedclient.xo getkeys.xo getchannels.xo test_lazyfree.xo timer.xo defragtest.xo keyspecs.xo hash.xo zset.xo stream.xo mallocsize.xo aclcheck.xo list.xo subcommands.xo reply.xo cmdintrospection.xo eventloop.xo moduleconfigs.xo moduleconfigstwo.xo publish.xo usercall.xo\n",
      "make[2]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/tests/modules'\n",
      "(rm -f .make-*)\n",
      "echo STD=-pedantic -DREDIS_STATIC='' -std=c11 >> .make-settings\n",
      "echo WARN=-Wall -W -Wno-missing-field-initializers >> .make-settings\n",
      "echo OPT=-O2 >> .make-settings\n",
      "echo MALLOC=jemalloc >> .make-settings\n",
      "echo BUILD_TLS= >> .make-settings\n",
      "echo USE_SYSTEMD= >> .make-settings\n",
      "echo CFLAGS= >> .make-settings\n",
      "echo LDFLAGS= >> .make-settings\n",
      "echo REDIS_CFLAGS= >> .make-settings\n",
      "echo REDIS_LDFLAGS= >> .make-settings\n",
      "echo PREV_FINAL_CFLAGS=-pedantic -DREDIS_STATIC='' -std=c11 -Wall -W -Wno-missing-field-initializers -O2 -g -ggdb   -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -I../deps/hdr_histogram -DUSE_JEMALLOC -I../deps/jemalloc/include >> .make-settings\n",
      "echo PREV_FINAL_LDFLAGS=  -g -ggdb -rdynamic >> .make-settings\n",
      "(cd ../deps && make hiredis linenoise lua hdr_histogram jemalloc)\n",
      "make[2]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps'\n",
      "(cd hiredis && make clean) > /dev/null || true\n",
      "(cd linenoise && make clean) > /dev/null || true\n",
      "(cd lua && make clean) > /dev/null || true\n",
      "(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true\n",
      "(cd hdr_histogram && make clean) > /dev/null || true\n",
      "(rm -f .make-*)\n",
      "(echo \"\" > .make-cflags)\n",
      "(echo \"\" > .make-ldflags)\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mhiredis\u001b[0m\n",
      "cd hiredis && make static \n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/hiredis'\n",
      "cc -std=c99 -c -O3 -fPIC   -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic alloc.c\n",
      "cc -std=c99 -c -O3 -fPIC   -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic net.c\n",
      "cc -std=c99 -c -O3 -fPIC   -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic hiredis.c\n",
      "cc -std=c99 -c -O3 -fPIC   -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic sds.c\n",
      "cc -std=c99 -c -O3 -fPIC   -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic async.c\n",
      "cc -std=c99 -c -O3 -fPIC   -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic read.c\n",
      "cc -std=c99 -c -O3 -fPIC   -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic sockcompat.c\n",
      "ar rcs libhiredis.a alloc.o net.o hiredis.o sds.o async.o read.o sockcompat.o\n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/hiredis'\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mlinenoise\u001b[0m\n",
      "cd linenoise && make\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/linenoise'\n",
      "cc  -Wall -Os -g  -c linenoise.c\n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/linenoise'\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mlua\u001b[0m\n",
      "cd lua/src && make all CFLAGS=\"-Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2 \" MYLDFLAGS=\"\" AR=\"ar rc\"\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/lua/src'\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lapi.o lapi.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lcode.o lcode.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ldebug.o ldebug.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ldo.o ldo.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ldump.o ldump.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lfunc.o lfunc.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lgc.o lgc.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o llex.o llex.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lmem.o lmem.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lobject.o lobject.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lopcodes.o lopcodes.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lparser.o lparser.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lstate.o lstate.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lstring.o lstring.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ltable.o ltable.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ltm.o ltm.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lundump.o lundump.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lvm.o lvm.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lzio.o lzio.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o strbuf.o strbuf.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o fpconv.o fpconv.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lauxlib.o lauxlib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lbaselib.o lbaselib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ldblib.o ldblib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o liolib.o liolib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lmathlib.o lmathlib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o loslib.o loslib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ltablib.o ltablib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lstrlib.o lstrlib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o loadlib.o loadlib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o linit.o linit.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua_cjson.o lua_cjson.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua_struct.o lua_struct.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua_cmsgpack.o lua_cmsgpack.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua_bit.o lua_bit.c\n",
      "ar rc liblua.a lapi.o lcode.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o lundump.o lvm.o lzio.o strbuf.o fpconv.o lauxlib.o lbaselib.o ldblib.o liolib.o lmathlib.o loslib.o ltablib.o lstrlib.o loadlib.o linit.o lua_cjson.o lua_struct.o lua_cmsgpack.o lua_bit.o\t# DLL needs all object files\n",
      "ranlib liblua.a\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua.o lua.c\n",
      "cc -o lua  lua.o liblua.a -lm \n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o luac.o luac.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o print.o print.c\n",
      "cc -o luac  luac.o print.o liblua.a -lm \n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/lua/src'\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mhdr_histogram\u001b[0m\n",
      "cd hdr_histogram && make\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/hdr_histogram'\n",
      "cc -std=c99 -Wall -Os -g  -DHDR_MALLOC_INCLUDE=\\\"hdr_redis_malloc.h\\\" -c  hdr_histogram.c \n",
      "ar rcs libhdrhistogram.a hdr_histogram.o\n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/hdr_histogram'\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mjemalloc\u001b[0m\n",
      "cd jemalloc && ./configure --with-version=5.2.1-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ CFLAGS=\"-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \" LDFLAGS=\"\" \n",
      "checking for xsltproc... false\n",
      "checking for gcc... gcc\n",
      "checking whether the C compiler works... yes\n",
      "checking for C compiler default output file name... a.out\n",
      "checking for suffix of executables... \n",
      "checking whether we are cross compiling... no\n",
      "checking for suffix of object files... o\n",
      "checking whether we are using the GNU C compiler... yes\n",
      "checking whether gcc accepts -g... yes\n",
      "checking for gcc option to accept ISO C89... none needed\n",
      "checking whether compiler is cray... no\n",
      "checking whether compiler supports -std=gnu11... yes\n",
      "checking whether compiler supports -Wall... yes\n",
      "checking whether compiler supports -Wextra... yes\n",
      "checking whether compiler supports -Wshorten-64-to-32... no\n",
      "checking whether compiler supports -Wsign-compare... yes\n",
      "checking whether compiler supports -Wundef... yes\n",
      "checking whether compiler supports -Wno-format-zero-length... yes\n",
      "checking whether compiler supports -pipe... yes\n",
      "checking whether compiler supports -g3... yes\n",
      "checking how to run the C preprocessor... gcc -E\n",
      "checking for g++... g++\n",
      "checking whether we are using the GNU C++ compiler... yes\n",
      "checking whether g++ accepts -g... yes\n",
      "checking whether g++ supports C++14 features by default... yes\n",
      "checking whether compiler supports -Wall... yes\n",
      "checking whether compiler supports -Wextra... yes\n",
      "checking whether compiler supports -g3... yes\n",
      "checking whether libstdc++ linkage is compilable... yes\n",
      "checking for grep that handles long lines and -e... /usr/bin/grep\n",
      "checking for egrep... /usr/bin/grep -E\n",
      "checking for ANSI C header files... yes\n",
      "checking for sys/types.h... yes\n",
      "checking for sys/stat.h... yes\n",
      "checking for stdlib.h... yes\n",
      "checking for string.h... yes\n",
      "checking for memory.h... yes\n",
      "checking for strings.h... yes\n",
      "checking for inttypes.h... yes\n",
      "checking for stdint.h... yes\n",
      "checking for unistd.h... yes\n",
      "checking whether byte ordering is bigendian... no\n",
      "checking size of void *... 8\n",
      "checking size of int... 4\n",
      "checking size of long... 8\n",
      "checking size of long long... 8\n",
      "checking size of intmax_t... 8\n",
      "checking build system type... x86_64-pc-linux-gnu\n",
      "checking host system type... x86_64-pc-linux-gnu\n",
      "checking whether pause instruction is compilable... yes\n",
      "checking number of significant virtual address bits... 48\n",
      "checking for ar... ar\n",
      "checking for nm... nm\n",
      "checking for gawk... no\n",
      "checking for mawk... mawk\n",
      "checking malloc.h usability... yes\n",
      "checking malloc.h presence... yes\n",
      "checking for malloc.h... yes\n",
      "checking whether malloc_usable_size definition can use const argument... no\n",
      "checking for library containing log... -lm\n",
      "checking whether __attribute__ syntax is compilable... yes\n",
      "checking whether compiler supports -fvisibility=hidden... yes\n",
      "checking whether compiler supports -fvisibility=hidden... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether tls_model attribute is compilable... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether alloc_size attribute is compilable... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether format(gnu_printf, ...) attribute is compilable... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether format(printf, ...) attribute is compilable... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether format(printf, ...) attribute is compilable... yes\n",
      "checking for a BSD-compatible install... /usr/bin/install -c\n",
      "checking for ranlib... ranlib\n",
      "checking for ld... /usr/bin/ld\n",
      "checking for autoconf... /usr/bin/autoconf\n",
      "checking for memalign... yes\n",
      "checking for valloc... yes\n",
      "checking whether compiler supports -O3... yes\n",
      "checking whether compiler supports -O3... yes\n",
      "checking whether compiler supports -funroll-loops... yes\n",
      "checking configured backtracing method... N/A\n",
      "checking for sbrk... yes\n",
      "checking whether utrace(2) is compilable... no\n",
      "checking whether a program using __builtin_unreachable is compilable... yes\n",
      "checking whether a program using __builtin_ffsl is compilable... yes\n",
      "checking whether a program using __builtin_popcountl is compilable... yes\n",
      "checking LG_PAGE... 12\n",
      "checking pthread.h usability... yes\n",
      "checking pthread.h presence... yes\n",
      "checking for pthread.h... yes\n",
      "checking for pthread_create in -lpthread... yes\n",
      "checking dlfcn.h usability... yes\n",
      "checking dlfcn.h presence... yes\n",
      "checking for dlfcn.h... yes\n",
      "checking for dlsym... yes\n",
      "checking whether pthread_atfork(3) is compilable... yes\n",
      "checking whether pthread_setname_np(3) is compilable... yes\n",
      "checking for library containing clock_gettime... none required\n",
      "checking whether clock_gettime(CLOCK_MONOTONIC_COARSE, ...) is compilable... yes\n",
      "checking whether clock_gettime(CLOCK_MONOTONIC, ...) is compilable... yes\n",
      "checking whether mach_absolute_time() is compilable... no\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether syscall(2) is compilable... yes\n",
      "checking for secure_getenv... yes\n",
      "checking for sched_getcpu... yes\n",
      "checking for sched_setaffinity... yes\n",
      "checking for issetugid... no\n",
      "checking for _malloc_thread_cleanup... no\n",
      "checking for _pthread_mutex_init_calloc_cb... no\n",
      "checking for TLS... yes\n",
      "checking whether C11 atomics is compilable... no\n",
      "checking whether GCC __atomic atomics is compilable... yes\n",
      "checking whether GCC 8-bit __atomic atomics is compilable... yes\n",
      "checking whether GCC __sync atomics is compilable... yes\n",
      "checking whether GCC 8-bit __sync atomics is compilable... yes\n",
      "checking whether Darwin OSAtomic*() is compilable... no\n",
      "checking whether madvise(2) is compilable... yes\n",
      "checking whether madvise(..., MADV_FREE) is compilable... yes\n",
      "checking whether madvise(..., MADV_DONTNEED) is compilable... yes\n",
      "checking whether madvise(..., MADV_DO[NT]DUMP) is compilable... yes\n",
      "checking whether madvise(..., MADV_[NO]HUGEPAGE) is compilable... yes\n",
      "checking for __builtin_clz... yes\n",
      "checking whether Darwin os_unfair_lock_*() is compilable... no\n",
      "checking whether glibc malloc hook is compilable... no\n",
      "checking whether glibc memalign hook is compilable... no\n",
      "checking whether pthreads adaptive mutexes is compilable... yes\n",
      "checking whether compiler supports -D_GNU_SOURCE... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether strerror_r returns char with gnu source is compilable... yes\n",
      "checking for stdbool.h that conforms to C99... yes\n",
      "checking for _Bool... yes\n",
      "configure: creating ./config.status\n",
      "config.status: creating Makefile\n",
      "config.status: creating jemalloc.pc\n",
      "config.status: creating doc/html.xsl\n",
      "config.status: creating doc/manpages.xsl\n",
      "config.status: creating doc/jemalloc.xml\n",
      "config.status: creating include/jemalloc/jemalloc_macros.h\n",
      "config.status: creating include/jemalloc/jemalloc_protos.h\n",
      "config.status: creating include/jemalloc/jemalloc_typedefs.h\n",
      "config.status: creating include/jemalloc/internal/jemalloc_preamble.h\n",
      "config.status: creating test/test.sh\n",
      "config.status: creating test/include/test/jemalloc_test.h\n",
      "config.status: creating config.stamp\n",
      "config.status: creating bin/jemalloc-config\n",
      "config.status: creating bin/jemalloc.sh\n",
      "config.status: creating bin/jeprof\n",
      "config.status: creating include/jemalloc/jemalloc_defs.h\n",
      "config.status: creating include/jemalloc/internal/jemalloc_internal_defs.h\n",
      "config.status: creating test/include/test/jemalloc_test_defs.h\n",
      "config.status: executing include/jemalloc/internal/public_symbols.txt commands\n",
      "config.status: executing include/jemalloc/internal/private_symbols.awk commands\n",
      "config.status: executing include/jemalloc/internal/private_symbols_jet.awk commands\n",
      "config.status: executing include/jemalloc/internal/public_namespace.h commands\n",
      "config.status: executing include/jemalloc/internal/public_unnamespace.h commands\n",
      "config.status: executing include/jemalloc/jemalloc_protos_jet.h commands\n",
      "config.status: executing include/jemalloc/jemalloc_rename.h commands\n",
      "config.status: executing include/jemalloc/jemalloc_mangle.h commands\n",
      "config.status: executing include/jemalloc/jemalloc_mangle_jet.h commands\n",
      "config.status: executing include/jemalloc/jemalloc.h commands\n",
      "===============================================================================\n",
      "jemalloc version   : 5.2.1-0-g0\n",
      "library revision   : 2\n",
      "\n",
      "CONFIG             : --with-version=5.2.1-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ 'CFLAGS=-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops ' LDFLAGS=\n",
      "CC                 : gcc\n",
      "CONFIGURE_CFLAGS   : -std=gnu11 -Wall -Wextra -Wsign-compare -Wundef -Wno-format-zero-length -pipe -g3 -fvisibility=hidden -O3 -funroll-loops\n",
      "SPECIFIED_CFLAGS   : -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \n",
      "EXTRA_CFLAGS       : \n",
      "CPPFLAGS           : -D_GNU_SOURCE -D_REENTRANT\n",
      "CXX                : g++\n",
      "CONFIGURE_CXXFLAGS : -Wall -Wextra -g3 -fvisibility=hidden -O3\n",
      "SPECIFIED_CXXFLAGS : \n",
      "EXTRA_CXXFLAGS     : \n",
      "LDFLAGS            : \n",
      "EXTRA_LDFLAGS      : \n",
      "DSO_LDFLAGS        : -shared -Wl,-soname,$(@F)\n",
      "LIBS               : -lm -lstdc++ -pthread\n",
      "RPATH_EXTRA        : \n",
      "\n",
      "XSLTPROC           : false\n",
      "XSLROOT            : \n",
      "\n",
      "PREFIX             : /usr/local\n",
      "BINDIR             : /usr/local/bin\n",
      "DATADIR            : /usr/local/share\n",
      "INCLUDEDIR         : /usr/local/include\n",
      "LIBDIR             : /usr/local/lib\n",
      "MANDIR             : /usr/local/share/man\n",
      "\n",
      "srcroot            : \n",
      "abs_srcroot        : /hugectr/notebooks/tmr/redis/deps/jemalloc/\n",
      "objroot            : \n",
      "abs_objroot        : /hugectr/notebooks/tmr/redis/deps/jemalloc/\n",
      "\n",
      "JEMALLOC_PREFIX    : je_\n",
      "JEMALLOC_PRIVATE_NAMESPACE\n",
      "                   : je_\n",
      "install_suffix     : \n",
      "malloc_conf        : \n",
      "documentation      : 1\n",
      "shared libs        : 1\n",
      "static libs        : 1\n",
      "autogen            : 0\n",
      "debug              : 0\n",
      "stats              : 1\n",
      "experimetal_smallocx : 0\n",
      "prof               : 0\n",
      "prof-libunwind     : 0\n",
      "prof-libgcc        : 0\n",
      "prof-gcc           : 0\n",
      "fill               : 1\n",
      "utrace             : 0\n",
      "xmalloc            : 0\n",
      "log                : 0\n",
      "lazy_lock          : 0\n",
      "cache-oblivious    : 1\n",
      "cxx                : 1\n",
      "===============================================================================\n",
      "cd jemalloc && make CFLAGS=\"-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \" LDFLAGS=\"\" lib/libjemalloc.a\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/jemalloc'\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/jemalloc.sym.o src/jemalloc.c\n",
      "nm -a src/jemalloc.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/jemalloc.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/arena.sym.o src/arena.c\n",
      "nm -a src/arena.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/arena.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/background_thread.sym.o src/background_thread.c\n",
      "nm -a src/background_thread.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/background_thread.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/base.sym.o src/base.c\n",
      "nm -a src/base.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/base.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/bin.sym.o src/bin.c\n",
      "nm -a src/bin.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/bin.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/bitmap.sym.o src/bitmap.c\n",
      "nm -a src/bitmap.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/bitmap.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ckh.sym.o src/ckh.c\n",
      "nm -a src/ckh.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ckh.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ctl.sym.o src/ctl.c\n",
      "nm -a src/ctl.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ctl.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/div.sym.o src/div.c\n",
      "nm -a src/div.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/div.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent.sym.o src/extent.c\n",
      "nm -a src/extent.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent_dss.sym.o src/extent_dss.c\n",
      "nm -a src/extent_dss.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent_dss.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent_mmap.sym.o src/extent_mmap.c\n",
      "nm -a src/extent_mmap.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent_mmap.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/hash.sym.o src/hash.c\n",
      "nm -a src/hash.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/hash.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/hook.sym.o src/hook.c\n",
      "nm -a src/hook.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/hook.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/large.sym.o src/large.c\n",
      "nm -a src/large.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/large.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/log.sym.o src/log.c\n",
      "nm -a src/log.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/log.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/malloc_io.sym.o src/malloc_io.c\n",
      "nm -a src/malloc_io.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/malloc_io.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/mutex.sym.o src/mutex.c\n",
      "nm -a src/mutex.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/mutex.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/mutex_pool.sym.o src/mutex_pool.c\n",
      "nm -a src/mutex_pool.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/mutex_pool.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/nstime.sym.o src/nstime.c\n",
      "nm -a src/nstime.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/nstime.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/pages.sym.o src/pages.c\n",
      "nm -a src/pages.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/pages.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/prng.sym.o src/prng.c\n",
      "nm -a src/prng.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/prng.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/prof.sym.o src/prof.c\n",
      "nm -a src/prof.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/prof.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/rtree.sym.o src/rtree.c\n",
      "nm -a src/rtree.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/rtree.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/safety_check.sym.o src/safety_check.c\n",
      "nm -a src/safety_check.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/safety_check.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/stats.sym.o src/stats.c\n",
      "nm -a src/stats.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/stats.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/sc.sym.o src/sc.c\n",
      "nm -a src/sc.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/sc.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/sz.sym.o src/sz.c\n",
      "nm -a src/sz.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/sz.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/tcache.sym.o src/tcache.c\n",
      "nm -a src/tcache.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/tcache.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/test_hooks.sym.o src/test_hooks.c\n",
      "nm -a src/test_hooks.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/test_hooks.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ticker.sym.o src/ticker.c\n",
      "nm -a src/ticker.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ticker.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/tsd.sym.o src/tsd.c\n",
      "nm -a src/tsd.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/tsd.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/witness.sym.o src/witness.c\n",
      "nm -a src/witness.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/witness.sym\n",
      "/bin/sh include/jemalloc/internal/private_namespace.sh src/jemalloc.sym src/arena.sym src/background_thread.sym src/base.sym src/bin.sym src/bitmap.sym src/ckh.sym src/ctl.sym src/div.sym src/extent.sym src/extent_dss.sym src/extent_mmap.sym src/hash.sym src/hook.sym src/large.sym src/log.sym src/malloc_io.sym src/mutex.sym src/mutex_pool.sym src/nstime.sym src/pages.sym src/prng.sym src/prof.sym src/rtree.sym src/safety_check.sym src/stats.sym src/sc.sym src/sz.sym src/tcache.sym src/test_hooks.sym src/ticker.sym src/tsd.sym src/witness.sym > include/jemalloc/internal/private_namespace.gen.h\n",
      "cp include/jemalloc/internal/private_namespace.gen.h include/jemalloc/internal/private_namespace.gen.h\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc.o src/jemalloc.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/arena.o src/arena.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/background_thread.o src/background_thread.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/base.o src/base.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bin.o src/bin.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bitmap.o src/bitmap.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ckh.o src/ckh.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ctl.o src/ctl.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/div.o src/div.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent.o src/extent.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent_dss.o src/extent_dss.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent_mmap.o src/extent_mmap.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hash.o src/hash.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hook.o src/hook.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/large.o src/large.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/log.o src/log.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/malloc_io.o src/malloc_io.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex.o src/mutex.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex_pool.o src/mutex_pool.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/nstime.o src/nstime.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/pages.o src/pages.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prng.o src/prng.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prof.o src/prof.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/rtree.o src/rtree.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/safety_check.o src/safety_check.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/stats.o src/stats.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/sc.o src/sc.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/sz.o src/sz.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tcache.o src/tcache.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/test_hooks.o src/test_hooks.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ticker.o src/ticker.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tsd.o src/tsd.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/witness.o src/witness.c\n",
      "g++ -Wall -Wextra -g3 -fvisibility=hidden -O3 -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc_cpp.o src/jemalloc_cpp.cpp\n",
      "ar crus lib/libjemalloc.a src/jemalloc.o src/arena.o src/background_thread.o src/base.o src/bin.o src/bitmap.o src/ckh.o src/ctl.o src/div.o src/extent.o src/extent_dss.o src/extent_mmap.o src/hash.o src/hook.o src/large.o src/log.o src/malloc_io.o src/mutex.o src/mutex_pool.o src/nstime.o src/pages.o src/prng.o src/prof.o src/rtree.o src/safety_check.o src/stats.o src/sc.o src/sz.o src/tcache.o src/test_hooks.o src/ticker.o src/tsd.o src/witness.o src/jemalloc_cpp.o\n",
      "ar: `u' modifier ignored since `D' is the default (see `U')\n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/jemalloc'\n",
      "make[2]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps'\n",
      "    \u001b[34mCC\u001b[0m \u001b[33madlist.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mquicklist.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mae.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33manet.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mdict.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mserver.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msds.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mzmalloc.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlzf_c.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlzf_d.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mpqsort.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mzipmap.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msha1.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mziplist.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrelease.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mnetworking.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mutil.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mobject.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mdb.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mreplication.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrdb.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_string.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_list.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_set.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_zset.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_hash.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mconfig.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33maof.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mpubsub.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmulti.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mdebug.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msort.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mintset.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msyncio.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcluster.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcrc16.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mendianconv.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mslowlog.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33meval.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mbio.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrio.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrand.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmemtest.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msyscheck.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcrcspeed.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcrc64.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mbitops.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msentinel.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mnotify.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msetproctitle.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mblocked.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mhyperloglog.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlatency.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msparkline.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredis-check-rdb.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredis-check-aof.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mgeo.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlazyfree.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmodule.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mevict.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mexpire.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mgeohash.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mgeohash_helper.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mchildinfo.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mdefrag.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msiphash.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrax.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_stream.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlistpack.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlocaltime.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlolwut.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlolwut5.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlolwut6.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33macl.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mtracking.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mconnection.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mtls.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msha256.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mtimeout.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msetcpuaffinity.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmonotonic.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmt19937-64.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mresp_parser.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcall_reply.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mscript_lua.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mscript.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mfunctions.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mfunction_lua.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcommands.o\u001b[0m\n",
      "    \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-server\u001b[0m\n",
      "    \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-sentinel\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredis-cli.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredisassert.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcli_common.o\u001b[0m\n",
      "    \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-cli\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredis-benchmark.o\u001b[0m\n",
      "    \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-benchmark\u001b[0m\n",
      "    \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-check-rdb\u001b[0m\n",
      "    \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-check-aof\u001b[0m\n",
      "\n",
      "Hint: It's a good idea to run 'make test' ;)\n",
      "\n",
      "make[1]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/src'\n"
     ]
    }
   ],
   "source": [
    "!rm -f 7.0.8.tar.gz && wget https://github.com/redis/redis/archive/7.0.8.tar.gz\n",
    "!rm -rf redis-7.0.8 && tar -xf 7.0.8.tar.gz && ln -sf redis-7.0.8 redis\n",
    "!cd redis && make"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "407a64da-3d6c-4d1e-89ed-34eac0bed36d",
   "metadata": {},
   "source": [
    "If you see the message `Hint: It's a good idea to run 'make test' ;)` followed by `make[1]: Leaving directory ...`, the compilation should have completed successfully."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "958de89f-5b2d-4e9d-b777-f147489b9baa",
   "metadata": {},
   "source": [
    "**Step 2: Configure a mock Redis cluster**\n",
    "\n",
    "*WARNING: The following commands will erase the all contents in the following directories: `redis-server-1`, `redis-server-2` and `redis-server-3`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cc215aa4-61ce-4afe-88ad-aa5048eb1ca3",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p redis-server-1 redis-server-2 redis-server-3\n",
    "!rm -f redis-server-1/* redis-server-2/* redis-server-3/*\n",
    "\n",
    "!ln -sf $PWD/redis/src/redis-server redis-server-1/redis-server\n",
    "!ln -sf $PWD/redis/src/redis-server redis-server-2/redis-server\n",
    "!ln -sf $PWD/redis/src/redis-server redis-server-3/redis-server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8565c081-079f-4f6d-966e-92e06c883452",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing redis-server-1/redis.conf\n"
     ]
    }
   ],
   "source": [
    "%%writefile redis-server-1/redis.conf\n",
    "daemonize yes\n",
    "port 7000\n",
    "cluster-enabled yes\n",
    "cluster-config-file nodes.conf\n",
    "appendonly no\n",
    "save \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d93aa587-3442-41d0-9976-875593810e6b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing redis-server-2/redis.conf\n"
     ]
    }
   ],
   "source": [
    "%%writefile redis-server-2/redis.conf\n",
    "daemonize yes\n",
    "port 7001\n",
    "cluster-enabled yes\n",
    "cluster-config-file nodes.conf\n",
    "appendonly no\n",
    "save \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d064a1e5-e09c-47e5-8416-c01e179dee31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing redis-server-3/redis.conf\n"
     ]
    }
   ],
   "source": [
    "%%writefile redis-server-3/redis.conf\n",
    "daemonize yes\n",
    "port 7002\n",
    "cluster-enabled yes\n",
    "cluster-config-file nodes.conf\n",
    "appendonly no\n",
    "save \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832bca57-d675-48e5-b2bd-ca661dfd3983",
   "metadata": {},
   "source": [
    "**Step 3: Form Redis cluster**\n",
    "\n",
    "*WARNING: The following command will shutdown any processes called `redis-cluster` in the current system!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8f3284a6-541c-4141-b6db-805cb8ae4c6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[29;1m>>> Performing hash slots allocation on 3 nodes...\n",
      "\u001b[0mMaster[0] -> Slots 0 - 5460\n",
      "Master[1] -> Slots 5461 - 10922\n",
      "Master[2] -> Slots 10923 - 16383\n",
      "M: fa9bb82124685a6438a696cc1562693ccc815ff0 127.0.0.1:7000\n",
      "   slots:[0-5460] (5461 slots) master\n",
      "M: c6d7ad6353bf568d17a147e65b8198ded9d65717 127.0.0.1:7001\n",
      "   slots:[5461-10922] (5462 slots) master\n",
      "M: e26ae6cfbeea8a1e6367444445364d963ae17436 127.0.0.1:7002\n",
      "   slots:[10923-16383] (5461 slots) master\n",
      "\u001b[29;1m>>> Nodes configuration updated\n",
      "\u001b[0m\u001b[29;1m>>> Assign a different config epoch to each node\n",
      "\u001b[0m\u001b[29;1m>>> Sending CLUSTER MEET messages to join the cluster\n",
      "\u001b[0mWaiting for the cluster to join\n",
      ".\n",
      "\u001b[29;1m>>> Performing Cluster Check (using node 127.0.0.1:7000)\n",
      "\u001b[0mM: fa9bb82124685a6438a696cc1562693ccc815ff0 127.0.0.1:7000\n",
      "   slots:[0-5460] (5461 slots) master\n",
      "M: e26ae6cfbeea8a1e6367444445364d963ae17436 127.0.0.1:7002\n",
      "   slots:[10923-16383] (5461 slots) master\n",
      "M: c6d7ad6353bf568d17a147e65b8198ded9d65717 127.0.0.1:7001\n",
      "   slots:[5461-10922] (5462 slots) master\n",
      "\u001b[32;1m[OK] All nodes agree about slots configuration.\n",
      "\u001b[0m\u001b[29;1m>>> Check for open slots...\n",
      "\u001b[0m\u001b[29;1m>>> Check slots coverage...\n",
      "\u001b[0m\u001b[32;1m[OK] All 16384 slots covered.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# Shutdown existing cluster (if any).\n",
    "!pkill redis-server\n",
    "\n",
    "# Reset configuration and start 3 Redis servers.\n",
    "!cd redis-server-1 && rm -f nodes.conf && ./redis-server redis.conf\n",
    "!cd redis-server-2 && rm -f nodes.conf && ./redis-server redis.conf\n",
    "!cd redis-server-3 && rm -f nodes.conf && ./redis-server redis.conf\n",
    "\n",
    "# Form the cluster.\n",
    "!redis/src/redis-cli \\\n",
    "    --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 \\\n",
    "    --cluster-yes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eefc0e17-2790-4cf8-82a3-25f0317d54d9",
   "metadata": {},
   "source": [
    "**Step 4: Run HugeCTR**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5aaa2f05-04b9-4a34-bb7c-cfa7525d0bc1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Launching...\n",
      "HPS initialized\n",
      "[HCTR][06:54:27.572][WARNING][RK0][main]: default_value_for_each_table.size() is not equal to the number of embedding tables\n",
      "====================================================HPS Create====================================================\n",
      "[HCTR][06:54:27.572][INFO][RK0][main]: Creating RedisCluster backend...\n",
      "[HCTR][06:54:27.577][INFO][RK0][main]: RedisCluster: Connecting via 127.0.0.1:7000...\n",
      "[HCTR][06:54:27.577][INFO][RK0][main]: Volatile DB: initial cache rate = 1\n",
      "[HCTR][06:54:27.577][INFO][RK0][main]: Volatile DB: cache missed embeddings = 0\n",
      "[HCTR][06:54:27.577][DEBUG][RK0][main]: Created raw model loader in local memory!\n",
      "[HCTR][06:54:27.753][INFO][RK0][main]: Table: hps_et.hps_demo.sparse_embedding1; cached 18488 / 18488 embeddings in volatile database (RedisCluster); load: 18488 / 18446744073709551615 (0.00%).\n",
      "[HCTR][06:54:27.873][INFO][RK0][main]: Table: hps_et.hps_demo.sparse_embedding2; cached 18470 / 18470 embeddings in volatile database (RedisCluster); load: 18470 / 18446744073709551615 (0.00%).\n",
      "[HCTR][06:54:30.134][DEBUG][RK0][main]: Real-time subscribers created!\n",
      "[HCTR][06:54:30.134][INFO][RK0][main]: Creating embedding cache in device 0.\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: Model name: hps_demo\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: Max batch size: 1024\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: Fuse embedding tables: False\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: Number of embedding tables: 2\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: Use GPU embedding cache: True, cache size percentage: 0.500000\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: Embedding cache type: dynamic\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: Use I64 input key: True\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: Configured cache hit rate threshold: 1.000000\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: The size of thread pool: 80\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: The size of worker memory pool: 2\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: The size of refresh memory pool: 1\n",
      "[HCTR][06:54:30.140][INFO][RK0][main]: The refresh percentage : 0.000000\n",
      "[HCTR][06:54:30.156][INFO][RK0][main]: LookupSession i64_input_key: True\n",
      "[HCTR][06:54:30.156][INFO][RK0][main]: Creating lookup session for hps_demo on device: 0\n",
      "-------------------------------------------------------------------------------\n",
      "                         HPS demo without embedding                            \n",
      "-------------------------------------------------------------------------------\n",
      "Ground truth: (1024,) = [0.4895492  0.509022   0.38192913 ... 0.5264926  0.50650454 0.47927693]\n",
      "-------------------------------------------------------------------------------\n",
      "Prediction without embedding: (1024,) = [0.48954916 0.50902206 0.38192907 ... 0.52649266 0.5065045  0.4792769 ]\n",
      "MSE between prediction and ground_truth: 2.3887142264200634e-15\n",
      "-------------------------------------------------------------------------------\n",
      "                           HPS demo with embedding                             \n",
      "-------------------------------------------------------------------------------\n",
      "Ground truth: (1024,) = [0.4895492  0.509022   0.38192913 ... 0.5264926  0.50650454 0.47927693]\n",
      "-------------------------------------------------------------------------------\n",
      "Prediction with embedding: (1024,) = [0.48954916 0.50902206 0.38192907 ... 0.52649266 0.5065045  0.4792769 ]\n",
      "MSE between prediction and ground_truth: 2.3887142264200634e-15\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-09-20 06:54:30.230052244 [W:onnxruntime:, graph.cc:3543 CleanUnusedInitializersAndNodeArgs] Removing initializer 'key_to_indice_hash_all_tables'. It is not used by any node and should be removed from the model.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import time\n",
    "import multiprocessing as mp\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import onnxruntime as ort\n",
    "from hugectr import DatabaseType_t\n",
    "from hugectr.inference import HPS, ParameterServerConfig, InferenceParams, VolatileDatabaseParams\n",
    "\n",
    "slot_size_array = [10000, 10000, 10000, 10000]\n",
    "key_offset = np.insert(np.cumsum(slot_size_array), 0, 0)[:-1]\n",
    "batch_size = 1024\n",
    "\n",
    "print('Launching...')\n",
    "\n",
    "# 1. Configure the HPS hyperparameters.\n",
    "ps_config = ParameterServerConfig(\n",
    "       emb_table_name = {'hps_demo': ['sparse_embedding1', 'sparse_embedding2']},\n",
    "       embedding_vec_size = {'hps_demo': [16, 32]},\n",
    "       max_feature_num_per_sample_per_emb_table = {'hps_demo': [2, 2]},\n",
    "       inference_params_array = [\n",
    "          InferenceParams(\n",
    "            model_name = 'hps_demo',\n",
    "            max_batchsize = batch_size,\n",
    "            hit_rate_threshold = 1.0,\n",
    "            dense_model_file = '',\n",
    "            sparse_model_files = ['hps_demo0_sparse_1000.model', 'hps_demo1_sparse_1000.model'],\n",
    "            deployed_devices = [0],\n",
    "            use_gpu_embedding_cache = True,\n",
    "            cache_size_percentage = 0.5,\n",
    "            i64_input_key = True)\n",
    "       ],\n",
    "       volatile_db = VolatileDatabaseParams(\n",
    "            DatabaseType_t.redis_cluster,\n",
    "            address = '127.0.0.1:7000',\n",
    "            num_partitions = 15,\n",
    "            num_node_connections = 5,\n",
    "       ))\n",
    "\n",
    "# 2. Initialize the HPS object.\n",
    "hps = HPS(ps_config)\n",
    "print('HPS initialized')\n",
    "\n",
    "# 3. Load query data.\n",
    "df = pd.read_parquet('data_parquet/val/gen_0.parquet')\n",
    "dense_input_columns = df.columns[1:11]\n",
    "cat_input1_columns = df.columns[11:13]\n",
    "cat_input2_columns = df.columns[13:15]\n",
    "dense_input = df[dense_input_columns].loc[0:batch_size-1].to_numpy(dtype=np.float32)\n",
    "cat_input1 = (df[cat_input1_columns].loc[0:batch_size-1].to_numpy(dtype=np.int64) + key_offset[0:2]).reshape((batch_size, 2, 1))\n",
    "cat_input2 = (df[cat_input2_columns].loc[0:batch_size-1].to_numpy(dtype=np.int64) + key_offset[2:4]).reshape((batch_size, 2, 1))\n",
    "\n",
    "# 4. Make inference from the HPS object and the ONNX inference session of `hps_demo_without_embedding.onnx`.\n",
    "embedding1 = hps.lookup(cat_input1.flatten(), 'hps_demo', 0).reshape(batch_size, 2, 16)\n",
    "embedding2 = hps.lookup(cat_input2.flatten(), 'hps_demo', 1).reshape(batch_size, 2, 32)\n",
    "sess = ort.InferenceSession('hps_demo_without_embedding.onnx')\n",
    "res = sess.run(output_names=[sess.get_outputs()[0].name],\n",
    "               input_feed={sess.get_inputs()[0].name: dense_input,\n",
    "               sess.get_inputs()[1].name: embedding1,\n",
    "               sess.get_inputs()[2].name: embedding2})\n",
    "pred = res[0].flatten()\n",
    "\n",
    "# 5. Check the correctness by comparing with dumped evaluation results.\n",
    "ground_truth = np.load(\"ground_truth.npy\").flatten()\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print('                         HPS demo without embedding                            ')\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print(f'Ground truth: {ground_truth.shape} = {ground_truth}')\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print(f'Prediction without embedding: {pred.shape} = {pred}')\n",
    "\n",
    "diff = pred - ground_truth\n",
    "mse = np.mean(diff * diff)\n",
    "print(f'MSE between prediction and ground_truth: {mse}')\n",
    "\n",
    "# 6. Make inference with the ONNX inference session of `hps_demo_with_embedding.onnx` (double check).\n",
    "sess_ref = ort.InferenceSession('hps_demo_with_embedding.onnx')\n",
    "res_ref = sess_ref.run(output_names=[sess_ref.get_outputs()[0].name],\n",
    "               input_feed={sess_ref.get_inputs()[0].name: dense_input,\n",
    "               sess_ref.get_inputs()[1].name: cat_input1,\n",
    "               sess_ref.get_inputs()[2].name: cat_input2})\n",
    "pred_ref = res_ref[0].flatten()\n",
    "\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print('                           HPS demo with embedding                             ')\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print(f'Ground truth: {ground_truth.shape} = {ground_truth}')\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print(f'Prediction with embedding: {pred_ref.shape} = {pred_ref}')\n",
    "\n",
    "diff_ref = pred_ref.flatten() - ground_truth\n",
    "mse_ref = np.mean(diff_ref * diff_ref)\n",
    "print(f'MSE between prediction and ground_truth: {mse_ref}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c3cba92-c244-43db-853a-6228ace3b869",
   "metadata": {},
   "source": [
    "**Step 5: Shutdown Redis cluster**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "596efa2e-54d5-4e3e-99ba-7ebb4736a7fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pkill redis-server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75de11c-cc31-471b-9cfe-8d1244656ffe",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "<a id=\"section-5\"></a>\n",
    "## 5. Redis Cluster deployment (with TLS/SSL)\n",
    "When using Redis as backing storage, HugeCTR can use make use of TLS/SSL to encrypt data transfers. In the following steps we setupt a small Redis cluster and enable SSL for it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa75efdc-5da4-49a9-a9ee-be7062814b00",
   "metadata": {},
   "source": [
    "**Step 1: Build a TLS/SSL capable distribution of Redis**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "158cc2ba-7d9f-4c31-9adc-3d09cb00c1ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-09-20 06:55:14--  https://github.com/redis/redis/archive/7.0.8.tar.gz\n",
      "Resolving github.com (github.com)... 192.30.255.112\n",
      "Connecting to github.com (github.com)|192.30.255.112|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://codeload.github.com/redis/redis/tar.gz/refs/tags/7.0.8 [following]\n",
      "--2023-09-20 06:55:14--  https://codeload.github.com/redis/redis/tar.gz/refs/tags/7.0.8\n",
      "Resolving codeload.github.com (codeload.github.com)... 192.30.255.121\n",
      "Connecting to codeload.github.com (codeload.github.com)|192.30.255.121|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: unspecified [application/x-gzip]\n",
      "Saving to: ‘7.0.8.tar.gz’\n",
      "\n",
      "7.0.8.tar.gz            [     <=>            ]   2.87M  3.24MB/s    in 0.9s    \n",
      "\n",
      "2023-09-20 06:55:15 (3.24 MB/s) - ‘7.0.8.tar.gz’ saved [3011655]\n",
      "\n",
      "cd src && make all\n",
      "make[1]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/src'\n",
      "./mkreleasehdr.sh: line 2: echo: write error: Broken pipe\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mMakefile.dep\u001b[0m\n",
      "./mkreleasehdr.sh: line 2: echo: write error: Broken pipe\n",
      "rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-rdb redis-check-aof *.o *.gcda *.gcno *.gcov redis.info lcov-html Makefile.dep\n",
      "rm -f adlist.d quicklist.d ae.d anet.d dict.d server.d sds.d zmalloc.d lzf_c.d lzf_d.d pqsort.d zipmap.d sha1.d ziplist.d release.d networking.d util.d object.d db.d replication.d rdb.d t_string.d t_list.d t_set.d t_zset.d t_hash.d config.d aof.d pubsub.d multi.d debug.d sort.d intset.d syncio.d cluster.d crc16.d endianconv.d slowlog.d eval.d bio.d rio.d rand.d memtest.d syscheck.d crcspeed.d crc64.d bitops.d sentinel.d notify.d setproctitle.d blocked.d hyperloglog.d latency.d sparkline.d redis-check-rdb.d redis-check-aof.d geo.d lazyfree.d module.d evict.d expire.d geohash.d geohash_helper.d childinfo.d defrag.d siphash.d rax.d t_stream.d listpack.d localtime.d lolwut.d lolwut5.d lolwut6.d acl.d tracking.d connection.d tls.d sha256.d timeout.d setcpuaffinity.d monotonic.d mt19937-64.d resp_parser.d call_reply.d script_lua.d script.d functions.d function_lua.d commands.d anet.d adlist.d dict.d redis-cli.d zmalloc.d release.d ae.d redisassert.d crcspeed.d crc64.d siphash.d crc16.d monotonic.d cli_common.d mt19937-64.d ae.d anet.d redis-benchmark.d adlist.d dict.d zmalloc.d redisassert.d release.d crcspeed.d crc64.d siphash.d crc16.d monotonic.d cli_common.d mt19937-64.d\n",
      "(cd ../deps && make distclean)\n",
      "make[2]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps'\n",
      "(cd hiredis && make clean) > /dev/null || true\n",
      "(cd linenoise && make clean) > /dev/null || true\n",
      "(cd lua && make clean) > /dev/null || true\n",
      "(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true\n",
      "(cd hdr_histogram && make clean) > /dev/null || true\n",
      "(rm -f .make-*)\n",
      "make[2]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps'\n",
      "(cd modules && make clean)\n",
      "make[2]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/src/modules'\n",
      "rm -rf *.xo *.so\n",
      "make[2]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/src/modules'\n",
      "(cd ../tests/modules && make clean)\n",
      "make[2]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/tests/modules'\n",
      "rm -f commandfilter.so basics.so testrdb.so fork.so infotest.so propagate.so misc.so hooks.so blockonkeys.so blockonbackground.so scan.so datatype.so datatype2.so auth.so keyspace_events.so blockedclient.so getkeys.so getchannels.so test_lazyfree.so timer.so defragtest.so keyspecs.so hash.so zset.so stream.so mallocsize.so aclcheck.so list.so subcommands.so reply.so cmdintrospection.so eventloop.so moduleconfigs.so moduleconfigstwo.so publish.so usercall.so commandfilter.xo basics.xo testrdb.xo fork.xo infotest.xo propagate.xo misc.xo hooks.xo blockonkeys.xo blockonbackground.xo scan.xo datatype.xo datatype2.xo auth.xo keyspace_events.xo blockedclient.xo getkeys.xo getchannels.xo test_lazyfree.xo timer.xo defragtest.xo keyspecs.xo hash.xo zset.xo stream.xo mallocsize.xo aclcheck.xo list.xo subcommands.xo reply.xo cmdintrospection.xo eventloop.xo moduleconfigs.xo moduleconfigstwo.xo publish.xo usercall.xo\n",
      "make[2]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/tests/modules'\n",
      "(rm -f .make-*)\n",
      "echo STD=-pedantic -DREDIS_STATIC='' -std=c11 >> .make-settings\n",
      "echo WARN=-Wall -W -Wno-missing-field-initializers >> .make-settings\n",
      "echo OPT=-O2 >> .make-settings\n",
      "echo MALLOC=jemalloc >> .make-settings\n",
      "echo BUILD_TLS=yes >> .make-settings\n",
      "echo USE_SYSTEMD= >> .make-settings\n",
      "echo CFLAGS= >> .make-settings\n",
      "echo LDFLAGS= >> .make-settings\n",
      "echo REDIS_CFLAGS= >> .make-settings\n",
      "echo REDIS_LDFLAGS= >> .make-settings\n",
      "echo PREV_FINAL_CFLAGS=-pedantic -DREDIS_STATIC='' -std=c11 -Wall -W -Wno-missing-field-initializers -O2 -g -ggdb   -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -I../deps/hdr_histogram -DUSE_JEMALLOC -I../deps/jemalloc/include -DUSE_OPENSSL  >> .make-settings\n",
      "echo PREV_FINAL_LDFLAGS=  -g -ggdb -rdynamic  >> .make-settings\n",
      "(cd ../deps && make hiredis linenoise lua hdr_histogram jemalloc)\n",
      "make[2]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps'\n",
      "(cd hiredis && make clean) > /dev/null || true\n",
      "(cd linenoise && make clean) > /dev/null || true\n",
      "(cd lua && make clean) > /dev/null || true\n",
      "(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true\n",
      "(cd hdr_histogram && make clean) > /dev/null || true\n",
      "(rm -f .make-*)\n",
      "(echo \"\" > .make-cflags)\n",
      "(echo \"\" > .make-ldflags)\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mhiredis\u001b[0m\n",
      "cd hiredis && make static USE_SSL=1\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/hiredis'\n",
      "cc -std=c99 -c -O3 -fPIC  -DHIREDIS_TEST_SSL -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic alloc.c\n",
      "cc -std=c99 -c -O3 -fPIC  -DHIREDIS_TEST_SSL -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic net.c\n",
      "cc -std=c99 -c -O3 -fPIC  -DHIREDIS_TEST_SSL -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic hiredis.c\n",
      "cc -std=c99 -c -O3 -fPIC  -DHIREDIS_TEST_SSL -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic sds.c\n",
      "cc -std=c99 -c -O3 -fPIC  -DHIREDIS_TEST_SSL -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic async.c\n",
      "cc -std=c99 -c -O3 -fPIC  -DHIREDIS_TEST_SSL -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic read.c\n",
      "cc -std=c99 -c -O3 -fPIC  -DHIREDIS_TEST_SSL -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic sockcompat.c\n",
      "ar rcs libhiredis.a alloc.o net.o hiredis.o sds.o async.o read.o sockcompat.o\n",
      "cc -std=c99 -c -O3 -fPIC  -DHIREDIS_TEST_SSL -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb -pedantic ssl.c\n",
      "ar rcs libhiredis_ssl.a ssl.o\n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/hiredis'\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mlinenoise\u001b[0m\n",
      "cd linenoise && make\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/linenoise'\n",
      "cc  -Wall -Os -g  -c linenoise.c\n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/linenoise'\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mlua\u001b[0m\n",
      "cd lua/src && make all CFLAGS=\"-Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2 \" MYLDFLAGS=\"\" AR=\"ar rc\"\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/lua/src'\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lapi.o lapi.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lcode.o lcode.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ldebug.o ldebug.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ldo.o ldo.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ldump.o ldump.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lfunc.o lfunc.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lgc.o lgc.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o llex.o llex.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lmem.o lmem.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lobject.o lobject.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lopcodes.o lopcodes.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lparser.o lparser.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lstate.o lstate.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lstring.o lstring.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ltable.o ltable.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ltm.o ltm.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lundump.o lundump.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lvm.o lvm.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lzio.o lzio.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o strbuf.o strbuf.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o fpconv.o fpconv.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lauxlib.o lauxlib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lbaselib.o lbaselib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ldblib.o ldblib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o liolib.o liolib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lmathlib.o lmathlib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o loslib.o loslib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o ltablib.o ltablib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lstrlib.o lstrlib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o loadlib.o loadlib.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o linit.o linit.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua_cjson.o lua_cjson.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua_struct.o lua_struct.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua_cmsgpack.o lua_cmsgpack.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua_bit.o lua_bit.c\n",
      "ar rc liblua.a lapi.o lcode.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o lundump.o lvm.o lzio.o strbuf.o fpconv.o lauxlib.o lbaselib.o ldblib.o liolib.o lmathlib.o loslib.o ltablib.o lstrlib.o loadlib.o linit.o lua_cjson.o lua_struct.o lua_cmsgpack.o lua_bit.o\t# DLL needs all object files\n",
      "ranlib liblua.a\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o lua.o lua.c\n",
      "cc -o lua  lua.o liblua.a -lm \n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o luac.o luac.c\n",
      "cc -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -DLUA_USE_MKSTEMP  -O2    -c -o print.o print.c\n",
      "cc -o luac  luac.o print.o liblua.a -lm \n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/lua/src'\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mhdr_histogram\u001b[0m\n",
      "cd hdr_histogram && make\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/hdr_histogram'\n",
      "cc -std=c99 -Wall -Os -g  -DHDR_MALLOC_INCLUDE=\\\"hdr_redis_malloc.h\\\" -c  hdr_histogram.c \n",
      "ar rcs libhdrhistogram.a hdr_histogram.o\n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/hdr_histogram'\n",
      "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mjemalloc\u001b[0m\n",
      "cd jemalloc && ./configure --with-version=5.2.1-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ CFLAGS=\"-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \" LDFLAGS=\"\" \n",
      "checking for xsltproc... false\n",
      "checking for gcc... gcc\n",
      "checking whether the C compiler works... yes\n",
      "checking for C compiler default output file name... a.out\n",
      "checking for suffix of executables... \n",
      "checking whether we are cross compiling... no\n",
      "checking for suffix of object files... o\n",
      "checking whether we are using the GNU C compiler... yes\n",
      "checking whether gcc accepts -g... yes\n",
      "checking for gcc option to accept ISO C89... none needed\n",
      "checking whether compiler is cray... no\n",
      "checking whether compiler supports -std=gnu11... yes\n",
      "checking whether compiler supports -Wall... yes\n",
      "checking whether compiler supports -Wextra... yes\n",
      "checking whether compiler supports -Wshorten-64-to-32... no\n",
      "checking whether compiler supports -Wsign-compare... yes\n",
      "checking whether compiler supports -Wundef... yes\n",
      "checking whether compiler supports -Wno-format-zero-length... yes\n",
      "checking whether compiler supports -pipe... yes\n",
      "checking whether compiler supports -g3... yes\n",
      "checking how to run the C preprocessor... gcc -E\n",
      "checking for g++... g++\n",
      "checking whether we are using the GNU C++ compiler... yes\n",
      "checking whether g++ accepts -g... yes\n",
      "checking whether g++ supports C++14 features by default... yes\n",
      "checking whether compiler supports -Wall... yes\n",
      "checking whether compiler supports -Wextra... yes\n",
      "checking whether compiler supports -g3... yes\n",
      "checking whether libstdc++ linkage is compilable... yes\n",
      "checking for grep that handles long lines and -e... /usr/bin/grep\n",
      "checking for egrep... /usr/bin/grep -E\n",
      "checking for ANSI C header files... yes\n",
      "checking for sys/types.h... yes\n",
      "checking for sys/stat.h... yes\n",
      "checking for stdlib.h... yes\n",
      "checking for string.h... yes\n",
      "checking for memory.h... yes\n",
      "checking for strings.h... yes\n",
      "checking for inttypes.h... yes\n",
      "checking for stdint.h... yes\n",
      "checking for unistd.h... yes\n",
      "checking whether byte ordering is bigendian... no\n",
      "checking size of void *... 8\n",
      "checking size of int... 4\n",
      "checking size of long... 8\n",
      "checking size of long long... 8\n",
      "checking size of intmax_t... 8\n",
      "checking build system type... x86_64-pc-linux-gnu\n",
      "checking host system type... x86_64-pc-linux-gnu\n",
      "checking whether pause instruction is compilable... yes\n",
      "checking number of significant virtual address bits... 48\n",
      "checking for ar... ar\n",
      "checking for nm... nm\n",
      "checking for gawk... no\n",
      "checking for mawk... mawk\n",
      "checking malloc.h usability... yes\n",
      "checking malloc.h presence... yes\n",
      "checking for malloc.h... yes\n",
      "checking whether malloc_usable_size definition can use const argument... no\n",
      "checking for library containing log... -lm\n",
      "checking whether __attribute__ syntax is compilable... yes\n",
      "checking whether compiler supports -fvisibility=hidden... yes\n",
      "checking whether compiler supports -fvisibility=hidden... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether tls_model attribute is compilable... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether alloc_size attribute is compilable... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether format(gnu_printf, ...) attribute is compilable... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether format(printf, ...) attribute is compilable... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether format(printf, ...) attribute is compilable... yes\n",
      "checking for a BSD-compatible install... /usr/bin/install -c\n",
      "checking for ranlib... ranlib\n",
      "checking for ld... /usr/bin/ld\n",
      "checking for autoconf... /usr/bin/autoconf\n",
      "checking for memalign... yes\n",
      "checking for valloc... yes\n",
      "checking whether compiler supports -O3... yes\n",
      "checking whether compiler supports -O3... yes\n",
      "checking whether compiler supports -funroll-loops... yes\n",
      "checking configured backtracing method... N/A\n",
      "checking for sbrk... yes\n",
      "checking whether utrace(2) is compilable... no\n",
      "checking whether a program using __builtin_unreachable is compilable... yes\n",
      "checking whether a program using __builtin_ffsl is compilable... yes\n",
      "checking whether a program using __builtin_popcountl is compilable... yes\n",
      "checking LG_PAGE... 12\n",
      "checking pthread.h usability... yes\n",
      "checking pthread.h presence... yes\n",
      "checking for pthread.h... yes\n",
      "checking for pthread_create in -lpthread... yes\n",
      "checking dlfcn.h usability... yes\n",
      "checking dlfcn.h presence... yes\n",
      "checking for dlfcn.h... yes\n",
      "checking for dlsym... yes\n",
      "checking whether pthread_atfork(3) is compilable... yes\n",
      "checking whether pthread_setname_np(3) is compilable... yes\n",
      "checking for library containing clock_gettime... none required\n",
      "checking whether clock_gettime(CLOCK_MONOTONIC_COARSE, ...) is compilable... yes\n",
      "checking whether clock_gettime(CLOCK_MONOTONIC, ...) is compilable... yes\n",
      "checking whether mach_absolute_time() is compilable... no\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether syscall(2) is compilable... yes\n",
      "checking for secure_getenv... yes\n",
      "checking for sched_getcpu... yes\n",
      "checking for sched_setaffinity... yes\n",
      "checking for issetugid... no\n",
      "checking for _malloc_thread_cleanup... no\n",
      "checking for _pthread_mutex_init_calloc_cb... no\n",
      "checking for TLS... yes\n",
      "checking whether C11 atomics is compilable... no\n",
      "checking whether GCC __atomic atomics is compilable... yes\n",
      "checking whether GCC 8-bit __atomic atomics is compilable... yes\n",
      "checking whether GCC __sync atomics is compilable... yes\n",
      "checking whether GCC 8-bit __sync atomics is compilable... yes\n",
      "checking whether Darwin OSAtomic*() is compilable... no\n",
      "checking whether madvise(2) is compilable... yes\n",
      "checking whether madvise(..., MADV_FREE) is compilable... yes\n",
      "checking whether madvise(..., MADV_DONTNEED) is compilable... yes\n",
      "checking whether madvise(..., MADV_DO[NT]DUMP) is compilable... yes\n",
      "checking whether madvise(..., MADV_[NO]HUGEPAGE) is compilable... yes\n",
      "checking for __builtin_clz... yes\n",
      "checking whether Darwin os_unfair_lock_*() is compilable... no\n",
      "checking whether glibc malloc hook is compilable... no\n",
      "checking whether glibc memalign hook is compilable... no\n",
      "checking whether pthreads adaptive mutexes is compilable... yes\n",
      "checking whether compiler supports -D_GNU_SOURCE... yes\n",
      "checking whether compiler supports -Werror... yes\n",
      "checking whether compiler supports -herror_on_warning... yes\n",
      "checking whether strerror_r returns char with gnu source is compilable... yes\n",
      "checking for stdbool.h that conforms to C99... yes\n",
      "checking for _Bool... yes\n",
      "configure: creating ./config.status\n",
      "config.status: creating Makefile\n",
      "config.status: creating jemalloc.pc\n",
      "config.status: creating doc/html.xsl\n",
      "config.status: creating doc/manpages.xsl\n",
      "config.status: creating doc/jemalloc.xml\n",
      "config.status: creating include/jemalloc/jemalloc_macros.h\n",
      "config.status: creating include/jemalloc/jemalloc_protos.h\n",
      "config.status: creating include/jemalloc/jemalloc_typedefs.h\n",
      "config.status: creating include/jemalloc/internal/jemalloc_preamble.h\n",
      "config.status: creating test/test.sh\n",
      "config.status: creating test/include/test/jemalloc_test.h\n",
      "config.status: creating config.stamp\n",
      "config.status: creating bin/jemalloc-config\n",
      "config.status: creating bin/jemalloc.sh\n",
      "config.status: creating bin/jeprof\n",
      "config.status: creating include/jemalloc/jemalloc_defs.h\n",
      "config.status: creating include/jemalloc/internal/jemalloc_internal_defs.h\n",
      "config.status: creating test/include/test/jemalloc_test_defs.h\n",
      "config.status: executing include/jemalloc/internal/public_symbols.txt commands\n",
      "config.status: executing include/jemalloc/internal/private_symbols.awk commands\n",
      "config.status: executing include/jemalloc/internal/private_symbols_jet.awk commands\n",
      "config.status: executing include/jemalloc/internal/public_namespace.h commands\n",
      "config.status: executing include/jemalloc/internal/public_unnamespace.h commands\n",
      "config.status: executing include/jemalloc/jemalloc_protos_jet.h commands\n",
      "config.status: executing include/jemalloc/jemalloc_rename.h commands\n",
      "config.status: executing include/jemalloc/jemalloc_mangle.h commands\n",
      "config.status: executing include/jemalloc/jemalloc_mangle_jet.h commands\n",
      "config.status: executing include/jemalloc/jemalloc.h commands\n",
      "===============================================================================\n",
      "jemalloc version   : 5.2.1-0-g0\n",
      "library revision   : 2\n",
      "\n",
      "CONFIG             : --with-version=5.2.1-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ 'CFLAGS=-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops ' LDFLAGS=\n",
      "CC                 : gcc\n",
      "CONFIGURE_CFLAGS   : -std=gnu11 -Wall -Wextra -Wsign-compare -Wundef -Wno-format-zero-length -pipe -g3 -fvisibility=hidden -O3 -funroll-loops\n",
      "SPECIFIED_CFLAGS   : -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \n",
      "EXTRA_CFLAGS       : \n",
      "CPPFLAGS           : -D_GNU_SOURCE -D_REENTRANT\n",
      "CXX                : g++\n",
      "CONFIGURE_CXXFLAGS : -Wall -Wextra -g3 -fvisibility=hidden -O3\n",
      "SPECIFIED_CXXFLAGS : \n",
      "EXTRA_CXXFLAGS     : \n",
      "LDFLAGS            : \n",
      "EXTRA_LDFLAGS      : \n",
      "DSO_LDFLAGS        : -shared -Wl,-soname,$(@F)\n",
      "LIBS               : -lm -lstdc++ -pthread\n",
      "RPATH_EXTRA        : \n",
      "\n",
      "XSLTPROC           : false\n",
      "XSLROOT            : \n",
      "\n",
      "PREFIX             : /usr/local\n",
      "BINDIR             : /usr/local/bin\n",
      "DATADIR            : /usr/local/share\n",
      "INCLUDEDIR         : /usr/local/include\n",
      "LIBDIR             : /usr/local/lib\n",
      "MANDIR             : /usr/local/share/man\n",
      "\n",
      "srcroot            : \n",
      "abs_srcroot        : /hugectr/notebooks/tmr/redis/deps/jemalloc/\n",
      "objroot            : \n",
      "abs_objroot        : /hugectr/notebooks/tmr/redis/deps/jemalloc/\n",
      "\n",
      "JEMALLOC_PREFIX    : je_\n",
      "JEMALLOC_PRIVATE_NAMESPACE\n",
      "                   : je_\n",
      "install_suffix     : \n",
      "malloc_conf        : \n",
      "documentation      : 1\n",
      "shared libs        : 1\n",
      "static libs        : 1\n",
      "autogen            : 0\n",
      "debug              : 0\n",
      "stats              : 1\n",
      "experimetal_smallocx : 0\n",
      "prof               : 0\n",
      "prof-libunwind     : 0\n",
      "prof-libgcc        : 0\n",
      "prof-gcc           : 0\n",
      "fill               : 1\n",
      "utrace             : 0\n",
      "xmalloc            : 0\n",
      "log                : 0\n",
      "lazy_lock          : 0\n",
      "cache-oblivious    : 1\n",
      "cxx                : 1\n",
      "===============================================================================\n",
      "cd jemalloc && make CFLAGS=\"-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \" LDFLAGS=\"\" lib/libjemalloc.a\n",
      "make[3]: Entering directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/jemalloc'\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/jemalloc.sym.o src/jemalloc.c\n",
      "nm -a src/jemalloc.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/jemalloc.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/arena.sym.o src/arena.c\n",
      "nm -a src/arena.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/arena.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/background_thread.sym.o src/background_thread.c\n",
      "nm -a src/background_thread.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/background_thread.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/base.sym.o src/base.c\n",
      "nm -a src/base.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/base.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/bin.sym.o src/bin.c\n",
      "nm -a src/bin.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/bin.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/bitmap.sym.o src/bitmap.c\n",
      "nm -a src/bitmap.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/bitmap.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ckh.sym.o src/ckh.c\n",
      "nm -a src/ckh.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ckh.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ctl.sym.o src/ctl.c\n",
      "nm -a src/ctl.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ctl.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/div.sym.o src/div.c\n",
      "nm -a src/div.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/div.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent.sym.o src/extent.c\n",
      "nm -a src/extent.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent_dss.sym.o src/extent_dss.c\n",
      "nm -a src/extent_dss.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent_dss.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent_mmap.sym.o src/extent_mmap.c\n",
      "nm -a src/extent_mmap.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent_mmap.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/hash.sym.o src/hash.c\n",
      "nm -a src/hash.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/hash.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/hook.sym.o src/hook.c\n",
      "nm -a src/hook.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/hook.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/large.sym.o src/large.c\n",
      "nm -a src/large.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/large.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/log.sym.o src/log.c\n",
      "nm -a src/log.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/log.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/malloc_io.sym.o src/malloc_io.c\n",
      "nm -a src/malloc_io.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/malloc_io.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/mutex.sym.o src/mutex.c\n",
      "nm -a src/mutex.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/mutex.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/mutex_pool.sym.o src/mutex_pool.c\n",
      "nm -a src/mutex_pool.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/mutex_pool.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/nstime.sym.o src/nstime.c\n",
      "nm -a src/nstime.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/nstime.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/pages.sym.o src/pages.c\n",
      "nm -a src/pages.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/pages.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/prng.sym.o src/prng.c\n",
      "nm -a src/prng.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/prng.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/prof.sym.o src/prof.c\n",
      "nm -a src/prof.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/prof.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/rtree.sym.o src/rtree.c\n",
      "nm -a src/rtree.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/rtree.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/safety_check.sym.o src/safety_check.c\n",
      "nm -a src/safety_check.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/safety_check.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/stats.sym.o src/stats.c\n",
      "nm -a src/stats.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/stats.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/sc.sym.o src/sc.c\n",
      "nm -a src/sc.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/sc.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/sz.sym.o src/sz.c\n",
      "nm -a src/sz.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/sz.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/tcache.sym.o src/tcache.c\n",
      "nm -a src/tcache.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/tcache.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/test_hooks.sym.o src/test_hooks.c\n",
      "nm -a src/test_hooks.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/test_hooks.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ticker.sym.o src/ticker.c\n",
      "nm -a src/ticker.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ticker.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/tsd.sym.o src/tsd.c\n",
      "nm -a src/tsd.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/tsd.sym\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/witness.sym.o src/witness.c\n",
      "nm -a src/witness.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/witness.sym\n",
      "/bin/sh include/jemalloc/internal/private_namespace.sh src/jemalloc.sym src/arena.sym src/background_thread.sym src/base.sym src/bin.sym src/bitmap.sym src/ckh.sym src/ctl.sym src/div.sym src/extent.sym src/extent_dss.sym src/extent_mmap.sym src/hash.sym src/hook.sym src/large.sym src/log.sym src/malloc_io.sym src/mutex.sym src/mutex_pool.sym src/nstime.sym src/pages.sym src/prng.sym src/prof.sym src/rtree.sym src/safety_check.sym src/stats.sym src/sc.sym src/sz.sym src/tcache.sym src/test_hooks.sym src/ticker.sym src/tsd.sym src/witness.sym > include/jemalloc/internal/private_namespace.gen.h\n",
      "cp include/jemalloc/internal/private_namespace.gen.h include/jemalloc/internal/private_namespace.gen.h\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc.o src/jemalloc.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/arena.o src/arena.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/background_thread.o src/background_thread.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/base.o src/base.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bin.o src/bin.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bitmap.o src/bitmap.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ckh.o src/ckh.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ctl.o src/ctl.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/div.o src/div.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent.o src/extent.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent_dss.o src/extent_dss.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent_mmap.o src/extent_mmap.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hash.o src/hash.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hook.o src/hook.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/large.o src/large.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/log.o src/log.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/malloc_io.o src/malloc_io.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex.o src/mutex.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex_pool.o src/mutex_pool.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/nstime.o src/nstime.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/pages.o src/pages.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prng.o src/prng.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prof.o src/prof.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/rtree.o src/rtree.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/safety_check.o src/safety_check.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/stats.o src/stats.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/sc.o src/sc.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/sz.o src/sz.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tcache.o src/tcache.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/test_hooks.o src/test_hooks.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ticker.o src/ticker.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tsd.o src/tsd.c\n",
      "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/witness.o src/witness.c\n",
      "g++ -Wall -Wextra -g3 -fvisibility=hidden -O3 -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc_cpp.o src/jemalloc_cpp.cpp\n",
      "ar crus lib/libjemalloc.a src/jemalloc.o src/arena.o src/background_thread.o src/base.o src/bin.o src/bitmap.o src/ckh.o src/ctl.o src/div.o src/extent.o src/extent_dss.o src/extent_mmap.o src/hash.o src/hook.o src/large.o src/log.o src/malloc_io.o src/mutex.o src/mutex_pool.o src/nstime.o src/pages.o src/prng.o src/prof.o src/rtree.o src/safety_check.o src/stats.o src/sc.o src/sz.o src/tcache.o src/test_hooks.o src/ticker.o src/tsd.o src/witness.o src/jemalloc_cpp.o\n",
      "ar: `u' modifier ignored since `D' is the default (see `U')\n",
      "make[3]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps/jemalloc'\n",
      "make[2]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/deps'\n",
      "    \u001b[34mCC\u001b[0m \u001b[33madlist.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mquicklist.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mae.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33manet.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mdict.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mserver.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msds.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mzmalloc.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlzf_c.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlzf_d.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mpqsort.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mzipmap.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msha1.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mziplist.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrelease.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mnetworking.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mutil.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mobject.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mdb.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mreplication.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrdb.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_string.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_list.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_set.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_zset.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_hash.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mconfig.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33maof.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mpubsub.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmulti.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mdebug.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msort.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mintset.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msyncio.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcluster.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcrc16.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mendianconv.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mslowlog.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33meval.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mbio.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrio.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrand.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmemtest.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msyscheck.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcrcspeed.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcrc64.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mbitops.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msentinel.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mnotify.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msetproctitle.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mblocked.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mhyperloglog.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlatency.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msparkline.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredis-check-rdb.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredis-check-aof.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mgeo.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlazyfree.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmodule.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mevict.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mexpire.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mgeohash.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mgeohash_helper.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mchildinfo.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mdefrag.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msiphash.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mrax.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mt_stream.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlistpack.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlocaltime.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlolwut.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlolwut5.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mlolwut6.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33macl.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mtracking.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mconnection.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mtls.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msha256.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mtimeout.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33msetcpuaffinity.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmonotonic.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mmt19937-64.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mresp_parser.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcall_reply.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mscript_lua.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mscript.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mfunctions.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mfunction_lua.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcommands.o\u001b[0m\n",
      "    \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-server\u001b[0m\n",
      "    \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-sentinel\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredis-cli.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredisassert.o\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mcli_common.o\u001b[0m\n",
      "    \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-cli\u001b[0m\n",
      "    \u001b[34mCC\u001b[0m \u001b[33mredis-benchmark.o\u001b[0m\n",
      "    \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-benchmark\u001b[0m\n",
      "    \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-check-rdb\u001b[0m\n",
      "    \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-check-aof\u001b[0m\n",
      "\n",
      "Hint: It's a good idea to run 'make test' ;)\n",
      "\n",
      "make[1]: Leaving directory '/hugectr/notebooks/tmr/redis-7.0.8/src'\n"
     ]
    }
   ],
   "source": [
    "!rm -f 7.0.8.tar.gz && wget https://github.com/redis/redis/archive/7.0.8.tar.gz\n",
    "!rm -rf redis-7.0.8 && tar -xf 7.0.8.tar.gz && ln -sf redis-7.0.8 redis\n",
    "!cd redis && make BUILD_TLS=yes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "567a3685-1562-4c45-8941-907718e91857",
   "metadata": {},
   "source": [
    "If you see the message `Hint: It's a good idea to run 'make test' ;)` followed by `make[1]: Leaving directory ...`, the compilation should have completed successfully."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe799ccf-f836-4fea-a115-f48b501d1e11",
   "metadata": {},
   "source": [
    "**Step 2: Configure a mock Redis cluster**\n",
    "\n",
    "Setup TLS/SSL certificates. Can skip if encyryption is not needed.\n",
    "\n",
    "*WARNING: The following commands will erase the all contents in the following directories: `test_certs`, `redis-server-1`, `redis-server-2` and `redis-server-3`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "83614bce-2f17-4561-9b19-fda1c6a1da87",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Certificate request self-signature ok\n",
      "subject=O = NVIDIA Merlin, CN = Redis Server\n",
      "Certificate request self-signature ok\n",
      "subject=O = NVIDIA Merlin, CN = HugeCTR Redis Client\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p test_certs\n",
    "!rm -f test_certs/*\n",
    "\n",
    "with open(\"test_certs/openssl.conf\", \"w\") as f:\n",
    "    f.write(\"\"\"[ redis_server ]\n",
    "keyUsage = digitalSignature, keyEncipherment\n",
    "\n",
    "[ hugectr_client ]\n",
    "keyUsage = digitalSignature, keyEncipherment\n",
    "nsCertType = client\"\"\")\n",
    "    \n",
    "# Create private keys for CA, Redis server and HugeCTR client.\n",
    "!openssl genrsa -out test_certs/ca-private.pem 4096\n",
    "!openssl genrsa -out test_certs/redis-private.pem 4096\n",
    "!openssl genrsa -out test_certs/hugectr-private.pem 4096\n",
    "\n",
    "# Create public keys for CA, Redis server and HugeCTR client.\n",
    "#!openssl rsa -pubout -in test_certs/ca-private.pem -out test_certs/ca-public.pem\n",
    "#!openssl rsa -pubout -in test_certs/redis-private.pem -out test_certs/redis-public.pem\n",
    "#!openssl rsa -pubout -in test_certs/hugectr-private.pem -out test_certs/hugectr-public.pem\n",
    "\n",
    "# Form dummy CA.\n",
    "!openssl req -new -nodes -sha256 -x509 -subj '/O=NVIDIA Merlin/CN=Certificate Authority' -days 365 \\\n",
    "    -key test_certs/ca-private.pem \\\n",
    "    -out test_certs/ca.crt\n",
    "    \n",
    "# Generate certificate for Redis server.\n",
    "!openssl req -new -sha256 -subj \"/O=NVIDIA Merlin/CN=Redis Server\" \\\n",
    "    -key test_certs/redis-private.pem | \\\n",
    "        openssl x509 -req -sha256 \\\n",
    "            -CA test_certs/ca.crt \\\n",
    "            -CAkey test_certs/ca-private.pem \\\n",
    "            -CAserial test_certs/redis.ser \\\n",
    "            -CAcreateserial \\\n",
    "            -days 365 \\\n",
    "            -extfile test_certs/openssl.conf -extensions redis_server \\\n",
    "            -out test_certs/redis.crt\n",
    "\n",
    "# Generate certificate for HugeCTR client.\n",
    "!openssl req -new -sha256 -subj \"/O=NVIDIA Merlin/CN=HugeCTR Redis Client\" \\\n",
    "        -key test_certs/hugectr-private.pem | \\\n",
    "        openssl x509 \\\n",
    "            -req -sha256 \\\n",
    "            -CA test_certs/ca.crt \\\n",
    "            -CAkey test_certs/ca-private.pem \\\n",
    "            -CAserial test_certs/hugectr.ser \\\n",
    "            -CAcreateserial \\\n",
    "            -days 365 \\\n",
    "            -extfile test_certs/openssl.conf -extensions hugectr_client \\\n",
    "            -out test_certs/hugectr.crt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e5105c03-f8c5-442a-9aba-9e175c24a551",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p redis-server-1 redis-server-2 redis-server-3\n",
    "!rm -f redis-server-1/* redis-server-2/* redis-server-3/*\n",
    "\n",
    "!ln -sf $PWD/redis/src/redis-server redis-server-1/redis-server\n",
    "!ln -sf $PWD/redis/src/redis-server redis-server-2/redis-server\n",
    "!ln -sf $PWD/redis/src/redis-server redis-server-3/redis-server\n",
    "\n",
    "!ln -sf $PWD/test_certs/ca.crt redis-server-1/ca.crt\n",
    "!ln -sf $PWD/test_certs/ca.crt redis-server-2/ca.crt\n",
    "!ln -sf $PWD/test_certs/ca.crt redis-server-3/ca.crt\n",
    "\n",
    "!ln -sf $PWD/test_certs/redis-private.pem redis-server-1/private.pem\n",
    "!ln -sf $PWD/test_certs/redis-private.pem redis-server-2/private.pem\n",
    "!ln -sf $PWD/test_certs/redis-private.pem redis-server-3/private.pem\n",
    "\n",
    "!ln -sf $PWD/test_certs/redis.crt redis-server-1/redis.crt\n",
    "!ln -sf $PWD/test_certs/redis.crt redis-server-2/redis.crt\n",
    "!ln -sf $PWD/test_certs/redis.crt redis-server-3/redis.crt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "53bb9d7b-b6a9-42c1-8e6c-f01c60c68639",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing redis-server-1/redis.conf\n"
     ]
    }
   ],
   "source": [
    "%%writefile redis-server-1/redis.conf\n",
    "daemonize yes\n",
    "port 0\n",
    "cluster-enabled yes\n",
    "cluster-config-file nodes.conf\n",
    "tls-port 7000\n",
    "tls-ca-cert-file ca.crt\n",
    "tls-cert-file redis.crt\n",
    "tls-key-file private.pem\n",
    "tls-cluster yes\n",
    "appendonly no\n",
    "save \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9492279d-94c3-4e83-8496-1a89161f02a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing redis-server-2/redis.conf\n"
     ]
    }
   ],
   "source": [
    "%%writefile redis-server-2/redis.conf\n",
    "daemonize yes\n",
    "port 0\n",
    "cluster-enabled yes\n",
    "cluster-config-file nodes.conf\n",
    "tls-port 7001\n",
    "tls-ca-cert-file ca.crt\n",
    "tls-cert-file redis.crt\n",
    "tls-key-file private.pem\n",
    "tls-cluster yes\n",
    "appendonly no\n",
    "save \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "1446fea9-0ec2-496c-b80a-c5bc2e363c98",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing redis-server-3/redis.conf\n"
     ]
    }
   ],
   "source": [
    "%%writefile redis-server-3/redis.conf\n",
    "daemonize yes\n",
    "port 0\n",
    "cluster-enabled yes\n",
    "cluster-config-file nodes.conf\n",
    "tls-port 7002\n",
    "tls-ca-cert-file ca.crt\n",
    "tls-cert-file redis.crt\n",
    "tls-key-file private.pem\n",
    "tls-cluster yes\n",
    "appendonly no\n",
    "save \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "466bdf2c-e802-4184-bf7b-700c404bb4ee",
   "metadata": {},
   "source": [
    "**Step 3: Form Redis cluster**\n",
    "\n",
    "*WARNING: The following command will shutdown any processes called `redis-cluster` in the current system!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "162ff528-a79c-4b0d-9c68-023217600196",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[29;1m>>> Performing hash slots allocation on 3 nodes...\n",
      "\u001b[0mMaster[0] -> Slots 0 - 5460\n",
      "Master[1] -> Slots 5461 - 10922\n",
      "Master[2] -> Slots 10923 - 16383\n",
      "M: a441806db5506b7600ee8ae794fa01dc31ac83c9 127.0.0.1:7000\n",
      "   slots:[0-5460] (5461 slots) master\n",
      "M: 6fa93392a396aa3c321736234b7eafc86bb1f979 127.0.0.1:7001\n",
      "   slots:[5461-10922] (5462 slots) master\n",
      "M: 8e9cd68cc229fcb568a84d7358011201b4246046 127.0.0.1:7002\n",
      "   slots:[10923-16383] (5461 slots) master\n",
      "\u001b[29;1m>>> Nodes configuration updated\n",
      "\u001b[0m\u001b[29;1m>>> Assign a different config epoch to each node\n",
      "\u001b[0m\u001b[29;1m>>> Sending CLUSTER MEET messages to join the cluster\n",
      "\u001b[0mWaiting for the cluster to join\n",
      "..\n",
      "\u001b[29;1m>>> Performing Cluster Check (using node 127.0.0.1:7000)\n",
      "\u001b[0mM: a441806db5506b7600ee8ae794fa01dc31ac83c9 127.0.0.1:7000\n",
      "   slots:[0-5460] (5461 slots) master\n",
      "M: 8e9cd68cc229fcb568a84d7358011201b4246046 127.0.0.1:7002\n",
      "   slots:[10923-16383] (5461 slots) master\n",
      "M: 6fa93392a396aa3c321736234b7eafc86bb1f979 127.0.0.1:7001\n",
      "   slots:[5461-10922] (5462 slots) master\n",
      "\u001b[32;1m[OK] All nodes agree about slots configuration.\n",
      "\u001b[0m\u001b[29;1m>>> Check for open slots...\n",
      "\u001b[0m\u001b[29;1m>>> Check slots coverage...\n",
      "\u001b[0m\u001b[32;1m[OK] All 16384 slots covered.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# Shutdown existing cluster (if any).\n",
    "!pkill redis-server\n",
    "\n",
    "# Reset configuration and start 3 Redis servers.\n",
    "!cd redis-server-1 && rm -f nodes.conf && ./redis-server redis.conf\n",
    "!cd redis-server-2 && rm -f nodes.conf && ./redis-server redis.conf\n",
    "!cd redis-server-3 && rm -f nodes.conf && ./redis-server redis.conf\n",
    "\n",
    "# Form the cluster.\n",
    "!redis/src/redis-cli \\\n",
    "    --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 \\\n",
    "    --cluster-yes \\\n",
    "    --tls \\\n",
    "    --cacert test_certs/ca.crt \\\n",
    "    --cert test_certs/hugectr.crt \\\n",
    "    --key test_certs/hugectr-private.pem"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ee876e2-0ff9-481d-bf51-8055bd95f926",
   "metadata": {},
   "source": [
    "**Step 4: Run HugeCTR**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f5de0b18-e700-44ef-9ed2-ea8079ed899c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Launching...\n",
      "[HCTR][07:00:07.643][WARNING][RK0][main]: default_value_for_each_table.size() is not equal to the number of embedding tables\n",
      "HPS initialized\n",
      "====================================================HPS Create====================================================\n",
      "[HCTR][07:00:07.643][INFO][RK0][main]: Creating RedisCluster backend...\n",
      "[HCTR][07:00:07.644][INFO][RK0][main]: RedisCluster: Connecting via 127.0.0.1:7000...\n",
      "[HCTR][07:00:07.667][INFO][RK0][main]: Volatile DB: initial cache rate = 1\n",
      "[HCTR][07:00:07.667][INFO][RK0][main]: Volatile DB: cache missed embeddings = 0\n",
      "[HCTR][07:00:07.667][DEBUG][RK0][main]: Created raw model loader in local memory!\n",
      "[HCTR][07:00:07.894][INFO][RK0][main]: Table: hps_et.hps_demo.sparse_embedding1; cached 18488 / 18488 embeddings in volatile database (RedisCluster); load: 18488 / 18446744073709551615 (0.00%).\n",
      "[HCTR][07:00:07.984][INFO][RK0][main]: Table: hps_et.hps_demo.sparse_embedding2; cached 18470 / 18470 embeddings in volatile database (RedisCluster); load: 18470 / 18446744073709551615 (0.00%).\n",
      "[HCTR][07:00:07.984][DEBUG][RK0][main]: Real-time subscribers created!\n",
      "[HCTR][07:00:07.984][INFO][RK0][main]: Creating embedding cache in device 0.\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: Model name: hps_demo\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: Max batch size: 1024\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: Fuse embedding tables: False\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: Number of embedding tables: 2\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: Use GPU embedding cache: True, cache size percentage: 0.500000\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: Embedding cache type: dynamic\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: Use I64 input key: True\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: Configured cache hit rate threshold: 1.000000\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: The size of thread pool: 80\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: The size of worker memory pool: 2\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: The size of refresh memory pool: 1\n",
      "[HCTR][07:00:07.990][INFO][RK0][main]: The refresh percentage : 0.000000\n",
      "[HCTR][07:00:07.995][INFO][RK0][main]: LookupSession i64_input_key: True\n",
      "[HCTR][07:00:07.995][INFO][RK0][main]: Creating lookup session for hps_demo on device: 0\n",
      "[HCTR][07:00:07.998][INFO][RK0][main]: RedisCluster: Awaiting background worker to conclude...\n",
      "[HCTR][07:00:07.998][INFO][RK0][main]: RedisCluster: Disconnecting...\n",
      "-------------------------------------------------------------------------------\n",
      "                         HPS demo without embedding                            \n",
      "-------------------------------------------------------------------------------\n",
      "Ground truth: (1024,) = [0.4895492  0.509022   0.38192913 ... 0.5264926  0.50650454 0.47927693]\n",
      "-------------------------------------------------------------------------------\n",
      "Prediction without embedding: (1024,) = [0.48954916 0.50902206 0.38192907 ... 0.52649266 0.5065045  0.4792769 ]\n",
      "MSE between prediction and ground_truth: 2.3887142264200634e-15\n",
      "-------------------------------------------------------------------------------\n",
      "                           HPS demo with embedding                             \n",
      "-------------------------------------------------------------------------------\n",
      "Ground truth: (1024,) = [0.4895492  0.509022   0.38192913 ... 0.5264926  0.50650454 0.47927693]\n",
      "-------------------------------------------------------------------------------\n",
      "Prediction with embedding: (1024,) = [0.48954916 0.50902206 0.38192907 ... 0.52649266 0.5065045  0.4792769 ]\n",
      "MSE between prediction and ground_truth: 2.3887142264200634e-15\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-09-20 07:00:08.022623188 [W:onnxruntime:, graph.cc:3543 CleanUnusedInitializersAndNodeArgs] Removing initializer 'key_to_indice_hash_all_tables'. It is not used by any node and should be removed from the model.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import time\n",
    "import multiprocessing as mp\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import onnxruntime as ort\n",
    "from hugectr import DatabaseType_t\n",
    "from hugectr.inference import HPS, ParameterServerConfig, InferenceParams, VolatileDatabaseParams\n",
    "\n",
    "slot_size_array = [10000, 10000, 10000, 10000]\n",
    "key_offset = np.insert(np.cumsum(slot_size_array), 0, 0)[:-1]\n",
    "batch_size = 1024\n",
    "\n",
    "print('Launching...')\n",
    "\n",
    "# 1. Configure the HPS hyperparameters.\n",
    "ps_config = ParameterServerConfig(\n",
    "       emb_table_name = {'hps_demo': ['sparse_embedding1', 'sparse_embedding2']},\n",
    "       embedding_vec_size = {'hps_demo': [16, 32]},\n",
    "       max_feature_num_per_sample_per_emb_table = {'hps_demo': [2, 2]},\n",
    "       inference_params_array = [\n",
    "          InferenceParams(\n",
    "            model_name = 'hps_demo',\n",
    "            max_batchsize = batch_size,\n",
    "            hit_rate_threshold = 1.0,\n",
    "            dense_model_file = '',\n",
    "            sparse_model_files = ['hps_demo0_sparse_1000.model', 'hps_demo1_sparse_1000.model'],\n",
    "            deployed_devices = [0],\n",
    "            use_gpu_embedding_cache = True,\n",
    "            cache_size_percentage = 0.5,\n",
    "            i64_input_key = True)\n",
    "       ],\n",
    "       volatile_db = VolatileDatabaseParams(\n",
    "            DatabaseType_t.redis_cluster,\n",
    "            address = '127.0.0.1:7000',\n",
    "            num_partitions = 15,\n",
    "            num_node_connections = 5,\n",
    "            enable_tls = True,\n",
    "            tls_ca_certificate = 'test_certs/ca.crt',\n",
    "            tls_client_certificate = 'test_certs/hugectr.crt',\n",
    "            tls_client_key = 'test_certs/hugectr-private.pem',\n",
    "            tls_server_name_identification = 'redis.localhost',\n",
    "       ))\n",
    "\n",
    "# 2. Initialize the HPS object.\n",
    "hps = HPS(ps_config)\n",
    "print('HPS initialized')\n",
    "\n",
    "# 3. Load query data.\n",
    "df = pd.read_parquet('data_parquet/val/gen_0.parquet')\n",
    "dense_input_columns = df.columns[1:11]\n",
    "cat_input1_columns = df.columns[11:13]\n",
    "cat_input2_columns = df.columns[13:15]\n",
    "dense_input = df[dense_input_columns].loc[0:batch_size-1].to_numpy(dtype=np.float32)\n",
    "cat_input1 = (df[cat_input1_columns].loc[0:batch_size-1].to_numpy(dtype=np.int64) + key_offset[0:2]).reshape((batch_size, 2, 1))\n",
    "cat_input2 = (df[cat_input2_columns].loc[0:batch_size-1].to_numpy(dtype=np.int64) + key_offset[2:4]).reshape((batch_size, 2, 1))\n",
    "\n",
    "# 4. Make inference from the HPS object and the ONNX inference session of `hps_demo_without_embedding.onnx`.\n",
    "embedding1 = hps.lookup(cat_input1.flatten(), 'hps_demo', 0).reshape(batch_size, 2, 16)\n",
    "embedding2 = hps.lookup(cat_input2.flatten(), 'hps_demo', 1).reshape(batch_size, 2, 32)\n",
    "sess = ort.InferenceSession('hps_demo_without_embedding.onnx')\n",
    "res = sess.run(output_names=[sess.get_outputs()[0].name],\n",
    "               input_feed={sess.get_inputs()[0].name: dense_input,\n",
    "               sess.get_inputs()[1].name: embedding1,\n",
    "               sess.get_inputs()[2].name: embedding2})\n",
    "pred = res[0].flatten()\n",
    "\n",
    "# 5. Check the correctness by comparing with dumped evaluation results.\n",
    "ground_truth = np.load(\"ground_truth.npy\").flatten()\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print('                         HPS demo without embedding                            ')\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print(f'Ground truth: {ground_truth.shape} = {ground_truth}')\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print(f'Prediction without embedding: {pred.shape} = {pred}')\n",
    "\n",
    "diff = pred - ground_truth\n",
    "mse = np.mean(diff * diff)\n",
    "print(f'MSE between prediction and ground_truth: {mse}')\n",
    "\n",
    "# 6. Make inference with the ONNX inference session of `hps_demo_with_embedding.onnx` (double check).\n",
    "sess_ref = ort.InferenceSession('hps_demo_with_embedding.onnx')\n",
    "res_ref = sess_ref.run(output_names=[sess_ref.get_outputs()[0].name],\n",
    "               input_feed={sess_ref.get_inputs()[0].name: dense_input,\n",
    "               sess_ref.get_inputs()[1].name: cat_input1,\n",
    "               sess_ref.get_inputs()[2].name: cat_input2})\n",
    "pred_ref = res_ref[0].flatten()\n",
    "\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print('                           HPS demo with embedding                             ')\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print(f'Ground truth: {ground_truth.shape} = {ground_truth}')\n",
    "print('-------------------------------------------------------------------------------')\n",
    "print(f'Prediction with embedding: {pred_ref.shape} = {pred_ref}')\n",
    "\n",
    "diff_ref = pred_ref.flatten() - ground_truth\n",
    "mse_ref = np.mean(diff_ref * diff_ref)\n",
    "print(f'MSE between prediction and ground_truth: {mse_ref}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dba3040-8965-4dd1-8988-c5f84023f315",
   "metadata": {},
   "source": [
    "**Step 5: Shutdown Redis cluster**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8790ab66-2379-48d6-b8de-53ac4aa881bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pkill redis-server"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}