examples/notebook-demo/protobag-demo-full.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Protobag Demo\n",
    "\n",
    "This notebook shows:\n",
    "  * [How to write and read Protobuf messsages to a `protobag` zip archive](#write_read)\n",
    "  * [How to convert protobag zip archives to and from Pandas Dataframes and Parquet tables](#dataframes)\n",
    "  \n",
    "If you only have 30 seconds, hit \"run all cells\" and try scrolling to these sections:\n",
    " * [Writing Plain Protobuf Messages](#write_plain)\n",
    " * [Reading Plain Protobuf Messages](#read_plain)\n",
    " * [Converting a Protobag to a Pandas Dataframe](#dataframes) (longer, but hopefully easy to skim)\n",
    "\n",
    "Protobag archives are just zip archives that contain serialized Protobuf messages as files. We hope this notebook shows how `protobag` may prove more useful than working with Protobuf + Zip archives directly.\n",
    "\n",
    "## Environment Set-up\n",
    "\n",
    "To run this notebook locally, try using the protobag dockerized environment from a clone of the protobag repo:\n",
    "```shell\n",
    "my-machine $ ./pb-dev --build-env\n",
    "my-machine $ ./pb-dev --shell\n",
    "in-docker % jupyter notebook --allow-root examples/notebook-demo\n",
    "```\n",
    "\n",
    "**Google Colab** You can also [run this notebook in Google Colab](https://colab.sandbox.google.com/github/StandardCyborg/protobag/blob/master/examples/notebook-demo/protobag-demo-full.ipynb). In the Colab environment, you'll need to install `protobag` and some other dependencies. Running the cell below will take care of that for you. You might need to restart the runtime (Use the menu option: Runtime > Restart runtime ...) in order for Colab to recognize the new modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We're running in the dockerized environment! We can simply add protobag to the PYTHONPATH\n",
      "running pytest\n",
      "running egg_info\n",
      "writing protobag.egg-info/PKG-INFO\n",
      "writing dependency_links to protobag.egg-info/dependency_links.txt\n",
      "writing requirements to protobag.egg-info/requires.txt\n",
      "writing top-level names to protobag.egg-info/top_level.txt\n",
      "reading manifest file 'protobag.egg-info/SOURCES.txt'\n",
      "writing manifest file 'protobag.egg-info/SOURCES.txt'\n",
      "running build_ext\n",
      "cmake version 3.10.2\n",
      "\n",
      "CMake suite maintained and supported by Kitware (kitware.com/cmake).\n",
      "-- The C compiler identification is Clang 8.0.0\n",
      "-- The CXX compiler identification is Clang 8.0.0\n",
      "-- Check for working C compiler: /usr/bin/cc\n",
      "-- Check for working C compiler: /usr/bin/cc -- works\n",
      "-- Detecting C compiler ABI info\n",
      "-- Detecting C compiler ABI info - done\n",
      "-- Detecting C compile features\n",
      "-- Detecting C compile features - done\n",
      "-- Check for working CXX compiler: /usr/bin/c++\n",
      "-- Check for working CXX compiler: /usr/bin/c++ -- works\n",
      "-- Detecting CXX compiler ABI info\n",
      "-- Detecting CXX compiler ABI info - done\n",
      "-- Detecting CXX compile features\n",
      "-- Detecting CXX compile features - done\n",
      "-- Found GTest: /opt/gtest/lib/libgtest.a  \n",
      "-- Found Threads: TRUE  \n",
      "-- Found Protobuf: /usr/local/lib/libprotobuf.a;-lpthread (found version \"3.11.3\") \n",
      "-- Found LibArchive: /usr/local/lib/libarchive.so (found version \"3.4.2\") \n",
      "-- Found PythonInterp: /usr/bin/python3 (found version \"3.6.9\") \n",
      "-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so\n",
      "-- Performing Test HAS_FLTO\n",
      "-- Performing Test HAS_FLTO - Success\n",
      "-- LTO enabled\n",
      "-- Configuring done\n",
      "-- Generating done\n",
      "-- Build files have been written to: /opt/protobag/python/build/temp.linux-x86_64-3.6\n",
      "\u001b[35m\u001b[1mScanning dependencies of target protobag_native\u001b[0m\n",
      "[  6%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/ArchiveUtil.cpp.o\u001b[0m\n",
      "[ 18%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/BagIndexBuilder.cpp.o\u001b[0m\n",
      "[ 31%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/Entry.cpp.o\u001b[0m\n",
      "[ 43%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/ReadSession.cpp.o\u001b[0m\n",
      "[ 43%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/Utils/Tempfile.cpp.o\u001b[0m\n",
      "[ 43%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/Protobag.cpp.o\u001b[0m\n",
      "[ 43%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/Utils/PBUtils.cpp.o\u001b[0m\n",
      "[ 50%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/WriteSession.cpp.o\u001b[0m\n",
      "[ 62%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/Utils/TimeSync.cpp.o\u001b[0m\n",
      "[ 68%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/archive/Archive.cpp.o\u001b[0m\n",
      "[ 68%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/archive/DirectoryArchive.cpp.o\u001b[0m\n",
      "[ 75%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/archive/LibArchiveArchive.cpp.o\u001b[0m\n",
      "[ 87%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag/archive/MemoryArchive.cpp.o\u001b[0m\n",
      "[ 87%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag/protobag_msg/ProtobagMsg.pb.cc.o\u001b[0m\n",
      "[ 93%] \u001b[32mBuilding CXX object CMakeFiles/protobag_native.dir/protobag_native/protobag_native.cpp.o\u001b[0m\n",
      "[100%] \u001b[32m\u001b[1mLinking CXX shared library /opt/protobag/python/protobag/protobag_native.cpython-36m-x86_64-linux-gnu.so\u001b[0m\n",
      "[100%] Built target protobag_native\n",
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform linux -- Python 3.6.9, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3\n",
      "cachedir: .pytest_cache\n",
      "rootdir: /opt/protobag/python, configfile: setup.cfg\n",
      "collected 18 items                                                             \u001b[0m\n",
      "\n",
      "protobag_test/test_protobag.py::test_type_url \u001b[32mPASSED\u001b[0m\u001b[32m                     [  5%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_to_pb_timestamp \u001b[32mPASSED\u001b[0m\u001b[32m              [ 11%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_to_sec_nanos \u001b[32mPASSED\u001b[0m\u001b[32m                 [ 16%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_to_topic_time \u001b[32mPASSED\u001b[0m\u001b[32m                [ 22%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_build_fds_for_msg \u001b[32mPASSED\u001b[0m\u001b[32m            [ 27%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_msg_entry_print \u001b[32mPASSED\u001b[0m\u001b[32m              [ 33%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_raw_entry_print \u001b[32mPASSED\u001b[0m\u001b[32m              [ 38%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_stamped_entry_print \u001b[32mPASSED\u001b[0m\u001b[32m          [ 44%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_typed_bytes \u001b[32mPASSED\u001b[0m\u001b[32m                  [ 50%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_serdes_msg_from_typed_bytes_empty \u001b[32mPASSED\u001b[0m\u001b[32m [ 55%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_serdes_msg_from_typed_bytes_default_serdes \u001b[32mPASSED\u001b[0m\u001b[32m [ 61%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_serdes_msg_from_typed_bytes_user_registered \u001b[32mPASSED\u001b[0m\u001b[32m [ 66%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_serdes_msg_from_typed_bytes_dynamic_decode \u001b[32mPASSED\u001b[0m\u001b[32m [ 72%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_write_read_msg \u001b[32mPASSED\u001b[0m\u001b[32m               [ 77%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_write_read_stamped_msg \u001b[32mPASSED\u001b[0m\u001b[32m       [ 83%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_write_read_raw \u001b[32mPASSED\u001b[0m\u001b[32m               [ 88%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_entry_to_dict_row \u001b[32mPASSED\u001b[0m\u001b[32m            [ 94%]\u001b[0m\n",
      "protobag_test/test_protobag.py::test_dict_row_to_entry \u001b[32mPASSED\u001b[0m\u001b[32m            [100%]\u001b[0m\n",
      "\n",
      "============================== slowest durations ===============================\n",
      "0.14s call     protobag_test/test_protobag.py::test_write_read_stamped_msg\n",
      "0.08s call     protobag_test/test_protobag.py::test_write_read_msg\n",
      "0.01s call     protobag_test/test_protobag.py::test_serdes_msg_from_typed_bytes_dynamic_decode\n",
      "0.01s call     protobag_test/test_protobag.py::test_build_fds_for_msg\n",
      "\n",
      "(50 durations < 0.005s hidden.  Use -vv to show these durations.)\n",
      "\u001b[32m============================== \u001b[32m\u001b[1m18 passed\u001b[0m\u001b[32m in 0.54s\u001b[0m\u001b[32m ==============================\u001b[0m\n",
      "Protobag added to PYTHONPATH\n",
      "Using protobag version 0.0.3 at /opt/protobag/python/protobag/__init__.py\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "if os.path.exists('/opt/protobag'):\n",
    "    print(\"We're running in the dockerized environment! We can simply add protobag to the PYTHONPATH\")\n",
    "    if not os.path.exists('/opt/protobag/python/protobag/protobag_native.cpython-36m-x86_64-linux-gnu.so'):\n",
    "        # We need to build the `protobag_native` module.  The easiest way is to:\n",
    "        !cd /opt/protobag/python && python3 setup.py test\n",
    "    sys.path.append('/opt/protobag/python/')\n",
    "    print(\"Protobag added to PYTHONPATH\")\n",
    "elif 'google.colab' in sys.modules:\n",
    "    try:\n",
    "      import protobag\n",
    "    except:\n",
    "      print(\"Installing protobag\")\n",
    "      !pip3 install protobag-0.0.3-cp36-cp36m-linux_x86_64.whl\n",
    "    print(\"Installing libarchive; Colab linux does not yet have the version we need\")\n",
    "    !ls /usr/local/lib/libarchive.so.17 || (cd /tmp && \\\n",
    "      wget https://github.com/libarchive/libarchive/archive/v3.4.2.tar.gz && \\\n",
    "      tar xfz v3.4.2.tar.gz && \\\n",
    "      (rm -rf /opt/libarchive || true) && \\\n",
    "      mv libarchive-3.4.2 /opt/libarchive && \\\n",
    "      cd /opt/libarchive && \\\n",
    "      mkdir -p build && cd build && \\\n",
    "      cmake .. && \\\n",
    "      make -j `nproc` && \\\n",
    "      make install)\n",
    "    print(\"Installing libc++\")\n",
    "    !apt-get install -y libc++-8-dev libc++abi-8-dev || echo \"ignoring 'libmkldnn.so.0 is not a symbolic link' issue\"\n",
    "\n",
    "import protobag\n",
    "print(\"Using protobag version %s at %s\" % (protobag.__version__, protobag.__file__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our Protobuf Messages\n",
    "\n",
    "*Note: If you're not super familiar with Protobuf, you might want to check out [their address book data tutorial](https://developers.google.com/protocol-buffers/docs/pythontutorial)*.\n",
    "\n",
    "Suppose we've developed a cool game called Dino Hunters where people run around on a deserted island and try to capture wild dinosaurs.  We're using Protobuf to persist data about the `DinoHunter` characters in our game.  Furthermore, we want to use Protobuf to log the 2D `Position`s of our hunters as the run around and hunt dinos.  Our Protobuf message schema is as follows:\n",
    "\n",
    "```protobuf\n",
    "syntax = \"proto3\";\n",
    "\n",
    "package my_messages;\n",
    "\n",
    "message DinoHunter {\n",
    "  string first_name = 1;\n",
    "  int32 id = 2;\n",
    "  \n",
    "  // Misc attributes of this hunter\n",
    "  map<string, string> attribs = 3;\n",
    "\n",
    "  enum DinoType {\n",
    "    IDK = 0;\n",
    "    VEGGIESAURUS = 1;\n",
    "    MEATIESAURUS = 2;\n",
    "    PEOPLEEATINGSAURUS = 3;\n",
    "  }\n",
    "\n",
    "  message Dino {\n",
    "    string name = 1;\n",
    "    DinoType type = 2;\n",
    "  }\n",
    "\n",
    "  // Dinos that this hunter has captured\n",
    "  repeated Dino dinos = 4;\n",
    "}\n",
    "\n",
    "message Position {\n",
    "  float x = 1;\n",
    "  float y = 2;\n",
    "}\n",
    "```\n",
    "\n",
    "We now need the `protoc`-generated Python code in order to use these messages.  For convenience, we'll just download a copy from the `protobag` repository:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('MyMessages_pb2.py'):\n",
    "    print(\"Downloading MyMessages_pb2.py from Github\")\n",
    "    !wget https://raw.githubusercontent.com/StandardCyborg/protobag/blob/master/examples/notebook-demo/MyMessages_pb2.py\n",
    "\n",
    "from MyMessages_pb2 import DinoHunter\n",
    "from MyMessages_pb2 import Position\n",
    "\n",
    "## Note: you can prove to yourself that the downloaded file matches the schema above using the code below:\n",
    "# import MyMessages_pb2\n",
    "# from google.protobuf.descriptor_pb2 import FileDescriptorProto\n",
    "# fd = FileDescriptorProto()\n",
    "# MyMessages_pb2.DESCRIPTOR.CopyToProto(fd)\n",
    "# print(fd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK! Let's create some dino hunters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first_name: \"py_max\"\n",
      "id: 1\n",
      "dinos {\n",
      "  name: \"py_nibbles\"\n",
      "  type: PEOPLEEATINGSAURUS\n",
      "}\n",
      "\n",
      "first_name: \"py_lara\"\n",
      "id: 2\n",
      "dinos {\n",
      "  name: \"py_bites\"\n",
      "  type: PEOPLEEATINGSAURUS\n",
      "}\n",
      "dinos {\n",
      "  name: \"py_stinky\"\n",
      "  type: VEGGIESAURUS\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "max_hunter = DinoHunter(\n",
    "      first_name='py_max',\n",
    "      id=1,\n",
    "      dinos=[\n",
    "        {'name': 'py_nibbles', 'type': DinoHunter.PEOPLEEATINGSAURUS},\n",
    "      ])\n",
    "print(max_hunter)\n",
    "\n",
    "lara_hunter = DinoHunter(\n",
    "      first_name='py_lara',\n",
    "      id=2,\n",
    "      dinos=[\n",
    "        {'name': 'py_bites', 'type': DinoHunter.PEOPLEEATINGSAURUS},\n",
    "        {'name': 'py_stinky', 'type': DinoHunter.VEGGIESAURUS},\n",
    "      ])\n",
    "\n",
    "print(lara_hunter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='write_read'></a>\n",
    "## Writing and Reading Protobuf messages to a `protobag` archive\n",
    "<a id='write_plain'></a>\n",
    "### Plain Messages (`protobag.MessageEntry`)\n",
    "\n",
    "`protobag` archives are just zip archives that contain serialized Protobuf messages as files. (Tar and other formats are also supported via [libarchive](https://github.com/libarchive/libarchive)).  So `protobag` offers a simple API for **writing** messages to an archive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "bag = protobag.Protobag(path='example.zip')\n",
    "writer = bag.create_writer()\n",
    "writer.write_msg(\"hunters/py_max\", max_hunter)\n",
    "writer.write_msg(\"hunters/py_lara\", lara_hunter)\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can verify that the above just wrote a zip archive for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  example.zip\r\n",
      "  Length      Date    Time    Name\r\n",
      "---------  ---------- -----   ----\r\n",
      "       72  1980-01-01 00:00   hunters/py_max\r\n",
      "       86  1980-01-01 00:00   hunters/py_lara\r\n",
      "     8691  1980-01-01 00:00   /_protobag_index/bag_index/1597696075.0.stampedmsg.protobin\r\n",
      "---------                     -------\r\n",
      "     8849                     3 files\r\n"
     ]
    }
   ],
   "source": [
    "!which unzip > /dev/null || (apt-get update && apt-get install unzip)\n",
    "!unzip -l example.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm, what's that `/_protobag_index/bag_index/xxxxx.stampedmsg.protobin` file?\n",
    "\n",
    "By default, `protobag` not only saves those messages but also **indexes Protobuf message descriptors** so that your `protobag` readers don't need your proto schemas to decode your messages.  (You can also disable this indexing if you wish.  For further discussion, see [the root README.md](https://github.com/StandardCyborg/protobag#protobag-indexes-protobuf-message-descriptors) ).\n",
    "\n",
    "<a id='read_plain'></a>\n",
    "To **read** specific messages from a `protobag` archive, you can use this simple API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MessageEntry:\n",
      "  entryname: hunters/py_max\n",
      "  type_url: type.googleapis.com/my_messages.DinoHunter\n",
      "  has serdes: True\n",
      "  has descriptor_data: False\n",
      "  msg:\n",
      "first_name: \"py_max\"\n",
      "id: 1\n",
      "dinos {\n",
      "  name: \"py_nibbles\"\n",
      "  type: PEOPLEEATINGSAURUS\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "bag = protobag.Protobag(\n",
    "        path='example.zip',\n",
    "        \n",
    "        # Tell protobag to use our protoc-generated python code:\n",
    "        msg_classes=(DinoHunter, Position))\n",
    "entry = bag.get_entry(\"hunters/py_max\")\n",
    "print(entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time-Series Data (`protobag.StampedEntry`)\n",
    "`protobag` features a **topic-timestamp-message** API for recording time-series data.  This API is modeled after [`rosbag`](http://wiki.ros.org/rosbag), [LCM log files](https://lcm-proj.github.io/log_file_format.html) (where topics are called \"channels\"), and your favorite message bus systems like [Kafka](https://kafka.apache.org/) or [AWS SQS](https://aws.amazon.com/sqs/).  Each **topic** contains Protobuf messages of a single type, and each message has a nanosecond-precision timestamp (using the `google.protobuf.Timestamp` object, which has built-in conversion to other timestamp datastructures like Python's `datetime`).  \n",
    "\n",
    "Protobag has special handling for these timestamped entries:\n",
    "  * For writing:\n",
    "      * Topics organized into archive \"folders\" and filenames are chosen automatically.\n",
    "      * Protobag indexes Protobuf Message Descriptors as described above.\n",
    "      * Protobag indexes the message timestamps for efficient time-ordered playback.\n",
    "  * For reading:\n",
    "     * Protobag offers a simple [Selection](https://github.com/StandardCyborg/protobag/blob/master/c%2B%2B/protobag/protobag_msg/ProtobagMsg.proto) API for reading specific sets of topics, time ranges, or even just individual events.\n",
    "     * Protobag offers a [TimeSync](https://github.com/StandardCyborg/protobag/blob/master/c%2B%2B/protobag/protobag/Utils/TimeSync.hpp) for synchronizing topics that have messages recorded at different rates.  \n",
    "  \n",
    "\n",
    "Using our Dino Hunters example, we'll log (**write**) the 2D positions of a dino and a hunter during a chase scene:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  example.zip\r\n",
      "  Length      Date    Time    Name\r\n",
      "---------  ---------- -----   ----\r\n",
      "      100  1980-01-01 00:00   positions/lara/0.0.stampedmsg.protobin\r\n",
      "      109  1980-01-01 00:00   positions/trex/0.1000000.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/lara/1.0.stampedmsg.protobin\r\n",
      "      111  1980-01-01 00:00   positions/trex/1.1000000.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/lara/2.0.stampedmsg.protobin\r\n",
      "      111  1980-01-01 00:00   positions/trex/2.1000000.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/lara/3.0.stampedmsg.protobin\r\n",
      "      111  1980-01-01 00:00   positions/trex/3.1000000.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/lara/4.0.stampedmsg.protobin\r\n",
      "      111  1980-01-01 00:00   positions/trex/4.1000000.stampedmsg.protobin\r\n",
      "    10707  1980-01-01 00:00   /_protobag_index/bag_index/1596831107.0.stampedmsg.protobin\r\n",
      "---------                     -------\r\n",
      "    11788                     11 files\r\n"
     ]
    }
   ],
   "source": [
    "bag = protobag.Protobag(path='example.zip')\n",
    "writer = bag.create_writer()\n",
    "for t in range(5):\n",
    "    lara_pos = Position(x=t, y=t+1)\n",
    "    writer.write_stamped_msg(\"positions/lara\", lara_pos, t_sec=t, t_nanos=0)\n",
    "\n",
    "    trex_pos = Position(x=t+2, y=t+3)\n",
    "    writer.write_stamped_msg(\"positions/trex\", trex_pos, t_sec=t, t_nanos=int(1e6))\n",
    "        # Note: trex positions are slightly out-of-sync with those of lara by 1ms\n",
    "writer.close()\n",
    "!unzip -l example.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now **read** them using the `protobag.SelectionBuilder` helper tool:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read just the positions of trex:\n",
      "Time: 1970-01-01 00:00:00.001000 Position: 2.0 3.0\n",
      "Time: 1970-01-01 00:00:01.001000 Position: 3.0 4.0\n",
      "Time: 1970-01-01 00:00:02.001000 Position: 4.0 5.0\n",
      "Time: 1970-01-01 00:00:03.001000 Position: 5.0 6.0\n",
      "Time: 1970-01-01 00:00:04.001000 Position: 6.0 7.0\n",
      "\n",
      "\n",
      "Read *all* timeseries data:\n",
      "Topic: positions/lara Time: 1970-01-01 00:00:00 Position: 0.0 1.0\n",
      "Topic: positions/trex Time: 1970-01-01 00:00:00.001000 Position: 2.0 3.0\n",
      "Topic: positions/lara Time: 1970-01-01 00:00:01 Position: 1.0 2.0\n",
      "Topic: positions/trex Time: 1970-01-01 00:00:01.001000 Position: 3.0 4.0\n",
      "Topic: positions/lara Time: 1970-01-01 00:00:02 Position: 2.0 3.0\n",
      "Topic: positions/trex Time: 1970-01-01 00:00:02.001000 Position: 4.0 5.0\n",
      "Topic: positions/lara Time: 1970-01-01 00:00:03 Position: 3.0 4.0\n",
      "Topic: positions/trex Time: 1970-01-01 00:00:03.001000 Position: 5.0 6.0\n",
      "Topic: positions/lara Time: 1970-01-01 00:00:04 Position: 4.0 5.0\n",
      "Topic: positions/trex Time: 1970-01-01 00:00:04.001000 Position: 6.0 7.0\n"
     ]
    }
   ],
   "source": [
    "bag = protobag.Protobag(\n",
    "        path='example.zip',\n",
    "        \n",
    "        # Tell protobag to use our protoc-generated python code:\n",
    "        msg_classes=(DinoHunter, Position))\n",
    "\n",
    "print(\"Read just the positions of trex:\")\n",
    "sel_trex = protobag.SelectionBuilder.select_window(topics=[\"positions/trex\"])\n",
    "for entry in bag.iter_entries(selection=sel_trex):\n",
    "    print(\"Time: %s Position: %s %s\" % (entry.timestamp.ToDatetime(), entry.msg.x, entry.msg.y))\n",
    "print()\n",
    "print()\n",
    "    \n",
    "print(\"Read *all* timeseries data:\")\n",
    "sel_all_time_series = protobag.SelectionBuilder.select_window_all()\n",
    "for entry in bag.iter_entries(selection=sel_all_time_series):\n",
    "    print(\"Topic: %s Time: %s Position: %s %s\" % (entry.topic, entry.timestamp.ToDatetime(), entry.msg.x, entry.msg.y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Synchronizing Time-Series Data\n",
    "Notice in the example that the positions of `lara` and `trex` are recorded at *almost* the same time, but not exactly.  But what if we wanted to synchronize that data?\n",
    "\n",
    "When handling time series data, it's often desirable to synchronize two or more topics (or series) of data and to process 'bundles' of messages with nearly (or exactly) the same timestamps.  There are a variety of algorithms and techniques for computing such bundlings.  Protobag includes a [MaxSlopTimeSync](https://github.com/StandardCyborg/protobag/blob/293ae8ee82510f3d207d22317997d5f432827e14/c%2B%2B/protobag/protobag/Utils/TimeSync.hpp#L55) utility that offers some robustness for a variety of uses:\n",
    "  * If some topics have 'missing' messages (e.g. the message was dropped during recording), then `MaxSlopTimeSync` will ignore those.  The user provides as parameters a 'max_slop' (a time duration) and a 'max_queue_size' (a number of messages to buffer) that controls how to handle recording gaps.  See the [MaxSlopTimeSync](https://github.com/StandardCyborg/protobag/blob/293ae8ee82510f3d207d22317997d5f432827e14/c%2B%2B/protobag/protobag/Utils/TimeSync.hpp#L55) docs for more detail.\n",
    "  * If topics have messages recorded at the same rates and at the same times (e.g. two sensors are configured to record with the exact same timestamps), then `MaxSlopTimeSync` will provide exactly-synchronized bundles.\n",
    "  * If topics have messages recorded at somewhat different times (e.g. two sensors both record at 5Hz, but one records 50ms before the other), then `MaxSlopTimeSync` will find the nearest matches.\n",
    "\n",
    "FMI consult the unit tests [for MaxSlopTimeSync](https://github.com/StandardCyborg/protobag/blob/293ae8ee82510f3d207d22317997d5f432827e14/c%2B%2B/protobag_test/protobag/Utils/TimeSyncTest.cpp#L71) and [for IterProducts](https://github.com/StandardCyborg/protobag/blob/293ae8ee82510f3d207d22317997d5f432827e14/c%2B%2B/protobag_test/protobag/Utils/IterProductsTest.cpp#L31).\n",
    "\n",
    "In the example below, we'll synchronize the data for `lara` and `trex` using the `MaxSlopTimeSync` algorithm in order to compute their relative distances at each timestep:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bundle: 1970-01-01 00:00:00.001000 | Distance trex <-> lara: 2.8284271247461903 | Time b/t entries: 0.001 sec\n",
      "Bundle: 1970-01-01 00:00:01.001000 | Distance trex <-> lara: 2.8284271247461903 | Time b/t entries: 0.001 sec\n",
      "Bundle: 1970-01-01 00:00:02.001000 | Distance trex <-> lara: 2.8284271247461903 | Time b/t entries: 0.001 sec\n",
      "Bundle: 1970-01-01 00:00:03.001000 | Distance trex <-> lara: 2.8284271247461903 | Time b/t entries: 0.001 sec\n",
      "Bundle: 1970-01-01 00:00:04.001000 | Distance trex <-> lara: 2.8284271247461903 | Time b/t entries: 0.001 sec\n"
     ]
    }
   ],
   "source": [
    "bag = protobag.Protobag(path='example.zip', msg_classes=(Position,))\n",
    "\n",
    "topics = [\"positions/trex\", \"positions/lara\"]\n",
    "\n",
    "# Select the topics we want\n",
    "sel = protobag.SelectionBuilder.select_window(topics=topics)\n",
    "\n",
    "# Configure synchronization\n",
    "from protobag.protobag_native import MaxSlopTimeSyncSpec\n",
    "spec = MaxSlopTimeSyncSpec()\n",
    "spec.topics = topics\n",
    "spec.set_max_slop(seconds=2, nanos=0) # Both topics are about 1Hz; we'll be sloppy and provide a 2-second window\n",
    "\n",
    "for bundle in bag.iter_entries(selection=sel, sync_using_max_slop=spec):\n",
    "    topic_to_entry = dict((entry.topic, entry) for entry in bundle)\n",
    "    lara_entry = topic_to_entry[\"positions/lara\"]\n",
    "    trex_entry = topic_to_entry[\"positions/trex\"]\n",
    "    \n",
    "    delta_t = trex_entry.timestamp.ToDatetime() - lara_entry.timestamp.ToDatetime()\n",
    "    \n",
    "    import numpy as np\n",
    "    lara_pos = np.array([lara_entry.msg.x, lara_entry.msg.y])\n",
    "    trex_pos = np.array([trex_entry.msg.x, trex_entry.msg.y])\n",
    "    distance = np.linalg.norm(lara_pos - trex_pos)\n",
    "    print(\"Bundle: %s | Distance trex <-> lara: %s | Time b/t entries: %s sec\" % (\n",
    "        trex_entry.timestamp.ToDatetime(), distance, delta_t.total_seconds()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Raw Data (`protobag.RawEntry`)\n",
    "We can also just put raw data like text files or images into our protobag archive because it's just a zip file.  You might need this API if you have consumers of your data who don't have Protobuf or `protobag` available.   Note that the C++ `protobag` API even includes an [`ArchiveUtil.hpp` module](https://github.com/StandardCyborg/protobag/blob/master/c++/protobag/protobag/ArchiveUtil.hpp) that has helper functions for common archive operations like zipping a directory, unarchiving a tar file, etc.  (In python, one would probably use the excellent `zipfile` or `tarfile` libraries, but those don't exist in C++).  The raw write API makes `protobag` skip all indexing and type-tracking activity.\n",
    "\n",
    "To **write** data, just use the `write_raw` API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  example.zip\n",
      "  Length      Date    Time    Name\n",
      "---------  ---------- -----   ----\n",
      "       19  1980-01-01 00:00   raw_data\n",
      "      129  1980-01-01 00:00   /_protobag_index/bag_index/1596831108.0.stampedmsg.protobin\n",
      "---------                     -------\n",
      "      148                     2 files\n",
      "Reading the raw data using unzip:\n",
      "\u0012\u0011i am a raw string"
     ]
    }
   ],
   "source": [
    "bag = protobag.Protobag(path='example.zip')\n",
    "writer = bag.create_writer()\n",
    "\n",
    "s = b\"i am a raw string\"\n",
    "writer.write_raw(\"raw_data\", s)\n",
    "\n",
    "writer.close()\n",
    "\n",
    "!unzip -l example.zip\n",
    "!echo \"Reading the raw data using unzip:\"\n",
    "!unzip -p example.zip raw_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To **read** raw data, you can use the same API we used for regular messages (but `protobag` will do no deserialization):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RawEntry:\n",
      "  entryname: raw_data\n",
      "  raw_bytes: i am a raw string ... (17 bytes)\n"
     ]
    }
   ],
   "source": [
    "bag = protobag.Protobag(path='example.zip')\n",
    "entry = bag.get_entry(\"raw_data\")\n",
    "print(entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='dataframes'></a>\n",
    "## Converting `protobag` archives to DataFrames\n",
    "\n",
    "For various data mining tasks, dealing with files and archives can be cumbersome.  Suppose we wanted to:\n",
    " * Load our `protobag` data into a `DataFrame` (a database table)\n",
    " * Compute some quick statistics of our data using python code\n",
    " * Query our `protobag` contents using SQL\n",
    "\n",
    "In each of these above cases, we might think of each entry of our `protobag` archive as a row in a table.  (You might also skip some entries, or want a single entry to map to several rows).  Protobag provides a `DictRowEntry` utility to help aid this usage.\n",
    "\n",
    "Protobuf 3.x introduced mature `json` support, and in python, `google.protobuf.json_format` also provides a means to convert Protobuf messages to and from *python dicts*.  `DictRowEntry` leverages this feature to help you translate between `protobag` archive entries and table rows using whatever database or DataFrame tool you have at hand.\n",
    "\n",
    "\n",
    "### From `protobag` to `pandas` and back\n",
    "In this section, we'll show how to use `DictRowEntry` to create a `pandas.DataFrame` from our `protobag` archive.  We can also convert from `DataFrame` to `protobag` if desired.\n",
    "\n",
    "First we need to have `pandas` available, as well as an example `protobag` archive:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages\r\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas)\r\n",
      "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from pandas)\r\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas)\r\n",
      "Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.6.1->pandas)\r\n"
     ]
    }
   ],
   "source": [
    "!pip3 install pandas\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  pandas_example.zip\r\n",
      "  Length      Date    Time    Name\r\n",
      "---------  ---------- -----   ----\r\n",
      "       72  1980-01-01 00:00   hunters/py_max\r\n",
      "       86  1980-01-01 00:00   hunters/py_lara\r\n",
      "      100  1980-01-01 00:00   positions/lara/0.0.stampedmsg.protobin\r\n",
      "      105  1980-01-01 00:00   positions/trex/0.0.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/lara/1.0.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/trex/1.0.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/lara/2.0.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/trex/2.0.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/lara/3.0.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/trex/3.0.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/lara/4.0.stampedmsg.protobin\r\n",
      "      107  1980-01-01 00:00   positions/trex/4.0.stampedmsg.protobin\r\n",
      "       19  1980-01-01 00:00   raw_data\r\n",
      "    10675  1980-01-01 00:00   /_protobag_index/bag_index/1596831111.0.stampedmsg.protobin\r\n",
      "---------                     -------\r\n",
      "    11913                     14 files\r\n"
     ]
    }
   ],
   "source": [
    "bag = protobag.Protobag(path='pandas_example.zip')\n",
    "writer = bag.create_writer()\n",
    "\n",
    "# Include some protobuf messages in the example\n",
    "writer.write_msg(\"hunters/py_max\", max_hunter)\n",
    "writer.write_msg(\"hunters/py_lara\", lara_hunter)\n",
    "\n",
    "# Include some time series data in the example\n",
    "for t in range(5):\n",
    "    writer.write_stamped_msg(\"positions/lara\", Position(x=t, y=t+1), t_sec=t)\n",
    "    writer.write_stamped_msg(\"positions/trex\", Position(x=t+2, y=t+3), t_sec=t)\n",
    "\n",
    "# Include some raw data\n",
    "s = b\"i am a raw string\"\n",
    "writer.write_raw(\"raw_data\", s)\n",
    "\n",
    "writer.close()\n",
    "\n",
    "!unzip -l pandas_example.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll read back that `protobag` data and load it into a `pandas.DataFrame`.  First, let's take a close look at what `DictRowEntry` actually provides for a single entry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "protobag.DictRowEntry:\n",
      "  entryname: hunters/py_lara\n",
      "  (not a time-series entry)\n",
      "  type_url: type.googleapis.com/my_messages.DinoHunter\n",
      "  has serdes: True\n",
      "  descriptor_data: (protobuf message) google.protobuf.FileDescriptorSet file {\n",
      "  name: \"MyMe (422 bytes)\n",
      "  msg_dict:\n",
      " {'dinos': [{'name': 'py_bites', 'type': 'PEOPLEEATINGSAURUS'},\n",
      "           {'name': 'py_stinky', 'type': 'VEGGIESAURUS'}],\n",
      " 'firstName': 'py_lara',\n",
      " 'id': 2}\n"
     ]
    }
   ],
   "source": [
    "bag = protobag.Protobag(path='pandas_example.zip', msg_classes=(DinoHunter, Position))\n",
    "\n",
    "entry = bag.get_entry(\"hunters/py_lara\")\n",
    "row = protobag.DictRowEntry.from_entry(entry)\n",
    "print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create the `DataFrame`.  For simplicity, we just create one `DataFrame` table row per archive file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read 13 rows\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 13 entries, 0 to 12\n",
      "Data columns (total 6 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   entryname        13 non-null     object        \n",
      " 1   type_url         13 non-null     object        \n",
      " 2   msg_dict         13 non-null     object        \n",
      " 3   topic            13 non-null     object        \n",
      " 4   timestamp        10 non-null     datetime64[ns]\n",
      " 5   descriptor_data  12 non-null     object        \n",
      "dtypes: datetime64[ns](1), object(5)\n",
      "memory usage: 752.0+ bytes\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entryname</th>\n",
       "      <th>type_url</th>\n",
       "      <th>msg_dict</th>\n",
       "      <th>topic</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>descriptor_data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>hunters/py_max</td>\n",
       "      <td>type.googleapis.com/my_messages.DinoHunter</td>\n",
       "      <td>{'firstName': 'py_max', 'id': 1, 'dinos': [{'n...</td>\n",
       "      <td></td>\n",
       "      <td>NaT</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>hunters/py_lara</td>\n",
       "      <td>type.googleapis.com/my_messages.DinoHunter</td>\n",
       "      <td>{'firstName': 'py_lara', 'id': 2, 'dinos': [{'...</td>\n",
       "      <td></td>\n",
       "      <td>NaT</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>positions/lara/0.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'y': 1.0}</td>\n",
       "      <td>positions/lara</td>\n",
       "      <td>1970-01-01 00:00:00</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>positions/trex/0.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 2.0, 'y': 3.0}</td>\n",
       "      <td>positions/trex</td>\n",
       "      <td>1970-01-01 00:00:00</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>positions/lara/1.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 1.0, 'y': 2.0}</td>\n",
       "      <td>positions/lara</td>\n",
       "      <td>1970-01-01 00:00:01</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>positions/trex/1.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 3.0, 'y': 4.0}</td>\n",
       "      <td>positions/trex</td>\n",
       "      <td>1970-01-01 00:00:01</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>positions/lara/2.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 2.0, 'y': 3.0}</td>\n",
       "      <td>positions/lara</td>\n",
       "      <td>1970-01-01 00:00:02</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>positions/trex/2.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 4.0, 'y': 5.0}</td>\n",
       "      <td>positions/trex</td>\n",
       "      <td>1970-01-01 00:00:02</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>positions/lara/3.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 3.0, 'y': 4.0}</td>\n",
       "      <td>positions/lara</td>\n",
       "      <td>1970-01-01 00:00:03</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>positions/trex/3.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 5.0, 'y': 6.0}</td>\n",
       "      <td>positions/trex</td>\n",
       "      <td>1970-01-01 00:00:03</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>positions/lara/4.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 4.0, 'y': 5.0}</td>\n",
       "      <td>positions/lara</td>\n",
       "      <td>1970-01-01 00:00:04</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>positions/trex/4.0.stampedmsg.protobin</td>\n",
       "      <td>type.googleapis.com/my_messages.Position</td>\n",
       "      <td>{'x': 6.0, 'y': 7.0}</td>\n",
       "      <td>positions/trex</td>\n",
       "      <td>1970-01-01 00:00:04</td>\n",
       "      <td>b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>raw_data</td>\n",
       "      <td></td>\n",
       "      <td>{'protobag_raw_entry_bytes': b'i am a raw stri...</td>\n",
       "      <td></td>\n",
       "      <td>NaT</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 entryname  \\\n",
       "0                           hunters/py_max   \n",
       "1                          hunters/py_lara   \n",
       "2   positions/lara/0.0.stampedmsg.protobin   \n",
       "3   positions/trex/0.0.stampedmsg.protobin   \n",
       "4   positions/lara/1.0.stampedmsg.protobin   \n",
       "5   positions/trex/1.0.stampedmsg.protobin   \n",
       "6   positions/lara/2.0.stampedmsg.protobin   \n",
       "7   positions/trex/2.0.stampedmsg.protobin   \n",
       "8   positions/lara/3.0.stampedmsg.protobin   \n",
       "9   positions/trex/3.0.stampedmsg.protobin   \n",
       "10  positions/lara/4.0.stampedmsg.protobin   \n",
       "11  positions/trex/4.0.stampedmsg.protobin   \n",
       "12                                raw_data   \n",
       "\n",
       "                                      type_url  \\\n",
       "0   type.googleapis.com/my_messages.DinoHunter   \n",
       "1   type.googleapis.com/my_messages.DinoHunter   \n",
       "2     type.googleapis.com/my_messages.Position   \n",
       "3     type.googleapis.com/my_messages.Position   \n",
       "4     type.googleapis.com/my_messages.Position   \n",
       "5     type.googleapis.com/my_messages.Position   \n",
       "6     type.googleapis.com/my_messages.Position   \n",
       "7     type.googleapis.com/my_messages.Position   \n",
       "8     type.googleapis.com/my_messages.Position   \n",
       "9     type.googleapis.com/my_messages.Position   \n",
       "10    type.googleapis.com/my_messages.Position   \n",
       "11    type.googleapis.com/my_messages.Position   \n",
       "12                                               \n",
       "\n",
       "                                             msg_dict           topic  \\\n",
       "0   {'firstName': 'py_max', 'id': 1, 'dinos': [{'n...                   \n",
       "1   {'firstName': 'py_lara', 'id': 2, 'dinos': [{'...                   \n",
       "2                                          {'y': 1.0}  positions/lara   \n",
       "3                                {'x': 2.0, 'y': 3.0}  positions/trex   \n",
       "4                                {'x': 1.0, 'y': 2.0}  positions/lara   \n",
       "5                                {'x': 3.0, 'y': 4.0}  positions/trex   \n",
       "6                                {'x': 2.0, 'y': 3.0}  positions/lara   \n",
       "7                                {'x': 4.0, 'y': 5.0}  positions/trex   \n",
       "8                                {'x': 3.0, 'y': 4.0}  positions/lara   \n",
       "9                                {'x': 5.0, 'y': 6.0}  positions/trex   \n",
       "10                               {'x': 4.0, 'y': 5.0}  positions/lara   \n",
       "11                               {'x': 6.0, 'y': 7.0}  positions/trex   \n",
       "12  {'protobag_raw_entry_bytes': b'i am a raw stri...                   \n",
       "\n",
       "             timestamp                                    descriptor_data  \n",
       "0                  NaT  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "1                  NaT  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "2  1970-01-01 00:00:00  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "3  1970-01-01 00:00:00  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "4  1970-01-01 00:00:01  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "5  1970-01-01 00:00:01  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "6  1970-01-01 00:00:02  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "7  1970-01-01 00:00:02  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "8  1970-01-01 00:00:03  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "9  1970-01-01 00:00:03  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "10 1970-01-01 00:00:04  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "11 1970-01-01 00:00:04  b'\\n\\xa3\\x03\\n\\x10MyMessages.proto\\x12\\x0bmy_m...  \n",
       "12                 NaT                                               None  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Collect \"rows\" here, which will be just python dicts.  Pandas is happy to convert dicts to rows.\n",
    "rows = []\n",
    "for entry in bag.iter_entries():\n",
    "    # ignore the index\n",
    "    if '_protobag_index' in entry.entryname:\n",
    "      continue\n",
    "\n",
    "    rows.append(protobag.DictRowEntry.from_entry(entry))\n",
    "\n",
    "print(\"Read %s rows\" % len(rows))\n",
    "\n",
    "df = pd.DataFrame([\n",
    "# Convert to pyarrow-friendly types\n",
    "    dict(\n",
    "      entryname=row.entryname,\n",
    "      type_url=row.type_url,\n",
    "      msg_dict=row.msg_dict,\n",
    "      topic=row.topic,\n",
    "      timestamp=\n",
    "        row.timestamp.ToDatetime() if row.timestamp else None,\n",
    "      descriptor_data=\n",
    "        row.descriptor_data.SerializeToString() if row.descriptor_data else None,\n",
    "    )\n",
    "    for row in rows\n",
    "])\n",
    "df.info()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now suppose we want to convert back to `protobag` format.  We need a few key things from our table:\n",
    " * A `msg_dict` message dictionary that we want to convert back to a Protobuf message.\n",
    " * Some `descriptor_data`, which Protobuf's underlying `json_format.ParseDict()` needs in order to convert `msg_dict` into a protobuf message.\n",
    " * Some other context like `entryname` or a `topic`-`timestamp` that defines where to put the message in the `protobag` archive.\n",
    " \n",
    "Note that `msg_dict` and `descriptor_data` are only required if you want a lossless / bijective transform between your Protobuf messages and table row data.  You may very well have a more complex mapping between your Protobuf and table data; `protobag` simply provides tooling to support as much of a basic bijective mapping as possible.\n",
    " \n",
    "Our table above has all the required data, so conversion is simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  pandas_to_protobag_example.zip\r\n",
      "  Length      Date    Time    Name\r\n",
      "---------  ---------- -----   ----\r\n",
      "       72  1980-01-01 00:00   hunters/py_max\r\n",
      "       86  1980-01-01 00:00   hunters/py_lara\r\n",
      "       49  1980-01-01 00:00   positions/lara/0.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/trex/0.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/lara/1.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/trex/1.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/lara/2.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/trex/2.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/lara/3.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/trex/3.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/lara/4.0.stampedmsg.protobin\r\n",
      "       54  1980-01-01 00:00   positions/trex/4.0.stampedmsg.protobin\r\n",
      "       19  1980-01-01 00:00   raw_data\r\n",
      "     9141  1980-01-01 00:00   /_protobag_index/bag_index/1596831111.0.stampedmsg.protobin\r\n",
      "---------                     -------\r\n",
      "     9853                     14 files\r\n"
     ]
    }
   ],
   "source": [
    "# Get row dicts from pandas\n",
    "rows = df.to_dict(orient='records')\n",
    "\n",
    "# We'll write to this archive:\n",
    "bag = protobag.Protobag(path='pandas_to_protobag_example.zip')\n",
    "writer = bag.create_writer()\n",
    "\n",
    "# Loop through rows, convering them back to `protobag` Entries using `DictRowEntry`\n",
    "for row in rows:\n",
    "    dict_row_entry = protobag.DictRowEntry(**row)\n",
    "    entry = dict_row_entry.to_entry()\n",
    "    writer.write_entry(entry)\n",
    "writer.close()\n",
    "\n",
    "!unzip -l pandas_to_protobag_example.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}