getting_started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Imaging - Getting Started"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from tqdm import tqdm\n",
    "\n",
    "from helpers import utils, tf_helpers\n",
    "\n",
    "utils.setup_logging()\n",
    "\n",
    "tf_helpers.print_versions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. General Toolbox Structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with the high-level toolbox structure, just to have a general feel of how things are organized.\n",
    "\n",
    "**Python Modules**\n",
    "```\n",
    ">> neural-imaging:\n",
    "├── compression                     - learned image codec, helper functions for JPEG and BPG codecs\n",
    "├── config                          - configuration files\n",
    "├── data                            - data directory (datasets, models, results)\n",
    "├── docs                            - documentation\n",
    "├── helpers                         - helper functions (working with datasets, results, plotting, image processing, etc.):\n",
    "├── models                          - Tensorflow models (camera ISPs, codecs, forensics, custom layers, etc.)\n",
    "├── pyfse                           - wrappers for the FSE entropy codec (Cython)\n",
    "├── training                        - training routines\n",
    "└── workflows                       - workflows for various applications\n",
    " \n",
    "```\n",
    "\n",
    "**Command Line Tools**\n",
    "```\n",
    ">> Training scripts:\n",
    "├── train_dcn.py                    - train lossy image codecs\n",
    "├── train_manipulation.py           - train manipulation classification workflows\n",
    "├── train_nip.py                    - train camera ISP models\n",
    "└── train_prepare_training_set.py   - prepare training data\n",
    "\n",
    ">> Testing scripts:\n",
    "├── test_dcn.py                     - test learned image codecs (plots, rate-distortion) \n",
    "├── test_dcn_rate_dist.py           - compare rate-distortion profiles for various codecs\n",
    "├── test_fan.py                     - test forensic manipulation classification \n",
    "├── test_framework.py               - high-level framework test\n",
    "├── test_jpeg.py                    - test differentiable JPEG codec\n",
    "└── test_nip.py                     - test camera ISPs\n",
    "\n",
    ">> Others:\n",
    "├── develop_images.py               - render RAW images with selected ISP\n",
    "├── diff_nip.py                     - compare output from 2 ISPs\n",
    "├── results.py                      - search & display results from manipulation classification\n",
    "└── summarize_nip.py                - summarize ISP training statistics\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Working with Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The toolbox features a `Dataset` class to help with loading and feeding the images into the model. The class splits images into training and testing subsets. The **training subset** is loaded as **full resolution** images, and patches will be randomly sampled on demand. The **validation subset** is **sampled upon loading** and only patches are stored in the memory. The dataset can load either RAW-RGB pairs (default) or just the RGB images - the behavior is controlled via the `load` argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Notice:</b> The following sections assume you have downloaded example datasets. Feel free to get example data from <a href=\"https://pkorus.pl/downloads/neural-imaging-resources\">pkorus.pl/downloads/neural-imaging-resources</a> or adjust the paths accordingly.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you download the data and models, they should be placed in the following folders:\n",
    "\n",
    "```\n",
    "data/raw/                               - RAW images used for camera ISP training\n",
    "  └ training_data/{camera name}          Bayer stacks (*.npy) and developed (*.png)\n",
    "data/models                             - pre-trained TF models\n",
    "  ├ nip/{camera name}/{nip}              NIP models (TF checkpoints)\n",
    "  ├ isp                                  Classic ISP models (TF checkpoints)\n",
    "  └ dcn/baselines/{dcn model}            DCN models (TF checkpoints)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from helpers import dataset, plots, utils, image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = dataset.Dataset('data/raw/training_data/D90/', n_images=10, v_images=10, val_n_patches=4, load='xy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. Plotting 101"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting and visualization functions are available in the `helpers.plots` module. We'll start with plotting images. Your swiss army knife is the `plots.images` function, which shows grids of image collections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample an example training batch\n",
    "batch_raw, batch_rgb = data.next_training_batch(0, 10, 128)\n",
    "plots.images(batch_rgb, '{}', ncols=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Patch Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sampling from the dataset can be fully random or biased. The behavior can be controlled via the `discard` argument. The following modes are available:\n",
    "\n",
    "- `flat` - attempts to discard flat patches based on patch variance (not strict)\n",
    "- `flat-aggressive` - a more aggressive version that avoids patches with variance < 0.01\n",
    "- `dark-n-textured` - avoid dark (mean < 0.35) and textured patches (variance > 0.005)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show differences between patch discarding policies\n",
    "image_id = 4\n",
    "patches = []\n",
    "discard_modes = [None, 'flat', 'flat-aggressive', 'dark-n-textured']\n",
    "\n",
    "for discard in discard_modes:\n",
    "    patches_current = [data.next_training_batch(image_id, 1, 128, discard)[-1].squeeze() for _ in range(10)]\n",
    "    patches.extend(patches_current)\n",
    "    print(f'discard={discard} -> mean: {np.mean(patches_current):.3f} & var : {np.var(patches_current, axis=(1, 2, 3)).mean():.3f}')\n",
    "\n",
    "plots.images(patches, '', ncols=-4, rowlabels=discard_modes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show distribution of patch variances for different discard policies\n",
    "patches = image.cati([data.next_training_batch(image_id, 1, 128, discard=None)[-1] for _ in range(500)])\n",
    "\n",
    "_ = plt.hist(np.var(patches, axis=(1,2,3)), 30)\n",
    "plt.xlabel('patch variance')\n",
    "plt.title(f'random sampling ({data.files[\"training\"][image_id]})')\n",
    "plt.legend([f'mean: {np.mean(patches):.3f}, var : {np.var(patches, axis=(1, 2, 3)).mean():.3f}'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Working with Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The toolbox contains several models:\n",
    "- camera ISP models: both **classic ISP** and **neural ISP**\n",
    "- image compression: a **differentiable JPEG codec** and a **fully learned codec**\n",
    "- image forensics: state-of-the-art **constrained CNN model**\n",
    "\n",
    "The models are defined in sub-modules of `models` and need to be derived from `tfmodel.TFModel`. Certain types of models have corresponding abstract model classes that simplify definition of new architectures by providing common functionality (e.g., `NIP` class for defining camera ISP models, or `DCN` for learned image codecs).\n",
    "\n",
    "For trainable models, creating new instances produces uninitialized models with random weights. Such models need to be either trained or restored with pre-trained weights. The snippet below shows 4 different ways in which models can be restored. The following sections will show more diverse practical examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# 1. Manual - create model instance and load weights\n",
    "dcn = compression.TwitterDCN(rounding=\"soft-codebook\",n_features=16)\n",
    "dcn.load_model('data/models/dcn/baselines/16c')`\n",
    "\n",
    "# 2. Restore a pre-trained model of a known class\n",
    "compression.TwitterDCN.restore('data/models/dcn/baselines/16c/', key='codec')\n",
    "\n",
    "# 3. Restore a pre-trained model from a known Python module\n",
    "tfmodel.restore('data/models/dcn/baselines/16c/', compression, key='codec')\n",
    "\n",
    "# 4. Convenience function to restore specific model classes (here, compression)\n",
    "codec.restore('16c')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's load a dataset for our experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = dataset.Dataset('D90', n_images=10, v_images=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Camera ISP "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this demonstration, we will compare the output of a classic ISP and a neural ISP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from models import pipelines\n",
    "\n",
    "# Restore a Classic ISP and a neural imaging pipeline\n",
    "isp = pipelines.ClassicISP.restore(camera='D90')\n",
    "nip = pipelines.DNet.restore('data/models/nip/D90/')\n",
    "batch_size = 6\n",
    "\n",
    "# Fetch a validation batch from the dataset\n",
    "batch_raw, batch_rgb = data.next_validation_batch(0, batch_size)\n",
    "\n",
    "# Compare ground truth images with ISP & NIP output\n",
    "plots.images(image.cati(batch_rgb, isp.process(batch_raw), nip.process(batch_raw)), '{}', ncols=-3, rowlabels=['Ground truth', 'Classic ISP', f'Neural Pipeline ({nip.class_name})'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. JPEG Compression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The toolbox provides a fully differentiable JPEG codec, which is useful for training components that happen earlier in the workflow (e.g., various types of watermark or fingerprint embedders). The best way of using the codec is via the `jpeg.JPEG` class which serves as a generic interface, and allows for switching the codec in a seamless way. \n",
    "\n",
    "In this example, we show how to take advantage of this behavior and compare the results of JPEG compression using the differentiable codec and the open source **libJPEG**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from models import jpeg\n",
    "\n",
    "batch_raw, batch_rgb = data.next_training_batch(0, batch_size, 64)\n",
    "\n",
    "codec_differentiable = jpeg.JPEG(50)\n",
    "codec_libjpeg = jpeg.JPEG(50, 'libjpeg')\n",
    "\n",
    "plots.images(image.cati(batch_rgb, codec_differentiable.process(batch_rgb), codec_libjpeg.process(batch_rgb)), '', ncols=-3, rowlabels=['Uncompressed', 'Diff. JPEG', 'LibJPEG'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3. Learned Compression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The toolbox provides a fully learned image compression codec and three pre-trained quality configurations. In this example, we use the low-quality settings and compare the results against\n",
    "a standard JPEG codec at a similar SSIM level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from compression import codec, jpeg_helpers\n",
    "from helpers import metrics\n",
    "\n",
    "batch_size = 6\n",
    "\n",
    "# Restore a learned codec with low-quality settings (16 channels)\n",
    "codec_dcn = codec.restore('16c')\n",
    "codec_jpg = jpeg.JPEG(codec='libjpeg')\n",
    "\n",
    "batch_rgb = data.next_validation_batch(0, batch_size)[-1]\n",
    "batch_dcn = codec_dcn.process(batch_rgb).numpy()\n",
    "\n",
    "# Find SSIM measurements for DCN-compressed images\n",
    "ssims_d = metrics.ssim(batch_dcn, batch_rgb)\n",
    "\n",
    "# Find JPEG quality factors that lead to the same SSIM levels\n",
    "jpeg_qf = [jpeg_helpers.match_quality(rgb, ssim) for rgb, ssim in zip(batch_rgb, ssims_d)]\n",
    "\n",
    "# Compress with JPEG to match the quality level\n",
    "batch_jpg = [codec_jpg.process(np.expand_dims(rgb, axis=0), qf) for rgb, qf in zip(batch_rgb, jpeg_qf)]\n",
    "ssims_j = metrics.ssim(batch_rgb, np.concatenate(batch_jpg))\n",
    "\n",
    "# Plot results\n",
    "labels_o = batch_size * ['']\n",
    "labels_j = [f'QF={qf} -> SSIM={ssim:.3f}' for qf, ssim in zip(jpeg_qf, ssims_j) ]\n",
    "labels_d = [f'SSIM={ssim:.3f}' for ssim in ssims_d]\n",
    "\n",
    "plots.images(image.cati(batch_rgb, batch_dcn, batch_jpg), labels_o + labels_d + labels_j, ncols=-3, rowlabels=['Uncompressed', f'Learned codec ({codec_dcn.summary_compact()})', 'JPEG @ similar SSIM'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Workflows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models can be combined into **workflows** that model specific applications. The current version of the toolbox provides an example workflow that models **manipulation classification**. The model involves training a forensic analysis network (FAN) to identify subtle post-processing operations applied to the image. The process starts with the camera ISP and is followed by photo manipulations and a distribution channel. The FAN can access images after they have been degraded (e.g., down-sampled and compressed) by the channel. The generic model is shown below:   \n",
    "\n",
    "![](docs/manipulation_detection_training_architecture.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we will train a toy version of the model with:\n",
    "- a dummy ISP (feeds RGB images), \n",
    "- 2 easy manipulations (sharpening and Gaussian blurring), \n",
    "- mild JPEG compression,\n",
    "- a small forensic CNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1. Construct the workflow to our specification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from workflows import manipulation_classification\n",
    "\n",
    "# Define a list of manipulations and their strength\n",
    "manipulations = ['sharpen:0.75', 'gaussian:1', 'jpeg:50']\n",
    "\n",
    "# Define the distribution channel: no resizing and JPEG compression\n",
    "distribution = {\n",
    "    'downsampling': 'none',\n",
    "    'compression': 'jpeg',\n",
    "    'compression_params': {\n",
    "        'quality': 90,\n",
    "        'codec': 'soft',\n",
    "    }\n",
    "}\n",
    "\n",
    "# Construct the workflow with a dummy ISP that allows for feeding RGB images\n",
    "flow = manipulation_classification.ManipulationClassification('ONet', manipulations, distribution, {'kernel': 3, 'n_dense': 2}, {}, raw_patch_size=32)\n",
    "\n",
    "print(flow.details())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2. Load an RGB dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from helpers import dataset\n",
    "\n",
    "data = dataset.Dataset('D90', n_images=50, v_images=50, val_n_patches=2, val_rgb_patch_size=64, load='y')\n",
    "print(data.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3. Visualize manipulations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_m = flow.run_manipulations(data.next_training_batch(0, 1, 64, 'flat-aggressive')).numpy()\n",
    "\n",
    "plots.images(batch_m, ['{} ()'.format(x) for x in flow._forensics_classes], ncols=flow.n_classes, figwidth=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4. Train the entire workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's train the entire workflow. In this case, we are training only the forensics model. \n",
    "\n",
    "We're playing with a toy example and small patches - the whole process should take no more than 10-15 minutes on a mid-tier desktop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from training.manipulation import train_manipulation_nip\n",
    "\n",
    "# Training setup\n",
    "training = {\n",
    "    'camera_name': 'D90',\n",
    "    'patch_size': 32,\n",
    "    'batch_size': 10,\n",
    "    'n_epochs': 501,\n",
    "    'validation_schedule': 10,\n",
    "    'lambda_nip': 0,\n",
    "}\n",
    "\n",
    "output_directory = train_manipulation_nip(flow, training, data, {'root': 'data/getting_started/m'}, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5. Validate the trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll test the model on the entire (validation) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from training.validation import validate_fan\n",
    "\n",
    "accuracy, conf_mat = validate_fan(flow, data)\n",
    "\n",
    "print(f'\\nAccuracy: {accuracy:.2f}')\n",
    "print('Confusion:')\n",
    "print(conf_mat.round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, let's run FAN predictions on a sample batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Classify different post-processed variations of a sample patch\n",
    "batch_x = data.next_validation_batch(1, 1)\n",
    "\n",
    "# Fetch processed & distributed patches - as seen by the FAN\n",
    "batch_f = flow.run_rgb_to_fan(batch_x)\n",
    "\n",
    "# Run the patches through the FAN\n",
    "predicted_class, confidence = flow.fan.process_and_decide(batch_f, True)\n",
    "\n",
    "# Prepare labels: real_class -> predicted_class (confidence)\n",
    "class_labels = flow._forensics_classes\n",
    "labels = [f'{class_labels[real_class]} -> {class_labels[pred_class]} (confidence={conf:.2f})' for real_class, (pred_class, conf) in enumerate(zip(predicted_class, confidence))]\n",
    "\n",
    "# Plot all images\n",
    "plots.images(batch_f, labels, ncols=-1, figwidth=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Working with Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The toolbox provides helper functions to work with results, e.g.:\n",
    "- finding results stored before (e.g., the results of our previous training runs)\n",
    "- rendering & visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example, let's print the above confusion matrix in a more accessible form: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from helpers import results_data, plots\n",
    "\n",
    "print(results_data.confusion_to_text(100 * conf_mat, flow._forensics_classes, flow.summary_compact()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1. The Result Cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of recomputing validation statistics, we can also fetch them from the training log. We will explore the most general way of accessing saved results - via the `ResultsCache`.\n",
    "\n",
    "`ResultCache` is a helper class which simplifies searching for and access to results stored as either `.json` or `.npz` (numpy archives). It uses templates for building the path and allows you to incrementally provide more specific search terms (see details in the docstrings). In this example, we will look up a predefined template for `manipulation_classification`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache = results_data.ResultCache('manipulation_classification', 'data/getting_started/m', camera='D90')\n",
    "print(str(cache))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using the `find` method we see the current pattern used for searching and the list of all matching results. In this example, we only have one training run, so we don't have to be too specific."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache.find()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can request more specific results by specifying values for (some of the) search terms. Note how the search pattern changes. Search terms that are expected to be common for all queries, can be specified in the constructor.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Notice:</b> We can request more specific results by specifying values for (some of the) search terms. Note how the search pattern changes. Search terms that are expected to be common for all queries, can be specified in the constructor.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache.find(isp='ONet')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Enough introduction, let's load our training log!**\n",
    "\n",
    "The returned data is exposed as a `dict` object. For bervity, we'll print only the relevant section about the forensic training. You will see several keys:\n",
    "- `model` and `args` contain class name and arguments that will allow you to restore this model later,\n",
    "- `performance` contains the values of the metrics that we tracked during training and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = cache.load()\n",
    "utils.printd(results['forensics'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot the training progress using helper functions from the `plots` module. we'll use the `utils.get` function to recursively find keys in our dict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plots.perf(utils.get(results, 'forensics.performance'), alpha=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Rendering Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have another look at the confusion matrix. We can compare the recently computed validation results to the one recorded during training. \n",
    "\n",
    "To make things more interesting, we can **render the new table as a LaTeX table** (useful when preparing your manuscript)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conf_log = np.array(utils.get(results, 'forensics.performance.confusion'))\n",
    "conf_labels = utils.get(results, 'manipulations')\n",
    "\n",
    "tex = results_data.confusion_to_text(100 * conf_log, conf_labels, fmt='tex')\n",
    "\n",
    "print(tex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're lucky enough to have your LaTeX environment ready, you can give it a shot and render the table directly in here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_data.render_tex(tex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Confusion matrices are rather specific, so you can also use a more general function to convert any 2D arrays. It has more options and more features that this short introduction can allow for, so I'll leave you with one example and the docstrings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_data.convert_table(100 * conf_log, conf_labels, fmt='df')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Final Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations, you made it through the tutorial!**\n",
    "\n",
    "We barely scratched the surface of the toolbox, but this should be enough to get you started. \n",
    "\n",
    "If you have any comments/suggestions or would like to contribute your code or models, feel free to reach out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Modules\n",
    "```\n",
    ">> neural-imaging:\n",
    "├── compression                     - learned image codec, helper functions for JPEG and BPG codecs\n",
    "│   ├── bpg_helpers\n",
    "│   ├── codec\n",
    "│   ├── jpeg_helpers\n",
    "│   └── ratedistortion\n",
    "├── config                          - configuration files\n",
    "├── data                            - data directory (datasets, models, results)\n",
    "├── docs                            - documentation\n",
    "├── helpers                         - helper functions:\n",
    "│   ├── dataset                       . loading and serving image datasets\n",
    "│   ├── debugging                     . memory usage\n",
    "│   ├── fsutil                        . dealing with filenames and directory listing\n",
    "│   ├── image                         . image processing\n",
    "│   ├── imdiff                        . comparing images\n",
    "│   ├── kernels                       . image filtering kernels\n",
    "│   ├── loading                       . loading images and extracting patches\n",
    "│   ├── metrics                       . image quality metrics\n",
    "│   ├── paramspec                     . handling and verification of hyper-parameters\n",
    "│   ├── plots                         . plotting & visualization\n",
    "│   ├── raw                           . rendering RAW files and working with Bayer data\n",
    "│   ├── results_data                  . rendering, saving & loading results\n",
    "│   ├── stats                         . detection statics (tpr, roc, auc)\n",
    "│   ├── summaries                     . writing complex data to TF logs\n",
    "│   ├── tf_helpers                    . various Tensorflow helpers \n",
    "│   └── utils                         . various helpers (logging, printing, number tests)\n",
    "├── models                          - Tensorflow models\n",
    "│   ├── compression                   . learned image codecs (base + TwitterDCN)\n",
    "│   ├── forensics                     . forensic analysis network (FAN)\n",
    "│   ├── jpeg                          . differentiable JPEG\n",
    "│   ├── layers                        . various layers (constrained convolution, quantization)\n",
    "│   ├── pipelines                     . camera ISP (classic & neural)\n",
    "│   └── tfmodel                       . base class for all models (TFModel)\n",
    "├── pyfse                           - wrappers for the FSE entropy codec (Cython)\n",
    "├── training                        - training routines\n",
    "│   ├── compression\n",
    "│   ├── manipulation\n",
    "│   ├── pipeline\n",
    "│   └── validation\n",
    "└── workflows                       - workflows for various applications\n",
    "    └── manipulation_classification\n",
    " \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Command Line Tools\n",
    "```\n",
    ">> neural-imaging:\n",
    "├── train_dcn.py                    - train lossy image codecs\n",
    "├── train_manipulation.py           - train manipulation classification workflows\n",
    "├── train_nip.py                    - train camera ISP models\n",
    "├── train_prepare_training_set.py   - prepare training data\n",
    "├── develop_images.py               - render RAW images with selected ISP\n",
    "├── diff_nip.py                     - compare output from 2 ISPs\n",
    "├── results.py                      - search & display results from manipulation classification\n",
    "├── summarize_nip.py                - summarize ISP training statistics\n",
    "├── test_dcn.py                     - test learned image codecs (plots, rate-distortion) \n",
    "├── test_dcn_rate_dist.py           - compare rate-distortion profiles for various codecs\n",
    "├── test_fan.py                     - test forensic manipulation classification \n",
    "├── test_framework.py               - high-level framework test\n",
    "├── test_jpeg.py                    - test differentiable JPEG codec\n",
    "└── test_nip.py                     - test camera ISPs\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}