pix2pixHD-DR-v1.1.ipynb

{"cells":[{"attachments":{},"cell_type":"markdown","metadata":{"id":"xVO-LxDxs1YR"},"source":["# Pix2PixHD\n","\n","**Last updated: March 31, 2023**\n","\n","This notebook was created by Doug Rosman, and uses code from [Doug's forked pix2pixHD repository](https://github.com/dougrosman/pix2pixHD). For a video tutorial showing how to use this notebook, visit this link here: [https://dougrosman.github.io/cvml-sp21/resources/pix2pixHD/](https://dougrosman.github.io/cvml-sp21/resources/pix2pixHD/) (Note, this video was created for a prior version of this notebook. Most of the functionality in the video matches this notebook, but there might be some differences in how files and folders are named and used.)\n","\n","## Running this notebook locally\n","\n","This notebook can be run locally (if you have an Nvidia GPU and are using Linux). The notebook functions basically the same whether you're running it locally or in Google Colab, but some of the setup steps are different for local (things are a bit easier). The first-time setup requires a couple steps, though. **These setup instructions assume you have your Nvidia and Cuda drivers installed, as well as Anaconda and VS Code with the necessary Python and Jupyter Notebook extensions. Setting up Ubuntu to run this code locally requires extra setup that, unfortunately, is beyond the scope of this notebook.**\n","\n","### First-time setup for local use\n","1. If you haven't already, create a folder somewhere on your computer and put this notebook in it. Call the notebook whatever you want, maybe something like \"pix2pixHD\"\n","2. In VS Code, go to File-->Open Folder and open the folder where you placed this notebook\n","3. Run any of the code cells below (Step 1 is fine)\n","4. Select **Python Environments** for your kernel\n","5. Select **+ Create Python Environment**\n","6. Select **Conda**\n","7. Select **Python 3.9**\n","\n","This Python environment will be saved in this folder, so you won't have to do these setup steps again"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["# Part 1: Setup"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"ewD4YcUPt3Qf"},"source":["## 1a. Connect to a GPU Instance\n","Google Colab: Required if training or generating images\n","\n","Local: Not required (but can be a useful way to initiate your Python environment for the first time you run this notebook.)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"PH4yKFd4zJ6C"},"source":["**In Google Colab, only run this step when you're ready to start training or generating images. Executing this cell will connect you to a GPU and officially start the clock on your instance, so you don't want to waste limited GPU time on tasks that don't require a GPU (like processing images, setting variable names, etc.)**\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":613,"status":"ok","timestamp":1680120419350,"user":{"displayName":"Doug Rosman","userId":"12469472491464808109"},"user_tz":300},"id":"KisNe7y5as8J","outputId":"248e5d3a-d55e-46a5-b300-802237843374"},"outputs":[],"source":["!nvidia-smi -L"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"cyD-3fgfwNd8"},"source":["## 1b. Mount your Google Drive\n","\n","Google Colab: Required\n","\n","Local: do not run"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":26529,"status":"ok","timestamp":1680120445875,"user":{"displayName":"Doug Rosman","userId":"12469472491464808109"},"user_tz":300},"id":"B_Id6wSjbH5u","outputId":"73b20468-5626-408b-be2d-5e4c167d7ac7"},"outputs":[],"source":["from google.colab import drive\n","drive.mount('/content/drive')"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"6DvGIfacxJL_"},"source":["## 1c. Cloning and/or changing directory into the pix2pixHD repo\n","\n","Google Colab: Required\n","\n","Local: Required"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"fdOJwnvTzbn0"},"source":["**Executing this cell will either install the pix2pixHD repo into your Google Drive, locally onto your computer, or change directory into the pix2pixHD repo if it already exists.**\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":15692,"status":"ok","timestamp":1680120634387,"user":{"displayName":"Doug Rosman","userId":"12469472491464808109"},"user_tz":300},"id":"ENQpxYVIbLja","outputId":"ea57e921-0052-4342-ede1-1c810cd4f5c0"},"outputs":[],"source":["import os\n","\n","if os.path.isdir(\"/content/drive/MyDrive/colab-pix2pixHD\"):\n","    %cd \"/content/drive/MyDrive/colab-pix2pixHD/pix2pixHD\"\n","elif os.path.isdir(\"pix2pixHD\"):\n","    %cd \"pix2pixHD\"\n","elif os.path.isdir(\"/content/drive/\"):\n","    #install script\n","    %cd \"/content/drive/MyDrive/\"\n","    !mkdir colab-pix2pixHD\n","    %cd colab-pix2pixHD\n","    !git clone https://github.com/dougrosman/pix2pixHD\n","    %cd pix2pixHD\n","   \n","else:\n","    !git clone https://github.com/dougrosman/pix2pixHD\n","    %cd pix2pixHD\n","    !mkdir generated_videos"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["## 1d. Installing dependencies\n","\n","Google Colab: Required\n","Local: Required only once, the first time you ever run this notebook."]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!pip install dominate\n","\n","# install python requirements for Derrick Schultz' dataset-tools.py\n","# #more info: https://github.com/dvschultz/dataset-tools\n","!pip install -r util/requirements.txt"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"Mxg6p_9CPylP"},"source":["# Part 2: Data Set Preparation\n","\n","This notebook includes the following commands to help you create your data set:\n","\n","1. Create the required folders for organizing your training data\n","1. Extract frames from a video file using FFMPEG\n","1. Unzip a folder full of images you uploaded to Google Drive\n","1. Create Canny edge versions of your images for your input (train_A) images"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"f-tEfNmWUlJR"},"source":["#### **Notes on preparing your dataset**"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"tz3387OwYhJ2"},"source":["1. **Identical image quantities:** The number of input images (train_A) should be identical to the number of output images (train_B) (e.g. 500 of each). Otherwise, your training data will be mismatched, and won't train well.\n","1. **Same order:** The images in your train_A folder should be in the same order as the images in your train_B folder. They are fed into pix2pixHD in alphatbetical/numerical order. Your images should automatically be in the same order, but keep this in mind just in case you notice mismatches in your results.\n","1. **Consistent image dimensions:** all your input images should have the same dimensions, and all your output images should have the same dimensions.\n","1. **Image dimensions that are divisible by 16** pix2pixHD is flexible with your image dimensions, but stick with a width and height where the values are divisible by 16 (e.g. 1024 x 512, 1280 x 720, 854 x 480, 512 x 512)\n","1. **Consistent input/output dimensions:** Technically, pix2pixHD can train datasets where the input dimensions are different from the output (like if you were training an image upscaler), but for most situations, your input and output dimensions should match.\n","1. **Dataset size:** It totally depends. Depending on your goals, and the type of data you're working with, the amount of images you need for a \"successful\" model will vary. Some general tips: I would aim for **_500 image pairs minimum, but more is better. Shoot for the 1,000-3,000 image pair range)._** For example, if your dataset has 1,000 images, that means it has 2x 1,000 images -- 1,000 input images (train_A), and 1,000 output images (train_B)"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"jQzZZsDwP9Wo"},"source":["## 2a. Create your folders\n","\n","Required any time you're going to train a new model"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"VgyGln8INiJR"},"source":["Inside your datasets folder, you'll need a folder for each data set, and each data set folder has a few necessary folders. \n","\n","1. **train_A**, for your input images (e.g. canny edges)\n","1. **train_B**, for your output images\n","1. **test_A**, for the images you'll use to generate images after training your model\n","1. **raw**, for unprocessed images or videos that you intend to process for training\n","\n","The following command creates these folders inside the **datasets** folder.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"executionInfo":{"elapsed":719,"status":"ok","timestamp":1680116459396,"user":{"displayName":"Doug Rosman","userId":"12469472491464808109"},"user_tz":300},"id":"8c8mtNpKQGON"},"outputs":[],"source":["dataset_name = 'my_dataset' # change 'my_dataset' to something specific to your dataset\n","\n","!mkdir ./datasets/$dataset_name\n","!mkdir ./datasets/$dataset_name/train_A\n","!mkdir ./datasets/$dataset_name/train_B\n","!mkdir ./datasets/$dataset_name/test_A\n","!mkdir ./datasets/$dataset_name/raw"]},{"attachments":{},"cell_type":"markdown","metadata":{"id":"ptpbVFmHY5QS"},"source":["### 2b. Getting the right images into the right folders\n","\n","With pix2pixHD, we're training a model to translate one type of image into another. Your \"input\" images go in the train_A folder, and the \"output\" images into train_B. There are a number of ways to prepare a data set, like scraping images from the web or extracting frames from video. Depending on how you're sourcing your images, these steps will differ. The important things are:\n","\n","1. Your train_A and train_B folders should have the same amount of images in each\n","2. The images in train_A and train_B should be in the same order, so that the correct images are matched together during training.\n","3. The width and heights of each image should be consistent, and divisible by **16** (e.g. 1024 x 768, 1024 x 512, 1280 x 720, 512 x 512, 854 x 512, etc.)\n","\n","\n","\n","This section has a couple useful commands for processing your images.\n","\n","Because pix2pixHD has multiple folders for different components of your data set, there are different ways your images can end up in the right folders. The following steps provide different tools for processing your images. Moving data around can be one of the most annoying part of the machine learning process, so practice finding ways to make these steps more efficient. This is where it can be really useful to be comfortable with the command line.\n","\n","#### Some example scenarios\n","\n","##### **Your input and output images were processed locally on your computer** \n","Name your input folder **train_A** and your output folder **train_B**. Zip up your folders individually (use Keka if you're on a Mac), and upload train_A.zip and train_B.zip to your Google Drive in the 'raw' folder for this dataset. Use the unzip and mv commands below to unzip your images and put them in the proper folders.\n","\n","##### **You want to extract your input images from a video using FFMPEG**\n","Upload the video file to the 'raw' folder for this dataset, and run the FFMPEG command below to extract your frames."]},{"cell_type":"markdown","metadata":{"id":"NTzUx2p33LMD"},"source":["### 4c. Unzip your input (train_A) folder\n","If you created your dataset outside of Google Drive and uploaded your train_A.zip and train_B.zip files to the 'raw' folder for this dataset, this command will **unzip your train_A folder**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"oUckUtC5XiEG"},"outputs":[],"source":["#!unzip path_to_file.zip -d path_to_directory\n","!unzip ./datasets/$dataset_name/raw/train_A.zip -d ./datasets/$dataset_name/raw"]},{"cell_type":"markdown","metadata":{"id":"CIDEdqyq4RkB"},"source":["### 4d. Unzip your output (train_B) folder\n","If you created your dataset outside of Google Drive and uploaded your train_A.zip and train_B.zip files to the 'raw' folder for this dataset, this command will **unzip your train_B folder**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"0UytTHO5401W"},"outputs":[],"source":["#!unzip path_to_file.zip -d path_to_directory\n","!unzip ./datasets/$dataset_name/raw/train_B.zip -d ./datasets/$dataset_name/raw"]},{"cell_type":"markdown","metadata":{"id":"PxU5jLsCqMLT"},"source":["### 4e. Extract frames from video – create output (train_B) images (if needed)\n","If you plan to create a dataset by extracting images from a video file, follow these steps.\n","First, upload the video file to the **dataset folder** that contains your train_A, train_B and test_A folders.\n","\n","Make sure the name of your video file is 'input.mp4'"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":2176078,"status":"ok","timestamp":1680119061683,"user":{"displayName":"Doug Rosman","userId":"12469472491464808109"},"user_tz":300},"id":"-UqqKDZIqMLU","outputId":"d0f91709-dcb1-42a1-b7af-4caa347e511c"},"outputs":[],"source":["# Change scale to your desired resolution to resize your images. (1280:-1 scales\n","# images to 1280 for the width, and the height is scaled to maintain the aspect ratio)\n","# change the width to whatever makes sense for your images (I haven't tested\n","# anything larger than 1280x720, but you may be able to go larger. If you experience\n","# an out of memory error, it might be because your images are too large.)\n","# The resolution you can train with depends on the GPU you get.\n","\n","# change the fps (number of frames per second to extract);\n","  ## higher fps = more images to extract, 3-6 is a good range. You likely don't need every\n","  ## single frame from your video to make a good data set. You can also use decimals\n","\n","width = 854 # (854x480) the height will be scaled proportionally to maintain the aspect ratio\n","fps = 1/10 # how many frames per second to extract from the video\n","\n","\n","!ffmpeg \\\n"," -i ./datasets/$dataset_name/raw/forest.mp4 \\\n"," -q:v 2 \\\n"," -vf scale=$width:-1,fps=$fps \\\n"," ./datasets/$dataset_name/train_B/output%5d.jpg"]},{"cell_type":"markdown","metadata":{"id":"PmDg5mMfieUN"},"source":["### 4f. Resize your images (train_B) images (if needed)\n","If you uploaded images in a zip folder that aren't the correct size, you can resize them here.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"OItktIlIiXTP"},"outputs":[],"source":["# change max_size to be the max dimension of your input images (e.g., if your\n","# input images are 1280x720, set max_size to '1280')\n","\n","!python util/dataset-tools.py \\\n","--input_folder ./datasets/$dataset_name/train_B/ \\\n","--output_folder ./datasets/$dataset_name/train_B/resized \\\n","--max_size 1280 \\\n","--verbose"]},{"cell_type":"markdown","metadata":{"id":"yF3xtN1SRrSC"},"source":["### 4g. Apply Canny Edge Detection - create train_A (input) images\n","This step uses [Canny Edge Detection](https://en.wikipedia.org/wiki/Canny_edge_detector) to find edges in your input images. The outlined images are rendered and stored in the train_A folder.\n","\n","**Whether you uploaded your train_B images, or got your train_B images by extracting them from a video, you should run this step if you're creating canny edge images.**\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fGqScjsWR0UT"},"outputs":[],"source":["# change blur amount if there are too many lines in your resulting Canny Edge\n","# images (odd numbers only; start with 5, then try 3 if 5 doesn't give you enough lines)\n","\n","# change max_size to be the max dimension of your input images (e.g., if your\n","# input images are 1280x720, set max_size to '1280')\n","\n","!python util/dataset-tools.py \\\n","--input_folder ./datasets/$dataset_name/train_B/ \\\n","--output_folder ./datasets/$dataset_name/train_A/ \\\n","--process_type canny \\\n","--blur_type gaussian \\\n","--blur_amount 3 \\\n","--max_size 854 \\\n","--verbose"]},{"cell_type":"markdown","metadata":{"id":"DoaXeKHDzln6"},"source":["## 5. Training"]},{"cell_type":"markdown","metadata":{"id":"qAYhoXtqSrij"},"source":["### **Some notes on training:** "]},{"cell_type":"markdown","metadata":{"id":"Bn5UCjrdEc26"},"source":["* **To stop your training manually**, click the stop button on the cell that's running your training.\n","* **There's no set time for how much training your model needs to get the results you want**, but shoot for at least 60 epochs (more is likely better)\n","* **Watch your checkpoints folder as you train.** You can track quality of your training by viewing the sample images that are automatically generated during training. These live in the checkpoints folder, in the folder for your current training. Inside of there, go into web-->images to see your sample images. If it looks like your training is getting __*worse*__, then stop your training.\n","* **When you start your training, stick around for the first 10-15 minutes,** Google Colab sometimes checks to see if you're a robot around that time, so make sure you're there to confirm your humanity.\n","* **Don't close this tab!** You can do other things on your computer, and browse other tabs, but just don't close the tab!\n","* **Don't close your laptop!**\n","* **Don't let your computer fall asleep.** Go into your system settings to make sure your computer won't fall asleep.\n","* **On a free account, you'll get around ~7-10 hours of continuous training.**\n","* **On a pro account, you'll get around ~18-24 hours of continuous training.**\n","* **For free accounts, if you train for around ~30 hours or so in a single week, Google may \"shadowban\" you for a bit**, meaning you might not be able to connect to a GPU until after waiting a few hours (or sometimes an entire day). If you're running into these issues, I recommend Google Colab Pro (it's only $9.99 for the month, and totally worth it).\n","* **You can't do anything else in this notebook while training.** If you want to generate images while training, I recommend opening up a second Colab notebook in another Google account. Note, Google might be on to you if it finds you're using like, 5 Colab notebooks simultaneously. Proceed with this at your own risk. Just make sure you don't mount the same Drive folder in step 2 from multiple Colab notebooks."]},{"cell_type":"markdown","metadata":{"id":"xhsyaD2T0Kdi"},"source":["### 5a. Setting your training variables (required for both training from scratch AND resuming training)\n","\n","Set the following variables, whether you're training from scratch or resuming training. After change the variables, click the play button in this cell to save your values."]},{"cell_type":"code","execution_count":null,"metadata":{"executionInfo":{"elapsed":427,"status":"ok","timestamp":1680120652947,"user":{"displayName":"Doug Rosman","userId":"12469472491464808109"},"user_tz":300},"id":"unqheWJPwd4A"},"outputs":[],"source":["##### REQUIRED: edit these each time you train a new model from scratch!\n","dataset_name = 'edges2forest' # this name needs to match the name of your folder\n","name = 'edges2forest'    # can be whatever you want; name this based on your dataset (e.g. edges2cats)\n","loadSize = 854     # The desired width of your outputs (note: images will be cropped to this) Default=1024 \n","fineSize = 480      # The desired height of your outputs\n","                    ## I haven't trained a model with images larger than 1280x720, though it may be possible!\n","which_epoch = 'latest' # The epoch you wish to resume training from. Keep this set to 'latest' if you want to\n","                        ## pick up from where you left off. Otherwise, put the number of the .pth file you want\n","                        ## to resume from (e.g. 10, 20, 30, etc.)\n","\n","##### OPTIONAL: change these if needed\n","resize_or_crop = 'none'  # keeping this unchanged will automatically resize\n","                                  ## your images to the loadSize and fineSize, then crop to\n","                                  ## those dimensions. Set to 'none' if your images are\n","                                  ## already the correct dimensions\n","                                  ## scale_width\n","\n","display_freq = 200    # frequency of showing training results on screen\n","print_freq = 100      # frequency of showing training results on console\n","save_latest_freq = 500     # frequency of saving the latest results\n","                              ##(lower = more frequent saving, 1000 ~ saves every 10 minutes)\n","save_epoch_freq = 4    # frequency of saving checkpoints at the end of epochs\n","                        # (1 epoch is completed after going through every image in your data set 1 time)"]},{"cell_type":"markdown","metadata":{"id":"wfzVC9CMg_9k"},"source":["### 5b. Running the training command"]},{"cell_type":"markdown","metadata":{"id":"jtamNKBNcm4E"},"source":["#### **Option 1: Training a new model from scratch**\n","\n","With StyleGAN2, we can train our models more quickly using transfer learning, where we resume from a pretrained model. With pix2pix, it's much more common to train a model from scratch."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"KSAXRvJecuyr","outputId":"4305ebc1-9070-4e6c-cb58-c5589ee195b1"},"outputs":[],"source":["# ONLY RUN THIS IF YOU ARE STARTING A NEW TRAINING\n","!python train.py --name=$name --dataroot=./datasets/$dataset_name --checkpoints_dir checkpoints --no_instance --label_nc 0 --resize_or_crop=$resize_or_crop --loadSize=$loadSize --fineSize=$fineSize"]},{"cell_type":"markdown","metadata":{"id":"7RTJPr0d1wid"},"source":["#### **Option 2: Resuming a training**\n","\n","Run this command if you are resuming training. You should still set your variables above before running this command."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"X_rDOKsb1wif"},"outputs":[],"source":["# ONLY RUN THIS IF YOU ARE RESUMING A TRAINING\n","!python train.py --name=$name --dataroot=./datasets/$dataset_name --checkpoints_dir checkpoints --no_instance --label_nc 0 --resize_or_crop=$resize_or_crop --continue_train --which_epoch=$which_epoch --loadSize=$loadSize --fineSize=$fineSize"]},{"cell_type":"markdown","metadata":{"id":"mCSrr8dQ_n8T"},"source":["## 6. Generating Images"]},{"cell_type":"markdown","metadata":{"id":"xbhWt6X1ddv-"},"source":["In order to generate images with pix2pixHD, you need to feed the model with \"test\" images. Your test images should look like your input (train_A) images. For example, if your train_A images used Canny Edge images, then your test images should also be Canny Edge images.\n","\n","pix2pixHD takes the images from your test_A folder and feeds them into your trained model. Any time you want to test new input images with your model, you'll need to replace the images in the test_A folder with your new images.\n","\n","Inevitably, pix2pixHD will have you working with a large amount of images spread across a number of different folders, and your file organization can get out of hand if you don't plan ahead a bit.\n","\n","The following code cells provide tools to prepare your test images. None are required, so read the description before each cell to see if that's something you want to do."]},{"cell_type":"markdown","metadata":{"id":"RErlE8BMjKut"},"source":["### 6a. Preparing your test images"]},{"cell_type":"markdown","metadata":{"id":"AJDzGIwTStcM"},"source":["#### **Set your experiment name**\n","\n","The experiment_name variable you set here will be used throughout the rest of the notebook. Any time you work with a new experiment, change the name here. **_Any instance of $experiment_name you encounter will refer to whatever you set in this code cell._**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"BZSuX--cPobs"},"outputs":[],"source":["experiment_name = 'edges2forest-doug_face' # change to whatever you want to call your experiment."]},{"cell_type":"markdown","metadata":{"id":"I-QOSV5bhdsh"},"source":["#### **Set your Dataset Name**\n","\n","(Note: if you're doing this step in the same session as your training, your dataset_name will have already been set. Feel free to set it again anytime you're generating images using a different dataset/trained model).\n","\n","The dataset_name variable you set here will be used throughout the rest of the notebook. Any time you work with a new dataset, change the name here. **_Any instance of $dataset_name you encounter will refer to whatever you set in this code cell._**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"FpVA791whdsi"},"outputs":[],"source":["dataset_name = 'edges2forest' # change to the name of the dataset you're working with"]},{"cell_type":"markdown","metadata":{"id":"32q351TCYGOa"},"source":["#### **Option 1:** Generating images from your original training data (the images from your train_A folder)"]},{"cell_type":"markdown","metadata":{"id":"3FYWs8Wdd2r4"},"source":["1. This cell removes any images from your test_A folder (if there are any), and... \n","2. Copies all the images from your train_A folder to your test_A folder.\n","Testing your trained model with the original data set can be useful to see how accurately the model can recreate the training data."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NbohBwO3KHFm"},"outputs":[],"source":["# remove any images currently in test_A\n","!rm -v ./datasets/$dataset_name/test_A/*.png\n","\n","# copy images from train_A to test_A\n","!cp -v ./datasets/$dataset_name/train_A/*.png ./datasets/$dataset_name/test_A"]},{"cell_type":"markdown","metadata":{"id":"big1f7SvaB7p"},"source":["#### **Option 2: Testing with Canny Edge images from a new input source.**\n"]},{"cell_type":"markdown","metadata":{"id":"or94rxGsd63A"},"source":["This is where it starts to get important to stay organized. This command creates a folder inside of \"input_test_images\" (which lives in your datasets folder). **Do this for each new experiment you create.**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"s1MbcuHTcjde"},"outputs":[],"source":["# create a folder + relevant subfolders for your new experiment\n","!mkdir ./experiments/$experiment_name\n","!mkdir ./experiments/$experiment_name/extracted_frames\n","!mkdir ./experiments/$experiment_name/canny_edges\n","!mkdir ./experiments/$experiment_name/raw"]},{"cell_type":"markdown","metadata":{"id":"NMh4tpXnHAmZ"},"source":["#### **Upload your source video**\n","After running the above cell, upload the video to the [experiment_name] folder you created above.\n","\n","Make sure the video file is called **input.mp4**."]},{"cell_type":"markdown","metadata":{"id":"iCSsYCKXIQZ1"},"source":["#### **Extract frames from source video**"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!pwd"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"a1FhLxYm9Rbw"},"outputs":[],"source":["# change scale to your desired resolution to resize your images. (1280:-1 scales\n","  ## images to 1280 for the width; the height is scaled to maintain the aspect ratio)\n","  ## change the width to whatever makes sense for your images (I haven't tested\n","  ## anything larger than 1280, but you might be able to go up to 1440 or 1600)\n","# change the fps (number of frames per second to extract);\n","  ## this time, you probably want all the frames from the video, so set\n","  ## the fps to the fps of your input video.\n","# change output%5d.png to include a reference to your experiment_name\n","\n","!ffmpeg \\\n"," -i ./experiments/$experiment_name/raw/doug-face.mp4 \\\n"," -vf scale=854:-1,fps=30 \\\n"," ./experiments/$experiment_name/extracted_frames/output%5d.png"]},{"cell_type":"markdown","metadata":{"id":"tFDYgh1qIFFz"},"source":["#### **Apply Canny Edge Detection - create test_A images**\n","If your images are already prepared and don't need to be converted to Canny edges, you don't need to do this step. This step uses [Canny Edge Detection](https://en.wikipedia.org/wiki/Canny_edge_detector) to find edges in your input images. The outlined images are rendered and stored in the canny_edges folder.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"naunWl-qAO1v"},"outputs":[],"source":["# change blur amount if there are too many lines in your resulting Canny Edge\n","# images (odd numbers only; start with 5, then try 3 if 5 doesn't give you enough lines)\n","\n","# change max_size to be the max dimension of your input images (e.g., if your\n","# input images are 1280x720, set max_size to '1280')\n","\n","!python util/dataset-tools.py \\\n","--input_folder ./experiments/$experiment_name/extracted_frames/ \\\n","--output_folder ./experiments/$experiment_name/canny_edges \\\n","--process_type canny \\\n","--blur_type gaussian \\\n","--blur_amount 1 \\\n","--max_size 854 \\\n","--verbose"]},{"cell_type":"markdown","metadata":{"id":"6C31h725KWLx"},"source":["#### **Put your test images in the correct folders**\n","Run this cell to remove any images currently in test_A, then copy your new Canny Edge Images to the test_A folder."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"9dNWXdhc6d4h"},"outputs":[],"source":["# remove any images currently in test_A (change dataset_name to your dataset_name)\n","!rm -v ./datasets/$dataset_name/test_A/*.png\n","\n","# copy your canny edge images into test_A (change experiment_name and dataset_name)\n","!mv -v ./experiments/$experiment_name/canny_edges/*.png ./datasets/$dataset_name/test_A"]},{"cell_type":"markdown","metadata":{"id":"FTkwfWoOAUbS"},"source":["### 6b. Running the generate command\n","\n","Change your variables as need below, then run the cell to save the changes."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iyljQEtxFjtd"},"outputs":[],"source":["##### REQUIRED: edit these each time each you generate images\n","loadSize = 854 # set this to the width of your input images\n","fineSize = 480  # set this to the height of your input images\n","how_many = 1400  # The number of images you wish to generate. Make sure this is\n","                ## greater than or equal to the number of images you're trying to generate\n","\n","##### OPTIONAL: change these if needed\n","which_epoch = 'latest' # The epoch you wish to generate images from (e.g. '20') (Defult: latest. I recommend this)\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":79518,"status":"ok","timestamp":1648663989653,"user":{"displayName":"Doug Rosman","userId":"12469472491464808109"},"user_tz":300},"id":"V73BNBbr_0R9","outputId":"feaa26db-12f0-4d63-9880-c6161de0e8ca"},"outputs":[],"source":["# Run this cell to generate your images. This may take a few minutes.\n","!python test.py --name=$dataset_name --dataroot=./datasets/$dataset_name --checkpoints_dir checkpoints --results_dir=./results/$experiment_name --which_epoch=$which_epoch --how_many=$how_many --no_instance --label_nc 0 --loadSize=$loadSize --fineSize=$fineSize"]},{"cell_type":"markdown","metadata":{"id":"AITt0XwQr5yX"},"source":["## 7. Creating videos from your generated images"]},{"cell_type":"markdown","metadata":{"id":"tIMtiI26BKVk"},"source":["### 7a. Create video of your test input images\n","_Thread your test_A images together into a video. If your input images used Canny Edge, then this will be a video of your Canny Edge inputs._"]},{"cell_type":"markdown","metadata":{"id":"GLTMmMolJa4Y"},"source":["1. **-i**: the input images _(the filepath to your synthesized images)_\n","1. **-r**: the framerate _(any value between 1-60)_\n","1. **-crf**: the compression quality of the output video _(lower is better, 17-25 is a good range)_\n","1. **_the output filename_**: the last argument; make sure to set this each time you create a new video"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"QEkEOfx5sJ6z"},"outputs":[],"source":["# Change 'input_images_sequence.mp4' to something more descriptive (e.g. edges2cats_input_edges.mp4).\n","# MAKE SURE TO CHANGE THIS EACH TIME TO AVOID OVERWRITING OTHER GENERATED VIDEOS\n","!ffmpeg \\\n","   -pattern_type glob \\\n","   -i \"./results/$experiment_name/$dataset_name/test_latest/images/*_input_label.png\" \\\n","   -r 30 \\\n","   -vcodec libx264 \\\n","   -crf 24 \\\n","   -pix_fmt yuv420p \\\n","   ./generated_videos/input_images_sequence.mp4"]},{"cell_type":"markdown","metadata":{"id":"_X7A6HUsBXx6"},"source":["### 7b. Create video of your synthesized images\n","_Thread your synthesized images together into a video._"]},{"cell_type":"markdown","metadata":{"id":"NoDrHlPaJOpC"},"source":["1. **-i**: the input images _(the filepath to your synthesized images)_\n","1. **-r**: the framerate _(any value between 1-60)_\n","1. **-crf**: the compression quality of the output video _(lower is better, 17-25 is a good range)_\n","1. **_the output filename_**: the last argument; make sure to set this each time you create a new video"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JwZDYXvSBdS6"},"outputs":[],"source":["# Change 'synthesized_output' to something more related to this experiment (e.g. edges2cats_synthesized.mp4).\n","# MAKE SURE TO CHANGE THIS EACH TIME TO AVOID OVERWRITING OTHER GENERATED VIDEOS\n","!ffmpeg \\\n","   -pattern_type glob \\\n","   -i \"./results/$experiment_name/$dataset_name/test_latest/images/*_synthesized_image.png\" \\\n","   -r 30 \\\n","   -vcodec libx264 \\\n","   -crf 23 \\\n","   -pix_fmt yuv420p \\\n","   ./generated_videos/doug2forest.mp4"]}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":["big1f7SvaB7p","NMh4tpXnHAmZ","iCSsYCKXIQZ1","tFDYgh1qIFFz","6C31h725KWLx","FTkwfWoOAUbS"],"provenance":[{"file_id":"https://github.com/dougrosman/pix2pixHD/blob/master/pix2pixHD.ipynb","timestamp":1619808178688}]},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.16"}},"nbformat":4,"nbformat_minor":0}