Integretion of YoloV5 as possible pose estimation model

.gitignore

@@ -129,3 +129,15 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+*.out
+results/
+minimal_patches/
+parallel_compute/
+*.pickle
+*.pkl
+*.sh
+*.sbatch
+*.pth
+test/
+src/custom_patches/diffusion_patches_100.npy

NATALIE_README.md

@@ -0,0 +1,52 @@
+## YOLO Adversarial Patches
+All changes are here: https://github.com/natalieh235/flying_adversarial_patch/tree/yolo
+
+### src/attacks.py
+`python3 attacks.py --file settings.yaml --task task_num`
+- added task_num param (int) for generating many patches in parallel
+- changed targeted_attack_joint to use the YOLOBounding class to predict the person coordinates instead
+- currently, only targets that are valid in the image pixel context are generated
+
+### src/run_attacks.py
+- generates many patches in parallel using multiprocessing module
+- change `start_task_id`, `end_task_id`, and `num_runs`
+- change `OUTPUT_PATH` to save log files (stderr and stdout)
+
+### src/test_yolo_patches.py 
+`python3 src/test_yolo_patches.py --start start_patch_id --end end_patch_id`
+- functions for calculating error of yolo patches and displaying them (params are starting and ending patch_id)
+- expects that patches are saved in PATH folder in the same format `attacks.py` saves them in (npy files), and each patch has a corresponding `settings_patchid.yaml` file
+<br><br/>
+For each patch:
+1. loads the patch from `PATH/last_patch_id.npy`
+2. loads the optimized position from `PATH/position_norm_id.npy`
+3. loads the targets from `PATH/settings.yaml`
+4. for each target, generates an image from the dataset with the patch placed at the optimized position for that target
+5. for each patch, calculates the MSE between the center of the highest confidence bounding box and the actual target for a random image in the dataset for all targets
+6. for each patch/target pair, saves an image in `RESULT_PATH/yolo_boxes_{patch_id}_target_{target_id}.png` that displays the patch placed on the image, the target, and the predicted YOLO bounding boxes (red is highest confidence, then blue, green, purple) <br> See example in `test_yolo_example.png`
+
+The error printed isn't the error over the entire dataset for all patches, just the average loss for all patches when placed on ONE randomly selected image from the Frontnet dataset.
+
+### src/yolo_bounding.py
+has class `YOLOBox`, a wrapper around the YOLO ultralytics model that skips the normal NMS step and takes a weighted average to calculate the highest confidence bounding box so that the output is differentiable
+- uses the ultralytics YOLOv5 nano module with `autoshape=False` so that it can take batched inputs
+- the score assigned to each box is the product of the objectness score and the person_class score, calculated in `extract_boxes_and_scores`
+- model(input) outputs bounding box coordinates of the highest confidence box for each image
+- used in `attacks.py` to train patches against the YOLO model
+
+### src/camera.py
+- wrapper class around camera functions, takes a cam config file as param
+Functions: 
+- `xyz_from_bb` calculates xyz coords from a bounding box
+- `tensor_xyz_from_bb` same thing, but with tensors
+- `point_from_xyz` calculates image point from xyz coords
+
+### src/create_dataset.py
+- creates a pkl file that stores patch data
+- call `save_pickle(start_patch_id, end_patch_id)` to create a pkl file for patches from start_id to end_id inclusive
+- saves only the best target for each patch
+
+### Other
+All the patches I generated (653, but only 578 with valid targets) are saved in `all_yolo_patches.pkl`
+Diffusion model trained for 1000 epochs, 1000 denoising steps (approx 23 min training) is saved in `yolo_conditioned_unet_80x80_1000_3256i_255.pth ` (uploaded to the TUB drive) <br>
+Samples from that model are saved in `samples_conditioning_4_80x80.png`

camera/__init__.py

@@ -0,0 +1 @@
+print("hi")

dataset.sh

@@ -0,0 +1,8 @@
+#!/bin/sh
+#SBATCH -c 4 
+#SBATCH --gres=gpu:1 
+#SBATCH -p gpu
+#SBATCH --time=00:30:00
+
+source /home/hanfeld/.front-env/bin/activate
+python src/attacks.py --file $1

dataset.yaml

@@ -0,0 +1,30 @@
+path: 'results/dataset'
+# +x is distance
+# +y is left
+# +z is top
+targets:
+ x : [ 1.0, 1.5, 0.5, 2.0, 1.0, 1.0]
+ y : [ 0.0, 0.0, 0.0, 0.0, 1.0, -1.0]
+ z : [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
+stlc: # small-time locally controllability
+ offsets:
+ # base
+ - target: [0,0,0]
+ position: [0,0,0]
+lr_pos: 0.001
+lr_patch: 0.01
+prob_weight: 1.0
+num_hl_iter: 50
+num_pos_restarts: 3
+num_pos_epochs: 1
+num_patch_epochs: 1
+batch_size: 32
+num_patches: 1
+scale_min: 0.2
+scale_max: 0.7
+patch: 
+ mode: 'random' # face, white, random
+ size: [80, 80] # patch size in [height, width]
+ path: 'src/custom_patches/custom_patch_96x160.npy' # if mode == 'face' set a path, otherwise path is ignored
+mode: "joint" # fixed, joint, split, hybrid
+quantized: false