Optimising Cascade Structure

[EngEmmanuel]

Thank you so much everyone who has contributed to this repo.
I generally avoid asking these but, I'm sorry, this has been bugging me for a while.

TL;DR

1. How can I chain independently trained nets together?
2. How to load weights into specific nets in a cascade and then continue training the untrained nets?
3. How to train the net at stage s using the output of the previous stage as an input. Not the current mimic method of resizing and blurring the original image from the dataloader

---

• Yes, we can train each part of the cascade separately but how do you combine independently trained nets?
  Example:
  I define a config.yaml containing just one (base) unet and train it. I do a hyperparameter search and find a good combination. I identify a checkpoint I like and would now like to attach another unet for upsampling to create a cascade. However, I don't know the optimal structure for this upsampling unet so would want to train many variations (e.g. wandb sweep) but keep the same base unet.

How do I load the weights for the pre-trained base model and then train only the newly added upsampler?

• Apparently, the upsampling nets don't take in the output from the previous network, but instead mimic it by blurring and resizing the image given by the dataloader, hence allowing the nets in the cascade to be trained individually.

Is there a way to change this behaviour? I would like a scenario where after training the net at stage s-1, I train the net at stage s and input_s = output_{s-1}

• LOSS (Clarity about losses #45)
  In the case when the inputs to later stages are mimicked, can I confirm that that is not true during inference. Meaning the validation loss is calculated using the image generated by passing the outputs of the previous nets as inputs to the next net?

If you think I may be overcomplicating things or have other ideas on how to best optimise the cascades, please share your thoughts :)