Inference of GCN model on unseen test data

[KristinaUlicna]

PR contribution summary

Why is this PR useful / good for? Please describe the problem(s) you're trying to address.

• Assuming we have a working GCN, this PR provides a new notebook to run inference & evaluate the performance of the GCN on real, annotated data.
• It builds a GraphLabelPredictor which updates the GraphAttrs.XXX_PREDICTION by a vector of numbers for each node and edge:
  

  grace/grace/evaluation/inference.py

  Lines 35 to 41 in 1e60b41

  	for idx, node in G.nodes(data=True):
  	prediction = [int(n_pred[idx].item()), n_probabs[idx].numpy()]
  	node[GraphAttrs.NODE_PREDICTION] = prediction
  	for e_idx, edge in enumerate(G.edges(data=True)):
  	prediction = (int(e_pred[e_idx].item()), e_probabs[e_idx].numpy())
  	edge[-1][GraphAttrs.EDGE_PREDICTION] = prediction

• It allows node Annotation.UNKNOWN relabelling to be separated for nodes & edges.
• This PR additionally annotates other notebooks to make sure that the flow of what is done is clearly explained.

List of proposed changes / linked issues & discussions

• Resolves [DEVELOPMENT] Split the unknown masking for nodes & edges #221
• Resolves [DEVELOPMENT] Rename normalise to normalize for tests to pass #224
• Resolves Measure area under [receiver-operator, precision-recall] curve metrics #194
• Resolves Run model inference on random graph + real annotated data #188
• Resolves Transform inference procedure from notebook to script #189
• Resolves [FEATURE REQUEST] Implement inference function #101
• Resolved Graph pre-process #133
• Resolves [DEVELOPMENT] Repurpose 2x2 confusion matrix for use outside of object metrics #225

What should a reviewer concentrate their feedback on?

• ✅ Scripts to check
• 🏃 Notebooks to run (especially infer_predictions.ipynb, others only have minor documentation / comments changes)
• 💻 Code quality
• 📝 Everything looks OK?

What type of PR is this? (check all applicable)

• 🪄 Feature
• 🐛 Bug fix
• #️⃣ Documentation / code annotation
• 🔥 Performance Improvements

Added tests?

• 🙋 no, because I need some help

---

PR review summary

Describe what this PR does & how you reviewed the individual items, where needed:

Some helper checks to tick off:

• Focus on image annotation
• Focus on model training
• Could any optimization be applied?
• Is there any redundant code?
• Are there any spelling errors?

In conclusion, after my review, I'd like to:

• 🙋 ask some clarifying questions
• 🙅 suggest some specific changes

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[crangelsmith]

I get an error trying to run this cell:
ModuleNotFoundError: No module named 'grace.evaluation'

I think there is a missing init.py in the evaluation directory that would make this work.

[KristinaUlicna]

Adding the __init__.py in the next commit 🚀

[crangelsmith]

If we want people to be able to run this easily maybe we need to add the line to download the resnet before or add a comment to point to where this is.

[KristinaUlicna]

Great point! I included the instructions on how to download the ResNet locally in the training subfolder README:

grace/grace/training/README.md

Lines 67 to 78 in 488f60b

	### Downloading the feature extractor:
	In case you decide to use a pre-trained image classifier, such as resnet-152, you can use this snippet to import the model, load the default weights & download the model:
	```sh
	import torch
	from grace.models.feature_extractor import resnet
	resnet_model = resnet(resnet_type="resnet152")
	extractor_fn = "/path/to/your/feature/extractor/resnet152.pt"
	torch.save(resnet_model, extractor_fn)
	```

Would you like to see it added in the notebook, too?

[crangelsmith]

In general we should remove path to personal directories if we want these notebooks to be run easily for everyone, we could add a new variable called: input_paht = /path/to/data/ to be replaced by the user.

Also, we need to put this data somewhere and easy to download.

[KristinaUlicna]

Fully agreed, we had this problem in the examples\ folder where I replaced the path for a user-specified input:

grace/examples/show_data.py

Lines 13 to 20 in 488f60b

	# e.g. /Users/kulicna/Desktop/dataset/shape_squares/MRC_Synthetic_File_000.mrc
	IMAGE_PATH = Path(
	input(
	"Enter absolute path to your file "
	"(e.g. /Users/path/to/your/data/image.mrc, omit ''): "
	)
	)

I'm not sure how to do this from the notebook tho...

[crangelsmith]

Hi @KristinaUlicna, the notebook runs except for the minor comments above but I'm a bit confused about why is the inference done on a notebook and not added to the run.py script as part of the pipeline. Couldn't we just add an eval option to the config file and do this in run.py after training?

[crangelsmith]

Also, it would be good to visualise the resulting graph on inference, as described in #222

[KristinaUlicna]

Also, it would be good to visualise the resulting graph on inference, as described in #222

Coming up in PR #229 ! 🚀