TSNE vis: update the model & embeddings #102
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some improvements in visualisation relevant to #62.
It's using a 'all-MiniLM-L6-v2' model (87Mb instead of 418Mb, the same 512 context size) that is faster & seems to provide better visualisation.
Before: the previous model
![Screenshot 2023-04-30 at 18 24 44](https://user-images.githubusercontent.com/5582506/235364382-fb5bcd43-eff6-4c76-8a7d-d3b4400621ae.png)
The previous model using both, abstracts and titles.
![Screenshot 2023-04-30 at 18 25 31](https://user-images.githubusercontent.com/5582506/235364410-11aaa931-3327-4aea-9ce2-f494f2188e1e.png)
The new model (batch size 1, abstracts only)
![Screenshot 2023-04-30 at 18 26 51](https://user-images.githubusercontent.com/5582506/235364463-543aaa33-76c5-4142-9caa-e3b5d1066953.png)
After: the new model + titles + batched
![Screenshot 2023-04-30 at 18 25 04](https://user-images.githubusercontent.com/5582506/235364390-a98b601b-fb48-413f-b987-f8fc02b74613.png)
I tied UMAP for it and the results seems less interesting (but didn't experiment much)
UMAP
I also tried using a larger 420Mb model fine-tuned on scientific papers from SciRepEval - allenai/specter2 \w proximity adaptor that takes 1.5min vs 30sec of the above. It can't be switched though the CLI only, as it requires loading an adaptor.
specter2 TSNE
Let me know what you think and which one do you prefer!