Skip to content

Compare Tree-LSTMs with and without a bottleneck to obtain a scalable compositionality metric (BCM).

Notifications You must be signed in to change notification settings

vernadankers/bottleneck_compositionality_metric

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Bottleneck Compositionality Metric (BCM)

Section 4: Arithmetic evaluation

Launch all required model training using the tree_lstms/scripts/submit_train_arithmetic.sh <SETUP> script.

  • Launch it twice, using bcm_pp and bcm_tt as setups.
  • The latter can only be started if the former finished running.
  • Afterwards, reproduce the graphs contained in the paper in section 4 using the analysis/arithmetic/visualise.ipynb notebook.

Section 5.1-5.3: Sentiment analysis

Launch all required model training using the tree_lstms/scripts/submit_train_sentiment.sh <SETUP> script.

  • Launch it twice, using bcm_pp and bcm_tt as setups.
  • The latter can only be started if the former finished running.

Afterwards, reproduce the graphs contained in the paper using the following notebooks:

  1. Figure 5: analysis/sentiment/visualise_task_performance.ipynb
  2. Figure 6: you can obtain the predictions from analysis/sentiment/run_baseline.py. The figure was created by hand afterwards.
  3. Figure 7: analysis/sentiment/compare_to_baseline.ipynb

Section 5.4: Sentiment analysis, example use cases

Launch all required model training using the sentiment_training/scripts/submit_train.sh script. Afterwards, reproduce the graphs contained in the paper using the analysis/sentiment/visualise_task_performance.ipynb notebook.

Appendix B.2:

First obtain:

  1. Topographic similarity: python topographic_similarity.py
  2. Memorization: memorization values of Zhang et al., that extracts memorization values as demonstrated in their notebook https://github.com/xszheng2020/memorization/blob/master/sst/05_Atypical_Phase.ipynb

Then visualise Figures 15 and 16 with the analysis/sentiment/alternative_metrics.ipynb notebook.

@inproceedings{dankers2022recursive,
  title={Recursive Neural Networks with Bottlenecks Diagnose (Non-) Compositionality},
  author={Dankers, Verna and Titov, Ivan},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2022},
  pages={4361--4378},
  year={2022}
}

About

Compare Tree-LSTMs with and without a bottleneck to obtain a scalable compositionality metric (BCM).

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published