internal change

PiperOrigin-RevId: 277064710

README.md

@@ -12,7 +12,7 @@ It supports a variety of different models, metrics and data sets:
 disentanglement_lib was created by Olivier Bachem and Francesco Locatello at Google Brain Zurich for the large-scale empirical study
 
 > [**Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations.**](https://arxiv.org/abs/1811.12359)
-> *Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem*. arXiv preprint, 2018.
+> *Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem*. ICML (Best Paper Award), 2019.
 
 The code is tested with Python 3 and is meant to be run on Linux systems (such as a [Google Cloud Deep Learning VM](https://cloud.google.com/deep-learning-vm/docs/)).
 It uses TensorFlow, Scipy, Numpy, Scikit-Learn, TFHub and Gin.
@@ -217,18 +217,64 @@ dlib_download_data
 ```
 Other datasets will be made available at the later stages of the competition. For more information on the competition kindly visit the [competition website](https://www.aicrowd.com/challenges/neurips-2019-disentanglement-challenge). More information about the dataset can be found [here](https://github.com/rr-learning/disentanglement_dataset) or in the arXiv preprint [On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset](https://arxiv.org/abs/1906.03292).
 
+
+## Abstract reasoning experiments
+
+The library also includes the code used for the experiments of the following paper in the `disentanglement_lib/evaluation/abstract_reasoning` subdirectory:
+> [**Are Disentangled Representations Helpful for Abstract Visual Reasoning?**](https://arxiv.org/abs/1905.12506)
+> *Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, Olivier Bachem*. NeurIPS, 2019.
+
+The experimental protocol consists of two parts:
+First, to train the disentanglement models, one may use the  the standard replication pipeline (`dlib_reproduce`), for example via the following command:
+
+```
+dlib_reproduce --model_num=<?> --study=abstract_reasoning_study_v1
+```
+where `<?>` should be replaced with a model index between 0 and 359 which
+corresponds to the ID of which model to train.
+
+Second, to train the abstract reasoning models, one can use the automatically installed pipeline `dlib_reason`.
+To configure the model, copy and modify `disentanglement_lib/config/abstract_reasoning_study_v1/stage2/example.gin` as needed.
+Then, use the following command to train and evaluate an abstract reasoning model:
+
+```
+dlib_reason --gin_config=<?> --input_dir=<?> --output_dir=<?>
+```
+The results can then be found in the `results` subdirectory of the output directory.
+
+## Fairness experiments
+
+The library also includes the code used for the experiments of the following paper in `disentanglement_lib/evaluation/metrics/fairness.py`:
+> [**On the Fairness of Disentangled Representations**](https://arxiv.org/abs/1905.13662)
+> *Francesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schoelkopf, Olivier Bachem*. NeurIPS, 2019.
+
+To train and evaluate all the models, simply use the following command:
+
+```
+dlib_reproduce --model_num=<?> --study=fairness_study_v1
+```
+where `<?>` should be replaced with a model index between 0 and 12'599 which
+corresponds to the ID of which model to train.
+
+If you only want to reevaluate an already trained model using the evaluation protocol of the paper, you may use the following command:
+
+```
+dlib_reproduce --model_dir=<model_output_directory> --output_directory=<output> --study=fairness_study_v1
+```
+
 ## Feedback
 Please send any feedback to bachem@google.com and francesco.locatello@tuebingen.mpg.de.
 
 ## Citation
 If you use **disentanglement_lib**, please consider citing:
 
 ```
-@article{locatello2018challenging,
-  title={Challenging common assumptions in the unsupervised learning of disentangled representations},
-  author={Locatello, Francesco and Bauer, Stefan and Lucic, Mario and Gelly, Sylvain and Sch{\"o}lkopf, Bernhard and Bachem, Olivier},
-  journal={arXiv preprint arXiv:1811.12359},
-  year={2018}
+@inproceedings{locatello2019challenging,
+  title={Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations},
+  author={Locatello, Francesco and Bauer, Stefan and Lucic, Mario and Raetsch, Gunnar and Gelly, Sylvain and Sch{\"o}lkopf, Bernhard and Bachem, Olivier},
+  booktitle={International Conference on Machine Learning},
+  pages={4114--4124},
+  year={2019}
 }
 ```
 

bin/dlib_reason

@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Script to run the abstract reasoning step."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from absl import app
+from absl import flags
+from disentanglement_lib.evaluation.abstract_reasoning import reason
+
+FLAGS = flags.FLAGS
+flags.DEFINE_string("input_dir", None, "Directory to take trained model from.")
+flags.DEFINE_string("output_dir", None,
+                    "Directory to save extracted representation to.")
+flags.DEFINE_multi_string("gin_config", [],
+                          "List of paths to the config files.")
+flags.DEFINE_multi_string("gin_bindings", [],
+                          "Newline separated list of Gin parameter bindings.")
+flags.DEFINE_boolean("overwrite", False,
+                     "Whether to overwrite output directory.")
+
+
+def main(unused_argv):
+  reason.reason_with_gin(FLAGS.input_dir, FLAGS.output_dir, FLAGS.overwrite,
+                         FLAGS.gin_config, FLAGS.gin_bindings)
+
+
+if __name__ == "__main__":
+  app.run(main)

bin/dlib_reproduce

@@ -14,7 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Pipeline to reproduce fixed models and evaluation protocols."""
+"""Pipeline to reproduce fixed models and evaluation protocols.
+
+This is the main pipeline for the reasoning step in the paper:
+Are Disentangled Representations Helpful for Abstract Visual Reasoning?
+Sjoerd van Steenkiste, Francesco Locatello, Juergen Schmidhuber, Olivier Bachem.
+NeurIPS, 2019.
+https://arxiv.org/abs/1905.12506
+"""
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function

bin/dlib_tests

@@ -26,6 +26,8 @@ python -m disentanglement_lib.utils.convolute_hub_test
 python -m disentanglement_lib.utils.results_test
 python -m disentanglement_lib.postprocessing.postprocess_test
 python -m disentanglement_lib.data.ground_truth.util_test
+python -m disentanglement_lib.evaluation.abstract_reasoning.reason_test
+python -m disentanglement_lib.evaluation.abstract_reasoning.relational_layers_test
 python -m disentanglement_lib.evaluation.evaluate_test
 python -m disentanglement_lib.evaluation.metrics.beta_vae_test
 python -m disentanglement_lib.evaluation.metrics.factor_vae_test

disentanglement_lib/config/abstract_reasoning_study_v1/__init__.py

@@ -0,0 +1,15 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/__init__.py

@@ -0,0 +1,15 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/beta_vae_sklearn.gin

@@ -0,0 +1,21 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @beta_vae_sklearn
+dataset.name="auto"
+evaluation.random_seed = 0
+beta_vae_sklearn.batch_size=64
+beta_vae_sklearn.num_train=10000
+beta_vae_sklearn.num_eval=5000

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/dci.gin

@@ -0,0 +1,20 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @dci
+dataset.name="auto"
+evaluation.random_seed = 0
+dci.num_train=10000
+dci.num_test=5000

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/downstream_task_boosted_trees.gin

@@ -0,0 +1,21 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @downstream_task
+dataset.name="auto"
+evaluation.random_seed = 0
+downstream_task.num_train=[10, 100,1000,10000]
+downstream_task.num_test=5000
+predictor.predictor_fn = @gradient_boosting_classifier

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/downstream_task_logistic_regression.gin

@@ -0,0 +1,21 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @downstream_task
+dataset.name="auto"
+evaluation.random_seed = 0
+downstream_task.num_train=[10, 100,1000,10000]
+downstream_task.num_test=5000
+predictor.predictor_fn = @logistic_regression_cv

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/factor_vae_metric.gin

@@ -0,0 +1,24 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @factor_vae_score
+dataset.name="auto"
+evaluation.random_seed = 0
+factor_vae_score.num_variance_estimate=10000
+factor_vae_score.num_train=10000
+factor_vae_score.num_eval=5000
+factor_vae_score.batch_size=64
+prune_dims.threshold = 0.05
+

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/mig.gin

@@ -0,0 +1,21 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @mig
+dataset.name="auto"
+evaluation.random_seed = 0
+mig.num_train=10000
+discretizer.discretizer_fn = @histogram_discretizer
+discretizer.num_bins = 20

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/modularity_explicitness.gin

@@ -0,0 +1,22 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @modularity_explicitness
+dataset.name="auto"
+evaluation.random_seed = 0
+modularity_explicitness.num_train=10000
+modularity_explicitness.num_test=5000
+discretizer.discretizer_fn = @histogram_discretizer
+discretizer.num_bins = 20

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/sap_score.gin

@@ -0,0 +1,21 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @sap_score
+dataset.name="auto"
+evaluation.random_seed = 0
+sap_score.num_train=10000
+sap_score.num_test=5000
+sap_score.continuous_factors=False

disentanglement_lib/config/abstract_reasoning_study_v1/stage1/metric_configs/unsupervised.gin

@@ -0,0 +1,21 @@
+# coding=utf-8
+# Copyright 2018 The DisentanglementLib Authors.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+evaluation.evaluation_fn = @unsupervised_metrics
+dataset.name="auto"
+evaluation.random_seed = 0
+unsupervised_metrics.num_train=10000
+discretizer.discretizer_fn = @histogram_discretizer
+discretizer.num_bins = 20