Initial commit, porting repository from private repository used in de…

…velopment.

.flake8

@@ -0,0 +1,8 @@
+[flake8]
+ignore = E203, E266, E501, W503, F403, F401, C901, E402
+max-line-length = 120
+max-complexity = 20
+select = B,C,E,F,W,T4,B9
+exclude =
+    srf/*
+    ckernel_fitting/*

.gitignore

@@ -0,0 +1,163 @@
+.idea
+wandb
+ckernel_fitting/wandb
+data/*
+!data/.gitkeep
+outputs/*
+plot-data/*
+plots/*
+runs/*
+
+# Ignore srf/ for now, until Silvia can say something about publishing it
+srf/
+ckernel_fitting/srf_fitting
+
+# Don't include the old kernel fitting summary, but don't remove it either.
+ckernel_fitting/summary_kernelfitting.ipynb
+
+# Don't include raw experiment commands file. Instead, we list commands in experiments/readme.md
+experiments/experimental-commands.txt
+
+# Don't include the invidual run experimental results, since there is no good reference to the original commands/seeds in it.
+experiments/exerperimental-results-all.xlsx
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+.vscode/

.pre-commit-config.yaml

@@ -0,0 +1,10 @@
+repos:
+  - repo: https://github.com/ambv/black
+    rev: stable
+    hooks:
+      - id: black
+        # language_version: python3.6
+  - repo: https://gitlab.com/pycqa/flake8
+    rev: 3.7.9
+    hooks:
+      - id: flake8

LICENSE

@@ -0,0 +1,36 @@
+MIT License
+
+Copyright (c) 2021 David W. Romero & Robert-Jan Bruintjes
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+
+
+BSD 2-Clause Simplified License, for models/cifar_resnet_akamaster.py.
+
+Copyright (c) 2018, Yerlan Idelbayev
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,79 @@
+## FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
+
+This repository contains the source code accompanying the paper:
+
+[FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes](https://arxiv.org/abs/2102.02611)  [[Slides]](#) <br/>**[David W. Romero*](https://www.davidromero.ml/), [Robert-Jan Bruintjes*](https://rjbruin.github.io), [Jakub M. Tomczak](https://jmtomczak.github.io/), [Erik J. Bekkers](https://erikbekkers.bitbucket.io/), [Mark Hoogendoorn](https://www.cs.vu.nl/~mhoogen/) & [Jan C. van Gemert](https://jvgemert.github.io)**.
+
+#### Abstract
+*When designing Convolutional Neural Networks (CNNs), one must select the size of the convolutional kernels before training. Recent works show CNNs benefit from different kernel sizes at different layers, but exploring all possible combinations is unfeasible in practice. A more efficient approach is to learn the kernel size during training. However, existing works that learn the kernel size have a limited bandwidth. These approaches scale kernels by dilation, and thus the detail they can describe is limited. In this work, we propose FlexConv, a novel convolutional operation with which high bandwidth convolutional kernels of learnable kernel size can be learned at a fixed parameter cost. FlexNets model long-term dependencies without the use of pooling, achieve state-of-the-art performance on several sequential datasets, outperform recent works with learned kernel sizes, and are competitive with much deeper ResNets on image benchmark datasets. Additionally, FlexNets can be deployed at higher resolutions than those seen during training. To avoid aliasing, we propose a novel kernel parameterization with which the frequency of the kernels can be analytically controlled. Our novel kernel parameterization shows higher descriptive power and faster convergence speed than existing parameterizations. This leads to important improvements in classification accuracy.*
+
+<img src="flexconv.png" alt="drawing" width="750"/>
+
+### Repository structure
+
+This repository is organized as follows:
+
+* `ckconv` contains the main PyTorch library of our model.
+
+* `models` and `datasets` contain the models and datasets used throughout our experiments;
+
+* `cfg` contains the default configuration of our `run_*.py` scripts, in YAML. We use Hydra with OmegaConf to manage the configuration of our experiments.
+
+* `experiments` contains commands to replicate the experiments from the paper.
+
+* `ckernel_fitting` contains source code to run experiments to approximate convolutional filters via MLPs. Please see `ckernel_fitting/README.md` for further details.
+
+### Using the code
+
+Image classification experiments are run with `run_experiment.py`. Cross-resolution image classification experiments are run with `run_crossres.py`, which trains on the source resolution for `train.epochs` epochs, before finetuning on the target resolution for `cross_res.finetune_epochs` epochs. The code can also be profiled using PyTorch's profiling tools with `run_profiler.py`.
+
+Flags are handled by [Hydra](https://hydra.cc/docs/intro). See `cfg/config.yaml` for all available flags. Flags can be passed as `xxx.yyy=value`.
+
+#### Useful flags
+
+- `net.*` describes settings for the FlexNet models (model definition `models/ckresnet.py`).
+- `kernel.*` describes settings for the MAGNet kernel generators in FlexConvs, for any model definition that uses FlexConvs.
+- `kernel.regularize_params.*` describes settings for the anti-aliasing regularization.
+  - `target=gabor` regularizes without the FlexConv Gaussian mask; `target=gabor+mask` regularized including the FlexConv mask.
+- `mask.*` describes settings for the FlexConv Gaussian mask.
+- `conv.*` describes settings for the convolution to use in FlexNet, excluding MAGNet settings. Can be used to switch between FlexConv, CKConv and regular Conv.
+- `debug=True`: By default, all experiment scripts connect to Weights & Biases to log the experimental results. Use this flag to run without connecting to Weights & Biases.
+- `pretrained` and related flags: Use these to load checkpoints before training, either from a local file (`pretrained` and `pretrained_params.filepath`) or from Weights & Biases (`pretrained_wandb` and associated flags).
+  - In cross-res training, flags can be combined to fine-tune from an existing source res model. Pre-load the final model trained at source resolution (by specifying the correct file), and set `train.epochs=0` so source res training is skipped.
+- `train.do=False`: Only test the model. Useful in combination with pre-training.
+  - Note that this flag doesn't work in cross-res training.
+
+### Install
+
+#### conda *(recommended)*
+In order to reproduce our results, please first install the required dependencies. This can be done by:
+```
+conda env create -f conda_requirements.yaml
+```
+This will create the conda environment `flexconv` with the correct dependencies.
+
+#### pip
+The same conda environment can be created with `pip` by running:
+```
+conda create -n flexconv python=3.8.5
+conda install pytorch==1.9.0 torchvision==0.10.0 torchaudio=0.9.0 cudatoolkit=10.2 -c pytorch
+conda activate flexconv
+pip install -r requirements.txt
+```
+
+### Reproducing experiments
+
+Please see [the Experiments readme](experiments/readme.md) for details on reproducing the paper's experiments.
+
+### Pretrained models
+
+TODO(rjbruin)
+
+### Cite
+If you found this work useful in your research, please consider citing:
+
+_TODO: Insert Arxiv citation here_
+
+### Acknowledgements
+
+We thank Nergis Tömen for her valuable contributions in discussing signal processing principles for FlexConv. We thank Silvia-Laura Pintea for her time and access to code for the Pintea et al (2021) work. David W. Romero is financed as part of the Efficient Deep Learning (EDL) programme (grant number P16-25), partly funded by the Dutch Research Council (NWO). Robert-Jan Bruintjes is financed by the Dutch Research Council (NWO) (project VI.Vidi.192.100). All authors sincerely thank everyone involved in funding this work. This work was carried out on the Dutch national e-infrastructure with the support of SURF Cooperative. We used [Weights & Biases](https://wandb.ai/site) for experiment tracking and visualizations to develop insights for this paper. We thank Yerlan Idelbayev for the use of [the CIFAR ResNet code](github.com/akamaster/pytorch_resnet_cifar10).