Skip to content

Latest commit

 

History

History
74 lines (52 loc) · 2.91 KB

README.md

File metadata and controls

74 lines (52 loc) · 2.91 KB
Raw

An Offline Metric for the Debiasedness of Click Models

Source code for the SIGIR 2023 paper An Offline Metric for the Debiasedness of Click Models. For a standalone implementation of the proposed CMIP metric, see this repository.

Setup

1. Virtual environment with Conda

Dependency management

  1. Setup conda / miniconda on your device.
  2. Create environment and install dependencies: conda env create -f environment.yaml
  3. Activating environment: conda activate sigir-cmip

2. Experiments

All experimental runs are documented inside the scripts/ directory. To execute an experiment:

  1. Make the scripts executable: chmod +x ./scripts/*
  2. Run a script locally use, e.g.: ./scripts/graded-pbm.sh
  3. To execute a script on a SLURM cluster add: ./scripts/graded-pbm.sh +launcher=slurm
  4. You can configure the SLURM resources in: config/launcher/slurm.yaml

Documentation of each experiment can be found inside the scripts.

3. Pre-commit

Automatically format and lint modified files in commit.

  1. Make sure you activate your environment
  2. Initialize pre-commit: pre-commit install
  3. (Optional) Run on checks against all files (not just changed): pre-commit run --all-files

4. Datasets

The project automatically downloads the dataset used in this work to: ~/.ltr_datasets.

  1. You can change the directory by modifying the base_dir variable in: config/env.yaml
  2. To avoid downloading datasets, you can directly place the original .zip file into the download subdirectory, e.g.: ~/.ltr_datasets/download/MSLR-WEB30K.zip

5. Logging

Log metrics with Weights & Biases.

  1. Make sure you activate your environment
  2. Log into Weights & Biases before your first run: wandb login
  3. Add your wandb entity and project name inside config/config.yaml

6. Visualizations

All code for plotting is in the notebooks/ directory. The code requires the results to be logged to Weights & Biases (Section 5).

  1. Make sure you activate your environment
  2. Start a jupyterlab server: python -m jupyterlab
  3. Add wandb parameters in notebook header and run all cells

Hyperparameters and configuration

You can find a list of model parameters and training configurations under config/.

Reference

@inproceedings{Deffayet2023Debiasedness,
  author = {Romain Deffayet and Philipp Hager and Jean-Michel Renders and Maarten de Rijke},
  title = {An Offline Metric for the Debiasedness of Click Models},
  booktitle = {Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR`23)},
  organization = {ACM},
  year = {2023},
}

License

This project uses the MIT license.