reproducibility

Reproducibility considerations for computational notebooks submissions

Reproducibility considerations across all kinds of accepted notebook submissions are not clear cut. When notebooks don’t contain any interactive components, they are expected to be executed sequentially. Such notebooks are “fully-reproducible” in the sense that they are executable and produce the same output when the cells are run in order from top to bottom. On the other extreme, notebooks that contain dynamic elements may lead to different outputs depending on user inputs or interactions. However, this doesn’t mean that they can’t be evaluated for reproducibility. Notebooks with interactive components are expected to contain clear instructions of how exactly users are supposed to interact with their notebooks and what the expected outcomes of those interactions are. This information should appear in the notebook alongside the interactive element when appropriate, but may otherwise be added to the repository’s README. Naturally, the more interactive elements present, the more instructions on use and expected outcomes we expect to see with submissions.