ReproHack is an initiative that provides the templates and infrastructure to run reproducibility hackathon events. You can read all about its history and and development on the ReproHack webpage.
At a ReproHack, people attempt to reproduce the results from a paper, using information, code and data provided by the author (sometimes called an "artifact"). They then provide feedback to the author on how easy this task was, what information was missing, and what could be improved.
This repository aims to provide instruction and guidance for a small self-guided climate science ReproHack. The material is adapted from the ReproHack Hub Resources.
You can do the ReproHack on your own or as a team - we find that it can be very benefitial to discuss the reviewing process as a team, and to learn from each other how to approach the process of getting an unknown codebase to run, and to reproduce results.
- In your teams, (or individually if you prefer) decide which paper you wish to work on.
- Create user accounts on the Hub for all reviewers that want to be associated with the review.
- Review the ReproHack Guidelines which are giving instructions on what to consider during the review process.
- Start reprohacking!
- Provide feedback to the authors.
There are lots of papers available on the ReproHack Hub. We have selected here a couple of papers that we thought were suitable for a short climate science ReproHack, but feel free to choose a different one to reproduce.
- Southern Ocean deep mixing band emerges from a competition between winter buoyancy loss and upper stratification strength
- Revisiting the zonally asymmetric extratropical circulation of the Southern Hemisphere spring using complex empirical orthogonal functions
- The Polar Transition from Alpha to Beta Regions Set by a Surface Buoyancy Flux Inversion
Follow any instructions/documentation associated with the papers and try and reproduce the work. As you work through your paper, keep in mind the main points on which feedback to the authors will provided, Access, Reproducibility, Documentation and Reusability (see our participant guidelines for more information). It might help to have a look at the Review form before you begin and keep notes during your review. Feel free to use the event hackpad to record general findings you wish to share with the group.
Should you finish reproducing your paper quickly, feel free to explore the work more deeply. For example, you could try and run additional analyses, create new plots or even combine materials with your own or other open materials available on the web!
The most important part of the day is recording our experiences as feedback to the authors. Please make sure to complete a Review feedback form for the paper you've selected, ideally, by the end of the day. Please also remember to be kind and constructive. Reproducibilty is hard and all authors submitting their papers have been incredibly brave. Feel free to browse any public feedback submissions to get inspiration.