Organizing Committee
Abstract

In addition to advancing research and discovery in pure and applied mathematics, computation is pervasive across the sciences and now computational research results are more crucial than ever for public policy, risk management, and national security. Reproducibility of carefully documented experiments is a cornerstone of the scientific method, and yet is often lacking in computational mathematics, science, and engineering. Setting and achieving appropriate standards for reproducibility in computation poses a number of interesting technological and social challenges. The purpose of this workshop is to discuss aspects of reproducibility most relevant to the mathematical sciences among researchers from pure and applied mathematics from academics and other settings, together with interested parties from funding agencies, national laboratories, professional societies, and publishers. This will be a working workshop, with relatively few talks and dedicated time for breakout group discussions on the current state of the art and the tools, policies, and infrastructure that are needed to improve the situation. The groups will be charged with developing guides to current best practices and/or white papers on desirable advances.

Final Report

This report was developed collaboratively by the ICERM workshop participants, and compiled and edited by the organizers.

Setting the Default to Reproducible: Reproducibility in Computational and Experimental Mathematics

Confirmed Speakers & Participants

Talks will be presented virtually or in-person as indicated in the schedule below.

  • Speaker
  • Poster Presenter
  • Attendee
  • Virtual Attendee

Workshop Schedule

Monday, December 10, 2012
TimeEventLocationMaterials
9:00 - 9:30am ESTIssues in Reproducibility - Randy LeVeque, University of Washington11th Floor Lecture Hall
9:30 - 10:00am ESTWhat does it take to do reproducible computational science? What stands in our way? - Bill Rider, Sandia National Laboratories11th Floor Lecture Hall 
10:00 - 10:30am ESTExploratory Experimentation in Mathematics - Jon Borwein, Centre for Computer Assisted Research Mathematics and its Applications11th Floor Lecture Hall
11:00 - 11:30am ESTA brief history of the reproducibility movement - Victoria Stodden, Columbia University11th Floor Lecture Hall
11:30 - 12:00pm ESTOpen Discussion on issues11th Floor Lecture Hall 
1:30 - 2:00pm ESTTBA - David Donoho, Stanford University11th Floor Lecture Hall 
2:00 - 2:30pm ESTTrust in computation and mathematical proof - Tom Hales, University of Pittsburgh11th Floor Lecture Hall 
2:30 - 3:00pm ESTOpen Discussion on issues11th Floor Lecture Hall 
3:00 - 3:30pm ESTCoffee/Tea Break11th Floor Collaborative Space 
3:30 - 4:00pm ESTReproducibility issues important to applications - James Glimm, Stony Brook University  
4:00 - 4:30pm ESTHigh-Precision Computation and Reproducibility - David Bailey, Lawrence Berkeley National Laboratory11th Floor Lecture Hall
4:30 - 5:30pm ESTOpen discussion on issues11th Floor Lecture Hall 
5:30 - 7:00pm ESTWelcome Reception11th Floor Collaborative Space 
Tuesday, December 11, 2012
TimeEventLocationMaterials
9:00 - 9:30am ESTR & D at Google- Hybrid Research, Web-Scale Development - Peter Norvig, Google Inc11th Floor Lecture Hall
9:30 - 10:00am ESTRecent strategies and challenges for reproducibility in global climate modeling - Katherine Evans, Oak Ridge National Laboratory11th Floor Lecture Hall 
10:00 - 10:30am ESTIs mathematics a social machine? - Ursula Martin, Queen Mary University of London11th Floor Lecture Hall 
10:30 - 11:00am ESTCoffee/Tea Break11th Floor Collaborative Space 
11:00 - 12:30pm ESTPanel discussion with program managers / open discussion - Neil Chue Hong, University of Edinburgh; Ursula Martin, Queen Mary University of London; Chris Mentzel, Gordon and Betty Moore Foundation; Karen Pao, Department of Energy; Thomas Russell, NSF 11th Floor Lecture Hall 
12:30 - 2:00pm ESTBreak for Lunch  
2:00 - 2:30pm ESTWelcome to the collaborative web - Tim Clem, GitHub11th Floor Lecture Hall
2:30 - 3:00pm ESTSoftware Carpentry- What We've Done, What We've Learned, Where We're Going - Greg Wilson, Software Carpentry11th Floor Lecture Hall
3:00 - 3:30pm ESTCoffee/Tea Break11th Floor Collaborative Space 
3:30 - 4:00pm ESTWhat's happening with the World Digital Mathematics Library? - Peter Olver, University of Minnesota11th Floor Lecture Hall 
4:00 - 5:30pm ESTPanel discussion on journal policies / experiences - Ron Boisvert, National Institute of Standards and Technology; Nicolas Limare, École Normale Supérieure de Cachan; Neil Calkin, Clemson University; Randy LeVeque, University of Washington11th Floor Lecture Hall 
Wednesday, December 12, 2012
TimeEventLocationMaterials
9:00 - 9:30am ESTPublishing Results from MATLAB - Loren Shure, The MathWorks, Inc.11th Floor Lecture Hall
9:30 - 10:00am ESTSage notebooks - William Stein, University of Washington11th Floor Lecture Hall 
10:00 - 10:30am ESTIPython- tools for the lifecycle of research computing - Fernando Perez, University of California, Berkeley11th Floor Lecture Hall
10:30 - 10:35am ESTGroup Photo11th Floor Lecture Hall 
10:35 - 11:00am ESTCoffee/Tea Break11th Floor Collaborative Space 
11:00 - 11:30am ESTBalancing Performance and Reproducibility with Intel MKL CNR - Todd Rosenquist, Intel Corporation11th Floor Lecture Hall 
11:30 - 12:30pm ESTLightning talks - Session 111th Floor Lecture Hall
12:30 - 2:00pm ESTBreak for Lunch  
2:00 - 2:15pm ESTJoint session to discuss working groups and set goals11th Floor Lecture Hall 
2:15 - 4:30pm ESTIn parallel: Working groups on policy, best practices - Opportunity for participants to meet with developers for discussion of tools, attempt to apply to specific problems, etc.  
4:30 - 5:30pm ESTReport back from working groups11th Floor Lecture Hall 
7:00 - 8:30pm ESTPoster Session & Dessert Reception11th Floor Lecture Hall 
Thursday, December 13, 2012
TimeEventLocationMaterials
9:00 - 9:30am ESTMaking Computations and Publications Reproducible with VisTrails - Juliana Freire, New York University11th Floor Lecture Hall
9:30 - 10:00am ESTReproducible research as a community effort- experience of the Madagascar project - Sergey Fomel, University of Texas at Austin11th Floor Collaborative Space
10:00 - 10:30am ESTSumatra: a toolkit for provenance capture and reuse - Andrew Davison, Centre National de la Recherche Scientifique (CNRS)11th Floor Lecture Hall
10:30 - 11:00am ESTCoffee/Tea Break11th Floor Collaborative Space 
11:00 - 12:30pm ESTLightning talks - Session 211th Floor Lecture Hall
12:30 - 12:35pm ESTSurvey Distribution11th Floor Lecture Hall 
12:35 - 2:00pm ESTBreak for Lunch  
2:00 - 2:15pm ESTJoint session to discuss working groups11th Floor Lecture Hall 
2:15 - 4:30pm ESTIn parallel: Working groups on policy, best practices - Opportunity for participants to meet with developers for discussion of tools, attempt to apply to specific problems, etc.11th Floor Lecture Hall 
4:30 - 5:30pm ESTReport back from working groups11th Floor Lecture Hall 
Friday, December 14, 2012
TimeEventLocationMaterials
9:00 - 9:30am ESTInteractive theorem proving, automated reasoning, and mathematical computation - Jeremy Avigad, Carnegie Mellon University11th Floor Lecture Hall
9:30 - 10:00am ESTclearScience- tools for communicating in the computational sciences - Brian M. Bot, Sage Bionetworks11th Floor Lecture Hall 
10:00 - 10:30am ESTCase Study: Creating a reproducibility repository for a paper with large-scale HPC results - Aron Ahmadia, Columbia University11th Floor Lecture 
10:30 - 11:00am ESTCoffee/Tea Break11th Floor Collaborative Space 
11:00 - 12:30pm ESTFinal joint session, open discussion of next steps11th Floor Lecture Hall 
12:30 - 1:30pm ESTBox Lunches Available11th Floor Collaborative Space 
1:30 - 3:00pm ESTBreakout groups to work on workshop outcomes  
3:00 - 3:30pm ESTCoffee/Tea Break11th Floor Collaborative Space 
3:30 - 5:00pm ESTBreakout groups to work on workshop outcomes  

Lecture Videos

Sage notebooks

William Stein
University of Washington
December 12, 2012

Issues in Reproducibility

Randall LeVeque
University of Washington
December 10, 2012