Organizing Committee
Abstract

In addition to advancing research and discovery in pure and applied mathematics, computation is pervasive across the sciences and now computational research results are more crucial than ever for public policy, risk management, and national security. Reproducibility of carefully documented experiments is a cornerstone of the scientific method, and yet is often lacking in computational mathematics, science, and engineering. Setting and achieving appropriate standards for reproducibility in computation poses a number of interesting technological and social challenges. The purpose of this workshop is to discuss aspects of reproducibility most relevant to the mathematical sciences among researchers from pure and applied mathematics from academics and other settings, together with interested parties from funding agencies, national laboratories, professional societies, and publishers. This will be a working workshop, with relatively few talks and dedicated time for breakout group discussions on the current state of the art and the tools, policies, and infrastructure that are needed to improve the situation. The groups will be charged with developing guides to current best practices and/or white papers on desirable advances.

Final Report

This report was developed collaboratively by the ICERM workshop participants, and compiled and edited by the organizers.

Setting the Default to Reproducible: Reproducibility in Computational and Experimental Mathematics

Confirmed Speakers & Participants

Workshop Schedule

Monday, December 10, 2012
TimeEventLocationMaterials
9:00 - 9:30Issues in Reproducibility - Randy LeVeque, University of Washington11th Floor Lecture Hall
9:30 - 10:00What does it take to do reproducible computational science? What stands in our way? - Bill Rider, Sandia National Laboratories11th Floor Lecture Hall 
10:00 - 10:30Exploratory Experimentation in Mathematics - Jon Borwein, Centre for Computer Assisted Research Mathematics and its Applications11th Floor Lecture Hall
11:00 - 11:30A brief history of the reproducibility movement - Victoria Stodden, Columbia University11th Floor Lecture Hall
11:30 - 12:00Open Discussion on issues11th Floor Lecture Hall 
1:30 - 2:00TBA - David Donoho, Stanford University11th Floor Lecture Hall 
2:00 - 2:30Trust in computation and mathematical proof - Tom Hales, University of Pittsburgh11th Floor Lecture Hall 
2:30 - 3:00Open Discussion on issues11th Floor Lecture Hall 
3:00 - 3:30Coffee/Tea Break11th Floor Collaborative Space 
3:30 - 4:00Reproducibility issues important to applications - James Glimm, Stony Brook University  
4:00 - 4:30High-Precision Computation and Reproducibility - David Bailey, Lawrence Berkeley National Laboratory11th Floor Lecture Hall
4:30 - 5:30Open discussion on issues11th Floor Lecture Hall 
5:30 - 7:00Welcome Reception11th Floor Collaborative Space 
Tuesday, December 11, 2012
TimeEventLocationMaterials
9:00 - 9:30R & D at Google- Hybrid Research, Web-Scale Development - Peter Norvig, Google Inc11th Floor Lecture Hall
9:30 - 10:00Recent strategies and challenges for reproducibility in global climate modeling - Katherine Evans, Oak Ridge National Laboratory11th Floor Lecture Hall 
10:00 - 10:30Is mathematics a social machine? - Ursula Martin, Queen Mary University of London11th Floor Lecture Hall 
10:30 - 11:00Coffee/Tea Break11th Floor Collaborative Space 
11:00 - 12:30Panel discussion with program managers / open discussion - Neil Chue Hong, University of Edinburgh; Ursula Martin, Queen Mary University of London; Chris Mentzel, Gordon and Betty Moore Foundation; Karen Pao, Department of Energy; Thomas Russell, NSF 11th Floor Lecture Hall 
12:30 - 2:00Break for Lunch  
2:00 - 2:30Welcome to the collaborative web - Tim Clem, GitHub11th Floor Lecture Hall
2:30 - 3:00Software Carpentry- What We've Done, What We've Learned, Where We're Going - Greg Wilson, Software Carpentry11th Floor Lecture Hall
3:00 - 3:30Coffee/Tea Break11th Floor Collaborative Space 
3:30 - 4:00What's happening with the World Digital Mathematics Library? - Peter Olver, University of Minnesota11th Floor Lecture Hall 
4:00 - 5:30Panel discussion on journal policies / experiences - Ron Boisvert, National Institute of Standards and Technology; Nicolas Limare, École Normale Supérieure de Cachan; Neil Calkin, Clemson University; Randy LeVeque, University of Washington11th Floor Lecture Hall 
Wednesday, December 12, 2012
TimeEventLocationMaterials
9:00 - 9:30Publishing Results from MATLAB - Loren Shure, The MathWorks, Inc.11th Floor Lecture Hall
9:30 - 10:00Sage notebooks - William Stein, University of Washington11th Floor Lecture Hall 
10:00 - 10:30IPython- tools for the lifecycle of research computing - Fernando Perez, University of California, Berkeley11th Floor Lecture Hall
10:30 - 10:35Group Photo11th Floor Lecture Hall 
10:35 - 11:00Coffee/Tea Break11th Floor Collaborative Space 
11:00 - 11:30Balancing Performance and Reproducibility with Intel MKL CNR - Todd Rosenquist, Intel Corporation11th Floor Lecture Hall 
11:30 - 12:30Lightning talks - Session 111th Floor Lecture Hall
12:30 - 2:00Break for Lunch  
2:00 - 2:15Joint session to discuss working groups and set goals11th Floor Lecture Hall 
2:15 - 4:30In parallel: Working groups on policy, best practices - Opportunity for participants to meet with developers for discussion of tools, attempt to apply to specific problems, etc.  
4:30 - 5:30Report back from working groups11th Floor Lecture Hall 
7:00 - 8:30Poster Session & Dessert Reception11th Floor Lecture Hall 
Thursday, December 13, 2012
TimeEventLocationMaterials
9:00 - 9:30Making Computations and Publications Reproducible with VisTrails - Juliana Freire, New York University11th Floor Lecture Hall
9:30 - 10:00Reproducible research as a community effort- experience of the Madagascar project - Sergey Fomel, University of Texas at Austin11th Floor Collaborative Space
10:00 - 10:30Sumatra: a toolkit for provenance capture and reuse - Andrew Davison, Centre National de la Recherche Scientifique (CNRS)11th Floor Lecture Hall
10:30 - 11:00Coffee/Tea Break11th Floor Collaborative Space 
11:00 - 12:30Lightning talks - Session 211th Floor Lecture Hall
12:30 - 12:35Survey Distribution11th Floor Lecture Hall 
12:35 - 2:00Break for Lunch  
2:00 - 2:15Joint session to discuss working groups11th Floor Lecture Hall 
2:15 - 4:30In parallel: Working groups on policy, best practices - Opportunity for participants to meet with developers for discussion of tools, attempt to apply to specific problems, etc.11th Floor Lecture Hall 
4:30 - 5:30Report back from working groups11th Floor Lecture Hall 
Friday, December 14, 2012
TimeEventLocationMaterials
9:00 - 9:30Interactive theorem proving, automated reasoning, and mathematical computation - Jeremy Avigad, Carnegie Mellon University11th Floor Lecture Hall
9:30 - 10:00clearScience- tools for communicating in the computational sciences - Brian M. Bot, Sage Bionetworks11th Floor Lecture Hall 
10:00 - 10:30Case Study: Creating a reproducibility repository for a paper with large-scale HPC results - Aron Ahmadia, Columbia University11th Floor Lecture 
10:30 - 11:00Coffee/Tea Break11th Floor Collaborative Space 
11:00 - 12:30Final joint session, open discussion of next steps11th Floor Lecture Hall 
12:30 - 1:30Box Lunches Available11th Floor Collaborative Space 
1:30 - 3:00Breakout groups to work on workshop outcomes  
3:00 - 3:30Coffee/Tea Break11th Floor Collaborative Space 
3:30 - 5:00Breakout groups to work on workshop outcomes  

Lecture Videos

Sage notebooks

William Stein
University of Washington
December 12, 2012

Exploratory Experimentation in Mathematics

Jon Borwein
Centre for Computer Assisted Research Mathematics and its Applications
December 10, 2012

Issues in Reproducibility

Randy LeVeque
University of Washington
December 10, 2012