Home

Reproducibility in Computational and Experimental Mathematics (December 10-14, 2012)


Organizing Committee
Click for code to create this image.
Description

In addition to advancing research and discovery in pure and applied mathematics, computation is pervasive across the sciences and now computational research results are more crucial than ever for public policy, risk management, and national security. Reproducibility of carefully documented experiments is a cornerstone of the scientific method, and yet is often lacking in computational mathematics, science, and engineering. Setting and achieving appropriate standards for reproducibility in computation poses a number of interesting technological and social challenges. The purpose of this workshop is to discuss aspects of reproducibility most relevant to the mathematical sciences among researchers from pure and applied mathematics from academics and other settings, together with interested parties from funding agencies, national laboratories, professional societies, and publishers. This will be a working workshop, with relatively few talks and dedicated time for breakout group discussions on the current state of the art and the tools, policies, and infrastructure that are needed to improve the situation. The groups will be charged with developing guides to current best practices and/or white papers on desirable advances.



  • Sergey Fomel*
    (University of Texas at Austin)
  • Juliana Freire*
    (New York University)
  • James Glimm*
    (Stony Brook University)
  • Sigal Gottlieb
    (University of Massachusetts)
  • Josh Greenberg
    (Alfred P. Sloan Foundation)
  • Tom Hales *
    (University of Pittsburgh)
  • Nicolas Hengartner
    (Los Alamos National Laboratory)
  • David Ketcheson
    (King Abdullah University of Science & Technology)
  • Matt Knepley
    (University of Chicago)
  • David Koop
    (New York University)
  • Randall LeVeque
    (University of Washington)
  • Nicolas Limare *
    (École Normale Supérieure de Cachan)
  • Sébastien Li-Thiao-Té
    (Université de Paris XIII (Paris-Nord))
  • Elizabeth Loew
    (Springer)
  • Ursula Martin *
    (University of London)
  • Bruce McHenry
    (ME, Inc)
  • Chris Mentzel*
    (Gordon and Betty Moore Foundation)
  • Sarah Michalak
    (Los Alamos National Laboratory)
  • Ian Mitchell
    (University of British Columbia)
  • Victor Moll
    (Tulane University)
  • Hatef Monajemi
    (Stanford University)
  • Akil Narayan
    (University of Massachusetts)
  • Peter Norvig*
    (Google Inc.)
  • Travis Oliphant
    (NumFOCUS)
  • Peter Olver *
    (University of Minnesota)
  • Geoffrey Oxberry
    (Lawrence Livermore National Laboratory)
  • Fernando Perez *
    (University of California, Berkeley)
  • Konrad Polthier
    (Freie Universität Berlin)
  • Bill Rider*
    (Sandia National Laboratories)
  • Robert Robey
    (Los Alamos National Laboratory)
  • Todd Rosenquist*
    (Intel Corporation)
  • Michael Rubinstein
    (University of Waterloo)
  • Thomas Russell*
    (National Science Foundation)
  • Fernando Seabra Chirigati
    (New York University)
  • Benjamin Seibold
    (Temple University)
  • Loren Shure*
    (The MathWorks, Inc.)
  • Philip Stark
    (University of California, Berkeley)
  • William Stein *
    (University of Washington)
  • Victoria Stodden
    (Columbia University)
  • Benjamin Stubbs
    (Harvard Medical School)
  • Andrew Sutherland
    (Massachusetts Institute of Technology)
  • Matthias Troyer
    (ETH)
  • Jan Verschelde
    (University of Illinois)
  • Stephen Watt
    (University of Western Ontario)
  • Greg Wilson *
    (Software Carpentry)
  • Carol Woodward
    (Lawrence Livermore National Laboratory)
  • Yihui Xie
    (Iowa State University)

SundayDecember 9, 2012
TimeDescriptionSpeakerLocationAbstractsSlides
2:00 - 5:00Version Control TutorialRandy LeVeque, University of Washington; William Stein, University of Washington; Victoria Stodden, Columbia University11th Floor Lecture Hall (121 South Main Street, Providence, 11th Floor)
PDF

MondayDecember 10, 2012
TimeDescriptionSpeakerLocationAbstractsSlides
9:00 - 9:30Issues in ReproducibilityRandy LeVeque, University of Washington11th Floor Lecture Hall
PDF
9:30 - 10:00What does it take to do reproducible computational science? What stands in our way?Bill Rider, Sandia National Laboratories11th Floor Lecture Hall
PDF
PDF
10:00 - 10:30Exploratory Experimentation in MathematicsJon Borwein, Centre for Computer Assisted Research Mathematics and its Applications11th Floor Lecture Hall
PDF
11:00 - 11:30A brief history of the reproducibility movementVictoria Stodden, Columbia University11th Floor Lecture Hall
PDF
11:30 - 12:00Open Discussion on issues11th Floor Lecture Hall
1:30 - 2:00TBADavid Donoho, Stanford University11th Floor Lecture Hall
2:00 - 2:30Trust in computation and mathematical proofTom Hales, University of Pittsburgh11th Floor Lecture Hall
2:30 - 3:00Open Discussion on issues11th Floor Lecture Hall
3:00 - 3:30Coffee/Tea Break11th Floor Collaborative Space
3:30 - 4:00Reproducibility issues important to applicationsJames Glimm, Stony Brook University
4:00 - 4:30High-Precision Computation and ReproducibilityDavid Bailey, Lawrence Berkeley National Laboratory11th Floor Lecture Hall
PDF
PDF
4:30 - 5:30Open discussion on issues11th Floor Lecture Hall
5:30 - 7:00Welcome Reception11th Floor Collaborative Space

TuesdayDecember 11, 2012
TimeDescriptionSpeakerLocationAbstractsSlides
9:00 - 9:30R & D at Google- Hybrid Research, Web-Scale DevelopmentPeter Norvig, Google Inc11th Floor Lecture Hall
PDF
9:30 - 10:00Recent strategies and challenges for reproducibility in global climate modelingKatherine Evans, Oak Ridge National Laboratory11th Floor Lecture Hall
10:00 - 10:30Is mathematics a social machine?Ursula Martin, Queen Mary University of London11th Floor Lecture Hall
PDF
10:30 - 11:00Coffee/Tea Break11th Floor Collaborative Space
11:00 - 12:30Panel discussion with program managers / open discussionNeil Chue Hong, University of Edinburgh; Ursula Martin, Queen Mary University of London; Chris Mentzel, Gordon and Betty Moore Foundation; Karen Pao, Department of Energy; Thomas Russell, NSF 11th Floor Lecture Hall
12:30 - 2:00Break for Lunch
2:00 - 2:30Welcome to the collaborative webTim Clem, GitHub11th Floor Lecture Hall
PDF
2:30 - 3:00Software Carpentry- What We've Done, What We've Learned, Where We're GoingGreg Wilson, Software Carpentry11th Floor Lecture Hall
PDF
3:00 - 3:30Coffee/Tea Break11th Floor Collaborative Space
3:30 - 4:00What's happening with the World Digital Mathematics Library?Peter Olver, University of Minnesota11th Floor Lecture Hall
PDF
PDF
4:00 - 5:30Panel discussion on journal policies / experiences Ron Boisvert, National Institute of Standards and Technology; Nicolas Limare, École Normale Supérieure de Cachan; Neil Calkin, Clemson University; Randy LeVeque, University of Washington11th Floor Lecture Hall

WednesdayDecember 12, 2012
TimeDescriptionSpeakerLocationAbstractsSlides
9:00 - 9:30Publishing Results from MATLABLoren Shure, The MathWorks, Inc.11th Floor Lecture Hall
PDF
PDF
9:30 - 10:00Sage notebooksWilliam Stein, University of Washington11th Floor Lecture Hall
10:00 - 10:30IPython- tools for the lifecycle of research computingFernando Perez, University of California, Berkeley11th Floor Lecture Hall
PDF
10:30 - 10:35Group Photo11th Floor Lecture Hall
10:35 - 11:00Coffee/Tea Break11th Floor Collaborative Space
11:00 - 11:30Balancing Performance and Reproducibility with Intel MKL CNRTodd Rosenquist, Intel Corporation11th Floore Lecture Hall
11:30 - 12:30Lightning talksSession 111th Floor Lecture Hall
PDF
12:30 - 2:00Break for Lunch
2:00 - 2:15Joint session to discuss working groups and set goals11th Floor Lecture Hall
2:15 - 4:30In parallel: Working groups on policy, best practices Opportunity for participants to meet with developers for discussion of tools, attempt to apply to specific problems, etc.
4:30 - 5:30Report back from working groups11th Floor Lecture Hall
7:00 - 8:30Poster Session & Dessert Reception11th Floor Lecture Hall

ThursdayDecember 13, 2012
TimeDescriptionSpeakerLocationAbstractsSlides
9:00 - 9:30Making Computations and Publications Reproducible with VisTrailsJuliana Freire, New York University11th Floor Lecture Hall
PDF
9:30 - 10:00Reproducible research as a community effort- experience of the Madagascar projectSergey Fomel, University of Texas at Austin11th Floor Collaborative Space
PDF
PDF
10:00 - 10:30Sumatra: a toolkit for provenance capture and reuseAndrew Davison, Centre National de la Recherche Scientifique (CNRS)11th Floor Lecture Hall
PDF
10:30 - 11:00Coffee/Tea Break11th Floor Collaborative Space
11:00 - 12:30Lightning talksSession 211th Floor Lecture Hall
PDF
12:30 - 12:35Survey Distribution11th Floor Lecture Hall
12:35 - 2:00Break for Lunch
2:00 - 2:15Joint session to discuss working groups11th Floor Lecture Hall
2:15 - 4:30In parallel: Working groups on policy, best practices Opportunity for participants to meet with developers for discussion of tools, attempt to apply to specific problems, etc.11th Floor Lecture Hall
4:30 - 5:30Report back from working groups11th Floor Lecture Hall

FridayDecember 14, 2012
TimeDescriptionSpeakerLocationAbstractsSlides
9:00 - 9:30Interactive theorem proving, automated reasoning, and mathematical computationJeremy Avigad, Carnegie Mellon University11th Floor Lecture Hall
PDF
PDF
9:30 - 10:00clearScience- tools for communicating in the computational sciencesBrian M. Bot, Sage Bionetworks11th Floor Lecture Hall
PDF
10:00 - 10:30Case Study: Creating a reproducibility repository for a paper with large-scale HPC resultsAron Ahmadia, Columbia University11th Floor Lecture
10:30 - 11:00Coffee/Tea Break11th Floor Collaborative Space
11:00 - 12:30Final joint session, open discussion of next steps11th Floor Lecture Hall
12:30 - 1:30Box Lunches Available11th Floor Collaborative Space
1:30 - 3:00Breakout groups to work on workshop outcomes
3:00 - 3:30Coffee/Tea Break11th Floor Collaborative Space
3:30 - 5:00Breakout groups to work on workshop outcomes

TimeDescriptionSpeakerLocationAbstractsSlides
       ICERM Report developed collaboratively by the ICERM workshop participants, and compiled and edited by the Organizers:
       Setting the Default to Reproducible: Reproducibility in Computational and Experimental Mathematics PDF
       Wiki with additional resources