Csdrhrt

Many research papers include numerical results obtained through computation. Most of the time such computations are performed using software that is used by many mathematicians, i.e., Maple, Mathematica, or even C/C++ code. Should such code be included in the body of the published paper?

I've heard arguments from both sides:

• Including such code can greatly decrease the time taken by a referee to replicate the results,


• The code can be easily modified by further authors who wish to extend the result,


• The reader does not need to spend time searching the journal website or the Internet for any "auxilliary files" containing the code.

On the other hand,

• Pages of code degrade the aesthetic nature of the publication,


• The author might need to spend additional space explaining the coding decisions that were made in the algorithms,


• It is likely that there exist (much) better ways to write the same algorithms in the given, or any other, language.

So what is the standard in mathematical research papers that present numerical results, either as a main or as a side result? Should code be included within the body of the publication, as an auxilliary file, or not at all?

My answer is:

Don't put code in your paper. Do: put pseudocode in your paper, version control your code on Github, and add a link to your Github repository to your paper.

• The purpose of a paper is to be read; the purpose of code is to be executed by a computer. These purposes should not be mixed, so a readable representation of your code should be included in your paper. That is exactly why pseudocode was invented.


• All code intended to be used by more than one person should be version controlled. This balances the two most relevant concerns: the original version of the code is preserved for posterity, but the author retains the ability to update it as bugs or improvements are discovered. (Additionally, the forking mechanism in Github allows others to transparently modify your code or apply it to their own ends.)

In fact, I am willing to make a more general argument: mathematicians should version control their papers as well. The reasons are the same: the original version still exists (with a timestamp) so that priority disputes can be settled easily, but the paper can be maintained and updated - no more errata for old papers / textbooks!

The underlying premise of this answer is that maintaining and distributing code is a software engineering problem, and to the extent that mathematicians need to solve it they should follow software engineers' lead. This has two advantages: on one hand software engineers have a much more severe version of the problem and will therefore solve it better, and on the other hand as the solutions inevitably change they will be accompanied by tools and strategies for migrating old code into new frameworks which is ultimately the best way to ensure that the code survives as long as possible.

At least in my field (numerical linear algebra), the current standard is that including the full source code is not mandatory for a publication. That said, there are many reasons why sharing your code is a good idea; for instance this article on SIAM news makes some very compelling arguments.

Unless it's just a few lines, it is quite unusual to have code included verbatim in the publications. It would be cumbersome to copy and paste, for instance. Common solutions are:

• hosting it on your institutional page


• offering to share the source code to interested parties via e-mail


• having a Github repository


• including it into the Arxiv version of your paper as an ancillary_file
  


• sharing it on Zenodo.

If you are concerned about long-time archival, the last two items in my list are meant to solve this problem; although it could be argued that also Github is becoming "too big to fail" these days.

I think that Federico Poloni's answer gives good advice as of 2018, but as a mathematical community I think we should be thinking harder about this question. Simply making source code available, even via something like the arXiv which will be around "forever", is not a complete solution, because source code may be nearly useless after (say) 50 years because the compilers are no longer readily available, or worse, the code runs only on some proprietary software that no longer exists. This concern applies even if the computation has been formalized in a proof assistant, since who knows if today's proof assistants will be around 50 years from now?

One idea would be for professional societies such as the American Mathematical Society to develop a long-term archival plan, perhaps collaborating with government entities such as the Library of Congress.

This is more of a very extended comment than a complete answer.

I tend to find "should" questions boiling down as much to values as much as anything; "should" in order to achieve what?

Let me suggest that we need to understand a few things:

• The advantages


• The disadvantages


• Is there a real problem with reproducibility that needs fixed?


• The cost of not doing so or of doing so halfheartedly.


• The opportunity cost or motivational/ funding challenges


• Variation between sub fields


• Technical challenges, short and long term


• Expectations or even standards


• Cultural challenges

I'll try to avoid repeating the observations from the existing answers and comments, but let me add some thoughts:

• In software engineering, the process for shipping code is very different from the typical mathematical program. A key reason for this is quality, and of these correctness is the most important element. That is a value of overwhelming importance in any proof, so maybe open code and peer review would be a good thing.


• Related to that: writing code to be read is different to writing code to just convince oneself; how are mathematicians to learn that?


• There is a difference between learning enough about programming to get a result and the skills needed to write good tests, make code readable and convince readers that the code is valid. I'd ask, if you have not done that well, how do you expect credibility of your conclusions?


• What is the penalty for coding errors as things stand? I would have thought that in maths, publishing results that are subsequently proven false would not do one's career any good. This compares interestingly to other fields in science where to some extent one expects many "results" in papers to be subsequently not borne out. Interesting to hear feedback on this one as to what happens in practice.


• Do people feel that time spent publishing code would be unproductive?


• A software engineering style code review is not anonymous (at least usually); is this a problem?


• There is an argument to use "lowest common denominator" languages that might be old but that proves their longevity and wide accessibility; e.g. 'C'.


• Timothy Chow noted use of notebooks; they provide a great way to document code and the overall approach; I can see these becoming more and more used. Interestingly, I think this might conflict with "lowest common denominator" languages, as the notebook hosting language (Jupiter or Mathematica) might have less longevity.

There are some issues that are not emphasised enough in the previous comments and answers. Having the source code used by an author does not let you check that the author's theorems are correct. It only lets you check that the program does what the author claims. Transcribing the program output to the published paper is the step where an error is least likely to have occurred. Much more likely is an error in the program.

So, can you eyeball the program to check if it is correct? Not unless it is a very short simple program. I publish articles that rely on tens of thousands of lines of code that took me and others months of hard work to write and debug. Your chances of looking at it and checking its correctness in a reasonable amount of time are next to zero. One day there will be programs that can check correctness for you; the beginnings exist today but generally useful checkers are still a long way off.

So what to do? If you are an author, get a coauthor and aim for separately implemented programs that get the same result, hopefully using different methods. (An axiom of software engineering is that programmers solving the same problem using the same method tend to make the same mistakes.) Intermediate results are very useful for checking, especially when the final answer has low entropy (like "yes" or "empty set").

Another fact is that problems which needed very tricky programming and bulk computer time 20 years ago can now be solved in a reasonable time using simpler programs. Presumably that trend will continue. Any computational result that is important enough will eventually be replicated independently without so much effort.