2008-12-21 Scientific Publications and Proving Empirical Results

Reading scientific/academic publications in computer science can be frustrating due to various reasons. But the most frequent reason is the inability to reproduce the results described in a paper due to the lack of the software and tools to reproduce the empirical analysis described. You can regularly read reference in papers to internal software used for the analysis or survey but the paper lacks a link to download the software. Very often, I shared this frustration with my (work and academic) colleague but I was always expecting a more formal paper describing this major issue in scientific publication especially in computer science.

By sheer luck, I hit a paper called "Empiricism is Not a Matter of Faith" written by Ted Pedersen published in Computational Linguistics Volume 34, Issue 3 of September 2008. I want to share with you the conclusion of the article :

However, the other path is to accept (and in fact insist) that highly detailed empirical 
studies must be reproducible to be credible, and that it is unreasonable to expect that 
reproducibility to be possible based on the description provided in a publication. Thus, 
releasing software that makes it easy to reproduce and modify experiments should be 
an essential part of the publication process, to the point where we might one day only 
accept for publication articles that are accompanied by working software that allows for 
immediate and reliable reproduction of results. 

The paper from Ted Pedersen is clear and concise, I couldn't explain better that. I hope it will become a requirement in any open access publication to add the free software (along with the process) used to make the experiments. Science at large could only gain from such disclosure. Open access should better integrate such requirements (e.g. reproducibility of the experiments) to attract more academic people from computer science. Just Imagine the excellent arxiv.org also including a requirements in paper submission to include a link to the free software and process used to make the experiments, that would be great.

Tags: openaccess research education freesoftware