31 - Caution in Data Sharing

Summary

Years ago, when I was in graduate school, my adviser and I published a paper in which we described a structural equations analysis of some experimental data that we had collected. At that time, structural equations analyses were not yet common, and they nearly always focused on correlational data rather than experimental data. So there were few established guidelines to follow, and we had to rely on our own judgment, as well as the advice of some local experts.

The journal’s editor, at that time, had an unusual policy that has since become standard at many other journals; he required authors to make their raw data available to anyone who asked for them, for a period of up to five years.

Soon after our paper appeared, we received a request for our data from someone who was an expert in the statistical analysis we had used, but had done no prior work in our topic area. He and his graduate student had criticisms of our statistical analysis and wanted to redo it in a “better” way, using the same data, to see if our conclusions were justified. After their new analyses were completed, the results did not seem to agree with those reported in our article, so they submitted a critique of our paper to the same journal where our paper had been published. We were given the chance to publish a rebuttal to that critique. The plan was for both papers (their critique and our rebuttal) to appear together in a future issue of the journal.