Redesign 2025
#6202 | ResearchBox

ResearchBox #6202 - 'Figuring Out Figure 1'


Bingo Table
  Show file names
  Show file IDs
  Show timestamps

 txt

  


  ManyLabsData.txt


  Figuring Out Figure 1 R Code - To Post.R


There is no AsCollected linked to this project; results provenance has not been documented.

Previewing files
Files can be previewed by clicking on blue font.
Codebooks can be previewed by clicking on


  

Tell us if something is wrong with this Box

'DEAR READER' MESSAGE FROM THE AUTHORS

Dear Reader,


The R code contains (and runs) - using the reticulate package - the python code that Rocca and Yarkoni used to perform the cross-validation results contained in their Figure 1. That is why there is no python code. The R code also contains a newly built function (kfold_cv) that can reproduce these results, and that also allows one to perform cross-validation under different parameters (e.g., number of folds, using different benchmarks, etc.)


I used AI to generate the codebook for the Many Labs data file. I verified the accuracy of the relevant variable descriptions for this post - those about sunk costs and anchoring - but not the others.

This version: March 05, 2026
(may be edited at any time)

BOX INFORMATION

SUPPLEMENTARY FILES FOR
Joe Simmons, 'Figuring Out Figure 1', Data Colada
https://datacolada.org/134

CITING THIS RESEARCHBOX
Simmons, J. (2026). ResearchBox 6202, 'Figuring Out Figure 1', https://ResearchBox.org/6202. Zenodo. https://doi.org/10.5281/zenodo.19111975

LICENSE FOR USE
All content posted to ResearchBox is under a CC By 4.0 License (all use is allowed as long as authorship of the content is attributed). When using content from ResearchBox please cite the original work, and provide a link to the URL for this box (https://researchbox.org/6202).

BOX PUBLIC SINCE
March 19, 2026   

BOX CREATORS
Joseph Simmons (jsimmo@upenn.edu)

ABSTRACT
A few years ago our Journal Club discussed an interesting methods paper entitled, “Putting Psychology to the Test: Rethinking Model Evaluation Through Benchmarking and Prediction” (.htm). This post describes my attempt to understand what’s happening in Figure 1 of that paper, which shows that extremely simple experiments can generate extremely negative R2s.