{"id":6569,"date":"2022-04-08T07:00:46","date_gmt":"2022-04-08T11:00:46","guid":{"rendered":"https:\/\/datacolada.org\/?p=6569"},"modified":"2022-04-09T08:48:29","modified_gmt":"2022-04-09T12:48:29","slug":"100-groundhog-2-0-further-addressing-the-threat-r-poses-to-reproducible-research","status":"publish","type":"post","link":"https:\/\/datacolada.org\/100","title":{"rendered":"[100] Groundhog 2.0: Further addressing the threat R poses to reproducible research"},"content":{"rendered":"

About a year ago I wrote Colada[95]<\/a>, a post on the threat R poses to reproducible research. The core issue is the 'packages'. When using R, you can run library(some_package)<\/code> and R can all of a sudden scrape a website, cluster standard errors, maybe even help you levitate. The problem is that packages get updated often, and on occasion in 'backwards incompatible' ways, making your existing code obsolete. The code that works today, may not work tomorrow.\u00a0<\/span><\/p>\n

In that post I introduced a solution to this problem with R packages: a new R package. It's called groundhog.<\/span><\/p>\n

With groundhog, the only thing you need to change to make your R code reproducible is:<\/span><\/p>\n

Instead of: library(pkg)<\/code><\/span>
\nDo this: \u00a0 \u00a0\u00a0 groundhog.library(pkg, date)<\/code><\/span><\/p>\n

Now every time you run that code, you load the version of the package that was available on that date<\/code>, regardless of when you run it. That's really all<\/span> you need to do to dramatically improve the reproducibility of your R Code.<\/span><\/p>\n

This post is an update on three fronts:<\/span><\/p>\n

1) Share new evidence on iRreproducibility (geRit?)
\n2) Announce groundhog 2.0 (key new feature: it works with GitHub packages)
\n3) A 'show me the money' moment.<\/span><\/p>\n

If you are familiar with renv<\/code> and wonder how it compares to groundhog, check this out (.htm)<\/a><\/span><\/p>\n

1) New evidence that R is a threat to research reproducibility
\n<\/strong>Over the past year I have learned a few things that made me more<\/em> concerned about R's reproducibility (more concerned than when I was motivated to spend months of my life developing a fricking R package to address that concern). I will mention three things.<\/span><\/p>\n

Thing #1: Most posted R scripts apparently don't even run
\n<\/em><\/strong>A paper published a few weeks ago in Nature: Scientific Data (.htm<\/a>)<\/em> attempted to automatically re-execute 2335 R scripts posted as supporting materials for published papers. After cleaning the scripts (installing necessary packages and fixing paths to local files) only\u00a0 44% of scripts run without generating errors. So, most<\/em> scripts did not run. M<\/span>oreover, 21% of all<\/em> failures were attributed to packages not loading. This is an underestimate of the problem caused by packages, because a package may load successfully, but reproduce different results, or produce an error elsewhere (see \"show me the money\" example at the end of this post).<\/span><\/p>\n

The script cleaning was done in an automated way, and this is important. It seems likely that some of the scripts that did not run, could run if a person were reading and editing the code before executing it. But there is one key finding that is free of this ambiguity: the same<\/span> scripts run in some but not other versions of R.\u00a0<\/span><\/p>\n

\"\"<\/a><\/span><\/p>\n

Fig 1.<\/strong> Uri made this figure based on numbers reported in Figure 11 (.png<\/a>) of Trisovic et al. (2022).<\/span><\/p>\n

R-3.2 may feel like ancient history, but when Trump was running for president, R was running 3.2. <\/span><\/p>\n

Groundhog also helps with this source of irreproducibility. Specifically, when the date entered in groundhog.library()<\/code> does not match the R version being used, it gives a warning and suggests a date to switch the groundhog day to, or the version of R to use for that groundhog day.
\n<\/span><\/p>\n

Running multiple versions of R on the same computer is trivial in Windows and quite easy in a Mac. Groundhog's website provides step-by-step<\/a> instructions.<\/span><\/p>\n

Thing 2. Come think of it: this is a big pain in the Rs for books
\n<\/em><\/strong>I recently noticed that books about R, or that rely on R, often include printed out lists of the version of each package they used (see figure below). This highlights that I am not the only one worried about the issue of R stability. And just think about it. You may work for years on a book on causal inference, say, or interpreting interactions, and the examples you include in your book, if they involve R code, may stop working shortly after the book is finished (possibly even before it is published). This seems tolerable for a book about<\/em> R, say \"R for Data Science\" (as planned obsolescence is not a terrible business model), but it seems less tolerable for a book on anything else that happens to use R to give concrete examples. A book on causal inference is not going to be revised every 6 months just to make sure all the R examples still run. In fact, it may never<\/em> get revised.\u00a0 <\/span><\/p>\n

Attempting to fix a book's reproducibility problem by printing a list of used package versions is both impractical and dangerous (see footnote for why it is impractical and dangerous [1<\/a>]). An alternative is to write books relying on groundhog.<\/span><\/p>\n

In fact, textbooks provide a textbook example of groundhog's simplicity:<\/span><\/p>\n

\"\"<\/a><\/span><\/p>\n

That figure in the left is actually a small portion of the full list. Check. It. Out.:<\/span><\/p>\n

\"\"<\/a><\/span><\/p>\n

Thing 3. Personal Experience with abandoned packages<\/em>
\n<\/em><\/strong>While working on a research project I needed to revisit my own code from just a few months earlier. But, I could not install a package that I needed <\/em>because it was no longer available on CRAN (it had been archived because one of its dependencies was no longer being maintained). My own code would not have run in my own machine. But I had groundhog, and so it did.<\/span><\/p>\n

2. New features since groundhog 1.0.0
\n<\/strong>The current version of groundhog on CRAN is v1.5.0. It has some neat features introduced since the original release v1.0.0, including the possibility of loading\/installing a set of packages in a single call:\u00a0<\/em><\/span><\/p>\n

pkgs <- c('metafor','pwr', 'jsonlite')
\n<\/code>groundhog.library(pkgs,'2022-03-01')<\/code><\/span><\/p>\n

And having version-control for groundhog itself\u00a0<\/em><\/span><\/p>\n

meta.groundhog('2022-01-01')\u00a0<\/code> #load the version of groundhog available that date.<\/span>\u00a0<\/span><\/p>\n

A bigger innovation is that the next<\/u><\/em> release of groundhog, v2.0.0, will work not just with CRAN but also with git repositories (GitHub and GitLab).<\/span><\/p>\n

Like this:
\ngroundhog.library('crsh\/papaja' , '2022-03-01')<\/code><\/span>
\ngroundhog.library('gitlab::jimhester\/covr', '2022-03-01')<\/code><\/span><\/p>\n

Git packages arguably need version control even more than CRAN packages do; see footnote [2<\/a>]<\/span><\/p>\n

Groundhog 2.0.0 should be on CRAN by May 2022.
\nIn the meantime, you can use the almost there version: v1.9.9.9999 available on GitHub:<\/span><\/p>\n

remotes::install_github('CredibilityLab\/groundhog')<\/code><\/span><\/p>\n

3) Show me the money
\n<\/strong>I thought it would be interesting to try and see if groundhog could 'rescue' an R script that the Nature: Scientific Data<\/em> paper reported as failing to reproduce.\u00a0<\/span><\/p>\n

I downloaded one R script flagged as non-reproducible, supporting a paper published in Political Analysis<\/em> (.htm<\/a>)<\/a> [3<\/a>]. The script\u00a0 has just 9 active lines of code; it is supposed to estimate a regression with fixed effects using the package 'bife<\/code>' (after loading the data with 'foreign<\/code>'). <\/span><\/p>\n

After running that script in R-4.1.3 (current version as of this writing), however, all I got was this ugly thing:<\/span><\/p>\n

\"\"<\/a><\/span><\/p>\n

The R script had been uploaded to the Dataverse<\/em> on January 2019, so I thought to try late 2018 as a reference date. Back then R was running version R-3.5. So I started up R-3.5 in R Studio (again, it is easy to have multiple versions of R in the same computer; see how<\/a>).<\/span><\/p>\n

I then did install.packages(c('foreign','bife'))<\/code>, loaded both packages with library()<\/code> and ran the 9 lines again. And…
\n…no luck. Got the same error message. <\/span><\/p>\n

So I used groundhog.libray()<\/code> instead.And …
\n…that did<\/em> work. <\/span><\/p>\n

Specifically, I installed & loaded the packages like this:<\/span><\/p>\n

\"\"<\/a><\/span><\/p>\n

And the 9 lines of code produced this beautiful, oddly self-affirming, regression table:<\/span><\/p>\n

\"\"<\/a><\/span><\/p>\n

If curious, this is the R Code<\/a><\/span> \u00a0for the above example.
\nSee footnote for why the code ran with groundhog.library()<\/code> but not with library()<\/code> [4<\/a>].<\/span><\/p>\n

Conclusions
\n<\/strong>A big lesson from Bill Murray's conundrum in the movie Groundhog Day, is that if you want different results, you need to try something different. We know by now that relying on R alone for package management (install.packages()<\/code> + library()<\/code>) will lead researchers to share R scripts that do not work. So if we want the R scripts we share to work we need to try something different. Groundhog is something different. Nothing is perfect, but groundhog delivers a huge increase in reproducibility with absolutely minimal effort. <\/span><\/p>\n

groundhog.library(pkg, date)<\/code><\/span><\/p>\n

\"\"<\/a><\/span><\/p>\n

 <\/p>\n

Learn more about groundhog:\u00a0<\/strong><\/span>
\nColada[95] on Groundhog<\/a>\u00a0 \u00a0|\u00a0 \u00a0 groundhogr.com<\/a>\u00a0 \u00a0 |\u00a0 \u00a0 GitHub<\/a><\/span><\/p>\n

 <\/p>\n

\"Wide<\/span><\/p>\n

 <\/p>\n

Author feedback
\n<\/strong>I shared an early draft of this post with Ana Trisovic (.htm<\/a>), 1st author of the Nature: Scientific Data<\/em> paper and she provided useful feedback and clarifications. The post did change substantially since she read it.\u00a0<\/span><\/p>\n

\n

Subscribe to Blog via Email<\/h2>\n\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t\t

Enter your email address to subscribe to this blog and receive notifications of new posts by email.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t