Category Archives: scientometrics

The AGU EOS ‘Anonymous Peer Review’ debate of 2003-2004

Leave a reply

This summer I stumbled upon a cache of EOS newsletters from 2003. Among the pages was a series of comments and letters about anonymous review, specifically problems and possible solutions. It’s nice to know that we struggle with the same issues 14 years later.

The original article written on July 1, 2003 by Beck (2003) was focused on the rejection of a paper by 2 anonymous reviews and an anonymous AE. After listing and discussing potential reasons that a reviewer and/or AE would prefer to remain anonymous. Beck ends by writing:

“The only reviews I remember that left me perma­nently angry were anonymous. There is far too much unpleasantness in the world already to needlessly introduce even a little bit more. Anonymous reviews are unnecessary, unacceptable, and should not be permitted.” 

Strong statement! I have my own opinions about anonymity in peer review (I’m sure everyone does), but what is most interesting to me is that fact that this article produced such a large reaction — I can find 15 letters and comments published in EOS as a response to Beck (2003) — compared to the rare comment-reply pairs in JGR-ES.

On July 29th, 2003

  • Roninove (2003) writes in to support Beck (2003), having written a letter about problems with anonymous reviews (back in 1990).
  • Criss and Hofmeister (2003) suggest discounting anonymous reviews, and discuss the issues surrounding signed vs unsigned reviews for underrepresented groups.

On Sept 23rd, 2003

  • Geller (2003) writes in to suggest that AEs should always sign reviews because they often make the decision for the editor.
  • Goff (2003) writes in to suggest AEs should sign reviews and that AGU should encourage signed reviews and newer journals should require signed reviews
  • Walder (2003) writes in to suggest that AGU AEs should sign reviews and we should collect data — reviewers be asked ‘why’ they choose to remain anonymous.

Sept 30th, 2003

  • Forel (2003) is an ‘advocate’ for anonymous reviewing, but believes editors should not be anonymous.
  • Fisher (2003) writes in to suggest double blind reviewing
  • Savov (2003) writes that science should be “…discussed in the open air.” and suggests that the paper, reviews, and reviewer names should all be published together.
  • Okal (2003) writes that the current system should be preserved and personal preference (re: signed vs unsigned reviews) should be respected. Okal writes that this debate has been going on for decades with no clear solution:

“The debate on peer review has been going on for decades. It may be the worst possible system, but by and large it works. And to paraphrase Sir Winston Churchill, wait until you consider all the other ones….”

Dec 23rd, 2003

  • The editors of JGR-Atmopsheres respond in O’Dowd et al. (2003). They discuss the editorial process in the journal and highlight the role of anonymity for the AEs and reviewers.

Dec 30th, 2003

  • Kirwan Jr. (2003) writes that peer reviews should not be signed because it could be self serving. Furthermore authors should not speculate about the authors of their anonymous reviews because of possible negative and counterproductive consequences.
  • Wesolowski (2003) writes that finding reviewers is difficult enough without requiring the identification of reviewers, and forced signing of reviews may lead to overly positive reviews.

April 20th, 2004

  • Armstrong (2004) discusses the possibility of multiple review stages, some with or without anonymity.  
  • Sturrock (2004) presents a ‘Code of Ethics’ for peer review.

April 27th, 2004

  • Genereaux and Sen (2004) discuss the NSF proposal review process, specifically how proposers do not have an opportunity to respond to “Incorrect and Overly Negative Statements (IONS)”.

 

N.B. — There was an article on anonymous peer review in GSA Today by McBirney (2003) — here is a link to the issue — something must have been in the air. 

Age distribution of JGR-ES articles cited in a given year

Leave a reply

In a previous post I looked at yearly cohorts of JGR-ES papers, and the mean number of citations that papers from each yearly cohort accrues through time as the papers age. In this post I will look at something slightly different — what is the age distribution of cited JGR-ES articles in a given year? Put another way, if we had all the citations that JGR-ES papers racked up in a single year, say 2007, what fraction of are citations to JGR-ES papers published in 2003? or 2004? or 2005? The idea for this plot came from Clark and Hanson (2017).

CHplot.jpg

Each line on this plot represents all citations to JGR-ES articles in a given year. For instance, the dark blue curve represents all citations to JGR-ES articles in the year 2010. The big spike in the dark blue curve means that, in 2010, 27% of all citations to JGR-ES articles were specifically focused on JGR-ES articles from 3 years prior (i.e., 2007). Notice that the orange curve (2011 citations) has a peak at 4 years prior (i.e., 2007 as well). As we march forward in time, that peak in citations moves back, and always represents the 2007 cohort of JGR-ES papers.

So what happened in 2007? Looking further in the citation data, there is a rotating group of ~10 papers from 2007 that are all highly cited. In additional a single paper (Schoof, 2007) is highly cited every year after its publication. This group of papers from 2007 accounted for 10% – 30% of all citations that JGR-ES received from 2010-2016.

Share of JGR-ES paper citations received after 2 and 5 years

Leave a reply

Last week I looked at citation histories for JGR-ES articles — most articles tend to accrue citations steadily through time after publication, and mean citations/year do not tend to show a strong decrease with time.  So how does this relate to Journal Impact Factor (JIF), which is calculated by dividing the total number of citations received by a journal (for all articles) in the 2 previous years, by the total number of articles published in the previous two years. There is also a five year variant of the JIF. a time series of JIF for JGR-ES and other geomorphology journals can be seen here.

Specifically, what share of total citations to a JGR-ES paper are received within the first 2 years after publication; how about after the first 5 years? After downloading the JGR-ES data from the Web of Science, I grouped papers by year and investigated these yearly cohorts in the box and whisker plots below.

2yearfrac.jpg

 

The oldest JGR-ES articles (2003-2006) recieved between 0 – 40% of their total citations in the first 2 years of their life — the JIF time window.  The newest articles (2008-2010) receive a larger share of their total citations, but keep in mind that this share will continue to shrink through time as the paper accumulates more citations.

 

5yearfrac.jpg

The five year fraction is larger, and seems to represent 10-80% of total citations for the oldest JGR-ES papers (2003-2006).

These plots demonstrate some of the complications with JIFs for journals where citations might accrue more slowly that a 2 and 5 year window.

JGR-ES citation histories

Leave a reply

This post is focused on citation histories for JGR-ES papers — how a paper and groups of papers accumulate citations through time. JGR-ES started in 2003, and in a previous post I outlined the growth in number of papers published per year.

From the Web of Science I downloaded all of the data for all JGR papers and their respective citations. I grouped papers by year and investigated yearly groups/cohorts in the plots below.

I’ll call this first figure a set of cumulative citation plots — the x-axis is the year since publication, and the y-axis is fraction of total citations for each paper. Each panel represents a specific cohort of JGR-ES papers (all of the papers published in a given year), and each black line represents a single paper as it accumulates citations through time from 0 to 1 (i.e., no citations to its own maximum number of citations).

cumulative.jpg

Keep in mind that each x-axis is a different length (depending on the age of the cohort). Notice how each paper tends to accrue citations in a generally smooth manner — a notable outlier is the curve in 2006, where a paper waits 7 years before its first citation(s).

We can also look at the citation history of these yearly cohorts of papers in a different way— given group of papers published in a single year, what is the mean number of citations/paper in a given year as a function of paper age. Below is a plot for mean citations/year as a function of paper age for 8 cohorts —(all papers from JGR-ES published each year from 2003 to 2010). (‘mean’ might not be the best metric, but let’s stick with it for now, without ‘min’ and ‘max bounds).

meanhistory.jpg

One key aspect of each curve is that most tend to trend upward even after the 2 and 5 year windows — keep in mind that these are the time periods used for calculating Journal Impact Factor (and 5 year Journal Impact Factor). In fact the mean citations/year do not show strong downward trends, suggesting that papers from JGR-ES tend to have a long ‘life’.

One other note — cohorts from more recent times (2007 onward) tend to have larger mean citations/year. I imagine this is related to the general growth in JIF through time for geomorphology journals.

Some context for future posts: Journal Impact Factor for Geomorphology journals

Leave a reply

For the next few posts I will be digging into the citation data for JGR-ES. I want to look at how the papers from JGR-ES accumulates citations — from which journals, over what time frame, when citations peaks occur, etc.

My motivation is to look at how actual papers accrue citations relative to the 2 and 5 year windows over which the ‘Journal Impact Factor’ (JIF) is calculated. Therefore I want to show the history of ‘normal’ (2 year) and 5 year JIF for JGR-ES and the other three geomorphology journals that I have focused on previously.  All data was downloaded from the Journal Citation Report.

From the plots below, my general take away is that these journals are all very similar to each other, mirroring their similarity in citation distribution — see the 2014, 2015 and 2016 distributions.

JIF.jpg

5yr JIF.jpg

2 notes:

Citation distributions for Geomorphology journals (2016 impact factor)

Leave a reply

The 2017 Journal Citation report was released last month, which includes the new journal impact factor calculations for 2016.

A journal impact factor is calculated by dividing the total number of citations received (for all articles) in a 2 year period by the total number of articles published in the previous two years. It is a poor metric for individual papers. To see the full range of citations garnered by the papers published in the previous two years, it is useful to see citation distributions (the number of papers with a given number of citations published in a single journal), a useful visulaization that I became aware of from a paper by Larivière et al. 2016.

Here is a link to my post last year with distributions re: the 2014 and 2015 Impact factor calculation. Below I recalculate the citation distributions for geomorphology journals with data for the 2016 impact factor calculation (downloaded from the Web of Science). I also report the median of these distributions.

2016 JIF.jpg

Just to clarify further — this plot reports the number of papers with a given number of citations for each journal.

Similar to the last 2 years, the distributions are skewed and similar to each other. Also similar to the past 2 years, structure-from-motion papers are often in the long tail (far to the right; highlighting the broad interest of the technique).

  • A recent blog post by AGU discussed the adoption of alternative journal metrics that the organization will soon report.
  • A recent editorial in WRR also discusses citation metrics in regards to the hydrology community. 

AGU publications on Sci-Hub

Leave a reply

Sci-Hub, the web service with over 60 million academic papers, released the DOIs of its article holdings earlier this year:

https://twitter.com/Sci_Hub/status/843546352219017218

The data was quickly put into a figshare repository (Hahnel, 2017), and some analysis has already been done by Greshake (2017) on this list of DOIs.

Here I want do some simple analysis with this dataset, and look at how many papers published by AGU are part of this collection. Keep in mind a few things:

  1. I believe in acquiring papers through legal means, and do not advocate searching for/using illegally distributed copies (through Sci-Hub or ResearchGate). New great tools to look for free versions of manuscripts are Unpaywall and oadoi.
  2. Papers in AGU journals from 1997 – 24 months ago are freely available, so many of the papers on Sci-Hub are already free.

OK, back to the Sci-Hub dataset: article Digital Object Identifiers (DOIs), are broken into 2 parts: the prefix and a suffix. As I understand it, prior to being published by Wiley, AGU articles used the prefix (10.1029), so I first extracted all entries with this older prefix. This really restricts this analysis to pre-2013 AGU articles (I’m not sure when exactly the change occurred), but some of the older articles might be published prior to the 1997 ‘open access’ cutoff.

I was left with a list of 171,752 articles (from the original 62 million).

The suffix of AGU DOIs corresponds to a single article from a specific journal, and letter codes in the suffix are used to denote the journal — for example, a GRL article has a suffix that includes ‘GL’ or ‘gl’.

For example, a GRL article has a DOI that looks like this: 10.1029/2006GL028162

Each AGU journal has a unique letter combo in the suffix, so the list of DOIs can be counted based on this suffix. Here is the line of julia code that I used to search through the list of ‘10.1029’ DOIs to find GRL articles.

GRL=length(matchall(r"gl"i,s))

Parsing the 171,752 articles into specific journals yields:

Papers.jpg

  • For these AGU journals, GRL has the most articles on Sci-Hub, note also the huge volume of EOS articles (?!?), and Water Resources Research.
  • The older and higher volume JGR sections (Oceans, Atmopsheres, Solid Earth and Space Physics) outweigh the newer, smaller sections (Biogeoscience, Earth Surface, Planets). Here are some JGR publication stats.
  • ~5% of the 171,752 AGU DOIs did not conform to this search — they might be books, chapters, or other documents.

Here is another interesting article on Sci-Hub, from Science (Bohannon, 2016): Who’s Downloading Pirated Papers? Everyone

‘Sleeping Beauties’ of Geomorphology: cases from the Journal of Geology

Leave a reply

To recap from a previous post:

“Most papers in disciplinary geomorphology journals are cited at some point, but citations to papers do not always accrue immediately upon publication — ideas and papers might take time to be used by researchers and therefore cited. Extreme examples of delayed recognition (‘Sleeping Beauties‘) — where papers receive no citations for long stretches of time only to receive a large, late burst in citations — have been identified and investigated previously.

Do geomorphology ‘Sleeping Beauties’ exist? Using the methods of Ke et al. (2015) to find and score ‘Sleeping Beauties’, it turns out that 9 out of the 20 most delayed papers in GSA Bulletin are focused on quantitative geomorphology. What other papers show this interesting signature of delayed recognition?”

Today I want to look for Sleeping Beauties in ‘The Journal of Geology‘. 

JG has been published since 1893, and has been the venue for some classic geomorphology papers (e.g., Wolman And Miller, 1960; Magnitude and Frequency of Forces in Geomorphic Processes; which i will discuss in a future post..)

In January 2017 I downloaded the citation time series for the 500 most cited journal of geology articles and used the algorithm of Ke et al. (2015) to find papers with the highest ‘delayed recognition’ score — a ranking of each paper’s citation time series based on the largest, latest peak (read Ke et al. (2015) to learn more about the method).

The top for papers, published from 1922 to 1935, are all focused on grain size and shape:

  1. Wentworth, C. K. (1922). A scale of grade and class terms for clastic sediments. The Journal of Geology, 30(5), 377-392. (pdf here)
  2. Wadell, H. (1935). Volume, shape, and roundness of quartz particles. The Journal of Geology, 43(3), 250-280. (article here)
  3. Wadell, H. (1932). Volume, shape, and roundness of rock particles. The Journal of Geology, 40(5), 443-451. (article here)
  4. Wadell, H. (1933). Sphericity and roundness of rock particles. The Journal of Geology, 41(3), 310-331. (article here)

The citation time series for each paper is shown below:JG.jpg

As with the last post, I will not offer any ‘reasons’ why these papers have an explosion in citations in the past 10 years. To do this, a first step would be a careful look at co-citation networks — what papers often co-occur with the citations — and the actual in-text usages and citations.

I did a cursory look at co-cited papers, and all of the papers show an affinity to two recent well-cited papers:

  • Blott, S. J., & Pye, K. (2001). GRADISTAT: a grain size distribution and statistics package for the analysis of unconsolidated sediments. Earth Surface Processes and Landforms, 26(11), 1237-1248. http://doi.org/10.1002/esp.261

  • Blott, S. J., & Pye, K. (2008). Particle shape: a review and new methods of characterization and classification. Sedimentology, 55(1), 31-63. http://doi.org/10.1111/j.1365-3091.2007.00892.x

Last I looked Blott and Pye (2001) was the most cited paper in ESPL, and is cited in a policy document, a rare occurrence for a geomorphology paper.

‘Sleeping Beauties’ of Geomorphology: a case from the American Journal of Science

Leave a reply

Most papers in disciplinary geomorphology journals are cited at some point, but citations to papers do not always accrue immediately upon publication — ideas and papers might take time to be used by researchers and therefore cited. Extreme examples of delayed recognition (‘Sleeping Beauties‘) — where papers recieve no citations for long stretches of time only to recieve a large, late burst in citations — have been identified and investigated previously.

Do geomorphology ‘Sleeping Beauties’ exist? Using the methods of Ke et al. (2015) to find and score ‘Sleeping Beauties’, it turns out that 9 out of the 20 most delayed papers in GSA Bulletin are focused on quantitative geomorphology.

What other papers show this interesting signature of delayed recognition?

I have looked in other journals and found a few neat examples, which I hope to chronicle in a series of posts. Today, I will look at an example from the American Journal of Science (AJS):

The AJS has been published since 1818, and has long been a geology venue. In January 2017  I downloaded the 500 most cited AJS articles from the Web of Science. I used the algorithm presented in Ke et al. (2015) to find the papers with the highest ‘delayed recognition’ score — a ranking of each paper’s citation time series based on the largest, latest peak  (I urge you all to read Ke et al. (2015) which describes the method).

The most delayed paper is about brachiopods, but I want to focus on research related to geomorphology, so let’s look at the 2nd most delayed paper:

W.W.Rubey (1933): Settling velocities of gravel, sand, and silt particles. Am J Sci April 1, 1933 Series 5 Vol. 25:325-338; doi:10.2475/ajs.s5-25.148.325

(n.b., settling velocity has a special place in my heart)

Rubey’s paper has a score that is similar to the papers from GSA Bulletin. Here is the citation time series for the Rubey paper:Rubey CTS.jpg

So the natural quesiton is —  what happened that caused this 2014 burst of citations? As far as I can tell (from looking at the papers that cited Rubey), nothing in particular… Most papers that cite Rubey are focused on typical sediment transport questions. A close read of all the citing papers would be needed to figure out what is going on here, if there is some ‘signal’. Not a satisfying answer, and I apologize —leave a comment if you have an idea and I’ll update the post if I find anything out.

Web Mentions of JGR-ES Articles

Leave a reply

How many Journal of Geophysical Research – Earth Surface articles are mentioned on the web? And on what venues? Below, the percentage of JGR-ES papers published in a given year with at least one mention on a given platform:

figure_1.jpeg

This is a follow-up on my guest post on the ‘The Plainspoken Scientist’ blog regarding the percent of Geophysical Research Letters (GRL) papers mentioned somewhere on the web.

JGR-ES articles still lag behind GRL Twitter mentions, but it’s nice to see the increase in for JGR-ES papers.. All other sources seem to be marginal.