Grumpy Rumblings (of the formerly untenured)

DP asks:

Have you seen the paper saying that iPhones are responsible for fertility decline?  I recently saw Courtney Milan tearing apart an economics paper by Meyers and Hooper.  What do you think about the paper?

Disclaimer:  One of the authors is a friend of mine who I think does really solid work. I’m also a huge fangirl of Courtney Milan and one of my big accomplishments in life was winning an auction that allowed me to name a character in one of her books.  She also sent me a washi tape.  (I don’t like the way Courtney Milan is slagging off Caitlin Myers as some kind of natalist when Caitlin Myers almost single handedly organized an enormous Amicus brief summarizing all of the credible research on the positive effects of abortion availability and signed by multiple Nobel prize winners and other experts for Dobbs v. Jackson Women’s Health Organization that the Supreme Court justices ignored.  I want to shout, “You’re on the same side!” but not enough to get a BlueSky account.)

The paper in question is this one.  It is still a working paper– has not been published, but it did get a shout-out from the NBER LinkedIn feed which it doesn’t do for all of their working papers.

Courtney Milan comments on an NPR story and pulls out one of the sets of graphs from the paper in this thread here.

As economists know, correlation is not causation, and you cannot just look at post-trends after a change.  Courtney Milan is 100% correct that that would be a really bad research design.

What Myers and Hooper do is look at the location of AT&T stores between 2007 and 2011 because the iPhone was only sold on AT&T during this time period.  Other markets instead had Verizon or Sprint, which offered service, but did not offer iPhone service.

It is true that markets with AT&T are potentially different than markets with Verizon or Sprint.  Thus, if you just measured the comparison of fertility AT&T after the iPhone to Verizon/Sprint after the iPhone, you would get the true effect plus some sort of market bias.

It is also true that fertility before the iPhone would be different than fertility after the iPhone for non-iPhone related reasons.  If you just measured fertility differences post the iPhone you would be getting the true effect of the iPhone plus time bias.

The idea is that Verizon/Sprint markets will have the same time bias for fertility.  Or that market bias will be the same in the before iPhone period as the after iPhone period.  If that is true, then you can use a method called “Difference-in-Differences” to basically use the Verizon/Sprint market as a control for the AT&T market, and the before period as a control for the after period.

Fertility_AT&T_after – Fertility_AT&T_after = change in AT&T fertility = true effect + time bias

Fertility_Sprint/Verizon_after – Fertility_Sprint/Verizon_after = change in Sprint fertility = time bias

Then you subtract (change in AT&T fertility) – (change in Sprint fertility) to get the true effect.  This is the second difference where you subtract out the time bias.

Now a lot of things have to be true for this to give you the actual true effect– remember those assumptions above?  You have to make sure that the trends in those Sprint/Verizon markets don’t have different trends in the pre-period, because then you’re not actually measuring just time bias.  You have to make sure nothing else is happening the exact time the iPhone is introduced that would affect fertility differentially in AT&T markets vs. only Sprint/Verizon markets (like, those aren’t the places where they’re putting in new Planned Parenthoods or something, or AT&T stores are also the only places that sell specific gaming systems or are handing out free condoms… things like that)– A big concern here might be that counties with AT&T are differentially affected by the great recession compared to those without AT&T.  You also may want to make sure that you’re not including new AT&T stores that sprang up because of the iPhone, because those were potentially selected into markets in a way that might be related to fertility or other iPhone demographics (you might then instrument actual stores with original stores).

Here’s Caitlin Myers’ explanation of the paper in question from LinkedIn.

Looking at the paper more carefully– they’re not quite doing a straight-up DID.  Instead they’re doing a synthetic DID event study.  I don’t like these as much as doing a regular DID and then directly testing the assumptions.  Generally when someone is doing Synthetic DID it’s because one or more of the assumptions were violated (but sometimes people do it just to get a more precise estimate since it gets rid of the difference in the pre-period, or because either the treatment or the control group is small).  So this method is one of the many methods that matches and/or weights on observables in the pre-period (gross oversimplification– this isn’t technically propensity score or coarsened exact matching… but…) to get things to line up and then kind of hopes that the unobservables, the true omitted variables, will follow.  They’re sort of “in” right now, and have been for like the past 10 years give or take.  I don’t know that it’s better than doing coarsened exact matching and it’s possibly better than propensity score matching (because you throw out less stuff), but for whatever reason people trust it more (maybe because Alberto Abadie has been pushing it).

In terms of the sniff test– the authors find that about a third of the drop in fertility during this time period is from smart phones.  It still seems a bit large to me because we’re talking about early smart phone roll out and the biggest effects that they’re finding are on 15-19 year olds, and from other work I know that the biggest recent fertility drops overall are from the 15-19 group– and I just can’t think that this expensive phone is being taken up by a lot of teenagers, particularly the lower income teens who are seeing the biggest drops.  There wasn’t a robust used iPhone market yet and iPhones were pricey.  Heck, I didn’t get a smart phone until like 2015.  It might make sense for the older groups who have more disposable income, but they’re not really seeing big drops for those groups.  Or maybe the explanation is that parents are lending their kids their phones?

That said, the fact that they don’t see fertility changes for groups that shouldn’t be affected is good.  That they show increases in porn watching and decreases in people spending time with each other, both of which could decrease fertility, is good for their story. The fact they don’t see effects with Verizon only (ignoring other groups) is good.  The fact that they do actually see some effects with Sprint (in Appendix B 6) is less good.  The standard errors are larger but the patterns look the same.  I’m not clear on what the control is here or how correlated Sprint coverage is with AT&T coverage.

So.. bottom line… I haven’t seen the paper presented and it isn’t published yet.  They are aware of the simple correlation is not causation criticisms and they’re using a technique that is currently in vogue that is based on a solid technique but maybe isn’t as solid as we’d like it to be.  Personally I’ve thrown away projects rather than using synthetic DID.  I do think Courtney Milan’s general criticism is true, but not with respect to this paper.  They aren’t just looking at the post-period; they are using the pre-period as a control.  I’m not sure I’m convinced that the necessary assumptions hold, but I’m not unconvinced either.  There are things that look very reassuring (the mechanism questions, some of the placebo work and robustness checks), but there are some that look a little off.  For the things that are a little off my thought is that it’s likely that there is really an effect, but maybe it’s not quite as large as it seems, or the heterogeneity magnitudes are off.

Most of Grumpy Nation is probably in the “not affected” age group, but do you think smart phone use could affect youth fertility?