{"id":8206,"date":"2024-09-16T07:00:25","date_gmt":"2024-09-16T11:00:25","guid":{"rendered":"https:\/\/datacolada.org\/?p=8206"},"modified":"2024-09-16T09:22:15","modified_gmt":"2024-09-16T13:22:15","slug":"120-off-label-smirnov-how-many-subjects-show-an-effect-in-between-subject-experiments","status":"publish","type":"post","link":"https:\/\/datacolada.org\/120","title":{"rendered":"[120] Off-Label Smirnov: How Many Subjects Show an Effect in Between-Subjects Experiments?"},"content":{"rendered":"

There is a classic statistical test known as the Kolmogorov-Smirnov (KS) test (Wikipedia<\/a>).<\/span><\/p>\n

This post is about an off-label use of the KS-test that I don\u2019t think people know about (not even Kolmogorov or Smirnov), and which seems useful for experimentalists in behavioral science and beyond (most useful, I think, for clinical trials and field experiments of policies that could backfire on some people).<\/span><\/p>\n

If I\u2019m wrong and the off-label use is known, well, that\u2019s a little embarrassing, but please let me know.<\/span><\/p>\n

Like the t-test, the KS-test can be used to compare two samples1<\/a><\/sup>The KS-test can also be used to compare one sample to a theoretical distribution, e.g., to assess if a sample is statistically significantly not-normally distributed. We relied on it to when assessing whether the Ariely insurance driving data were uniformly distributed, see footnote 5 in Colada[98]<\/a>\u00a0 <\/span>.<\/span><\/p>\n

While the t-test involves only differences of means, <\/em>the KS-test considers any type of difference across samples. So, like the t-test, the KS-test could be statistically significant because the sample means are quite different, but, unlike the t-test, the KS-test could also be statistically significant because the sample variances are quite different, or the share of observations that are exactly 0 are quite different, etc.<\/span><\/p>\n

The KS test is seldom used, but when it is used, it is used as a a test, <\/em>as a way to obtain a p<\/em>-value quantifying whether things are significantly different across groups2<\/a><\/sup>Or one group, see Footnote 1<\/span>.<\/span><\/p>\n

But, there is an off-label use of the KS-test, one that does not use it as a test, but as a way to estimate<\/em> something. Specifically, it can estimate the share of observations in a between-subjects design that are impacted by the manipulation. So, instead of asking \u201cIs the average<\/em> of the dependent variable higher in A than in B?\u201d, we can ask \u201cHow many<\/em> people have a higher dependent variable in A than in B\u201d? Again. This is noteworthy because it involves between<\/u><\/em> subjects design where people are only in A or only in B so we don't know if any given individual has a higher DV in A or in B 3<\/a><\/sup>Technically, the KS test does not estimate the share showing an effect, it bounds it, meaning it tells you its smallest possible value, rather than its expected value. More on this later.<\/span>.<\/span><\/p>\n

\"\"<\/a><\/p>\n

What does the KS-test do?
\n<\/strong><\/span>The KS test compares the entire distribution of values between samples (the CDFs). It quantifies the biggest observed difference in those distributions. Let's illustrate with data from an old paper of mine (.pdf<\/a>) in which participants provided their valuation, in a between subject design, for either a $50 Gift Certificate, or for a 50:50 gamble that for sure paid either that $50 certificate, or a $100 one (people, puzzlingly, pay less<\/em> for the lottery, the objectively superior alternative; this is known as the \u201cUncertainty Effect\u201d.htm<\/a>). The figure below has the CDFs. For example, if we were to draw a horizontal line (that unfortunately STATA didn't draw) at the 50th<\/sup> percentile, the medians are about $10 and $35. About half the people pay $10 or less for the lottery, and about half the people pay $35 or less for the $50 gift certificate.<\/span><\/p>\n

\"\"<\/a>
\nFig 1.<\/strong> Distribution of valuations an the D+ and D- in a KS-test.<\/span><\/p>\n

OK, let's now get to the KS test. It computes the biggest positive and negative differences in these CDFs. The biggest positive difference is about 55%, seen around $10.\u00a0<\/span>In the KS-test that's the D+<\/sup><\/strong><\/em>\u00a0(test) statistic, the biggest positive difference. <\/span>There is also the D–<\/sup>, flagged here with the 2nd<\/sup> arrow; D– <\/sup><\/strong><\/em>= 18%.<\/span><\/p>\n

Researchers and statisticians usually ignore those D+ and D- values, they usually do not even report them. But it is those D+ and D- that give us the off-label use. Those D values inform <\/span>the share of people showing each effect. Returning to Figure 1 above, the Ds mean that at least<\/em> 55% of participants pay more for the $50 gift certificate than for the lottery, and at least<\/em> 18% of participants do the opposite. <\/span><\/p>\n

The KS-test off-label use involves using D+ and D- as bounds on the share of people impacted positively vs negatively by treatment.<\/span><\/strong><\/p>\n

I should say I did not discover or prove this fact; a paper published in Econometric Theory <\/em>in 2010<\/span> did. But the authors do not seem to have realized that that's what they proved (they do not mention the KS test in their paper). I wrote my personal journey to acquiring this tidbit of information, but it is long, so I put it behind the green button.<\/span><\/p>\n