{"id":1341,"date":"2017-01-03T07:03:46","date_gmt":"2017-01-03T12:03:46","guid":{"rendered":"http:\/\/datacolada.org\/?p=1341"},"modified":"2021-12-17T01:06:09","modified_gmt":"2021-12-17T06:06:09","slug":"56-twarking-test-weighting-results-known","status":"publish","type":"post","link":"https:\/\/datacolada.org\/56","title":{"rendered":"[56] TWARKing: Test-Weighting After Results are Known"},"content":{"rendered":"

On the last class of the semester I hold a \u201ctown-hall\u201d meeting; an open discussion about how to improve the course (content, delivery, grading, etc). I follow-up with a required online poll to \u201cvote\u201d on proposed changes [1<\/a>].
\n\"f1\"
\nGrading in my class is old-school. Two tests, each 40%, homeworks 20% (graded mostly on a completion 1\/0 scale). The downside of this model is that those who do poorly early on, get demotivated. Also, a bit of bad luck in a test hurts a lot. During the latest town-hall the idea of having multiple quizzes and dropping the worst was popular. One problem with this<\/em> model is that students can blow off a quiz entirely. After the town-hall I thought of why<\/u><\/em> students loved the drop-1 idea and whether I could capture the same psychological benefit with a smaller pedagogical loss.<\/p>\n

I came up with TWARKing: assigning test weights after results are known [2<\/a>]. With TWARKing, instead of each test counting 40% for every student, whichever test an individual student did better on, gets more weight; so Julie does better in Test 1 than Test 2, then Julie\u2019s test 1 gets 45% and test 2 35%, but Jason did better in Test 2, so Jason\u2019s test 2 gets 45%. [3<\/a>]. Dropping a quiz becomes a special case of TWARKing: worst gets 0% weight.<\/p>\n

It polls well<\/strong>
\nI expected TWARKing to do well in the online poll but was worried students would fall prey to competition-neglect, so I wrote a long question stacking the deck against TWARKing:
\n\"question\"<\/a><\/p>\n

\"f1\"<\/a><\/p>\n

70% of student were in favor, only 15% against (N=92, only 3 students did not complete the poll).<\/p>\n

The poll is not anonymous, so I looked at how TWARKing attitudes are correlated with actual performance.<\/p>\n

\"f2\"<\/a><\/p>\n

Panel A shows that students doing better like TWARking less, but the effect is not as strong as I would have expected. Students liking it 5\/5 perform in the bottom 40%, those liking 2\/5 are in the top 40%.<\/p>\n

Panel B shows that students with more uneven performance do like the TWARKing more, but the effect is small and unimpressive (Spearman's r=.21:, p<\/em>=.044).<\/p>\n

For Panel C I recomputed the final grades had TWARKing been implemented for this semester and saw how the change in ranking correlated with support of TWARKing. It did not. Maybe it was asking too much for this to work as students did not yet know their Test 2 scores.<\/p>\n

My read is that students cannot anticipate if it will help vs. hurt them, and they generally like it all the same.<\/p>\n

TWARKing could be pedagogically superior.
\n<\/strong>Tests serve two main roles: motivating students and measuring performance. I think TWARKing could be better on both fronts.<\/p>\n

Better measurement<\/strong>. <\/em>M<\/em>y tests tend to include insight-type questions: students either nail them or fail them. It is hard to get lucky in my tests, I think, hard to get a high score despite not knowing the material. But, easy, unfortunately, to get unlucky; to get no points on a topic you had a decent understanding of [4<\/a>].\u00a0 Giving more weight to the highest test is hence giving more weight to the more accurate of the two tests. \u00a0So it could improve<\/u><\/em> the overall validity of the grade.\u00a0 A student who gets a 90 and a 70 is, I presume, better than one getting 80 in both tests.<\/p>\n

This reminded me of what Shugan & Mitra (2009 .htm<\/a>) label the \u201cAnna Karenina effect\u201d in their under-appreciated paper (11 Google cites). Their<\/em> Anna Karenina effect (there are a few; each different from the other), occurs when less favorable outcomes carry less information than more favorable ones; for those situations, measures other than the average, e.g., the max, performs better for out-of-sample prediction. [5<\/a>]\n

To get an intuition for this Anna Karenina effect: think about what contains more information, a marathon runner\u2019s best vs worst running time? A researcher\u2019s most vs least cited paper?<\/p>\n

Note that one can TWARK within test, weighting the highest scored answer by each student more. I will.<\/p>\n

Motivation<\/strong>. <\/em>After doing very poorly in a test it must be very motivating to feel that if you study hard you can make this bad performance count less. I speculate that with TWARKing underperforming students in Test 1 are less likely to be demotivated for Test 2 (I will test this next semester, but without random assignment\u2026). \u00a0TWARKing has the magical psychological property that the gains are very concrete, every single student gets a higher average with TWARKing than without, and they see that; the losses, in contrast, are abstract and unverifiable (you don\u2019t see the students who benefited more than you did, leading to a net-loss in ranking).<\/p>\n

Bottom line<\/strong>
\nStudents seem to really like TWARKing.
\nIt may make things better for measurement.
\nIt may improve motivation.<\/p>\n

A free happiness boost.<\/p>\n

\"Wide<\/p>\n

\n

Subscribe to Blog via Email<\/h2>\n\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t\t\t

Enter your email address to subscribe to this blog and receive notifications of new posts by email.<\/p>\n<\/div>\n\t\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t