Skip to content

Address A7 (CHI 2025) human-in-loop counter-evidence in paper §7 #16

@UditAkhourii

Description

@UditAkhourii

External research review by u/mxriverlynn flagged a peer-reviewed CHI 2025 study (Yagil et al., "No Evidence for LLMs Being Useful in Problem Reframing", arXiv 2503.01631) as direct counter-evidence to the human-in-the-loop application of ADHD-style reframing.

The study: Controlled study of 280 design professionals. Found no statistically significant improvement in frame novelty or usefulness when LLMs were used for problem reframing across direct, structured, and free-form integration conditions. LLM use amplified the expert-novice gap.

Why this matters: the paper's "use cases" section (architecture decisions, naming, strategy/positioning) is dominated by human-in-the-loop contexts, but the paper does not cite or address A7.

Action:

  • Add a paragraph to §7 (Discussion and Limitations) explicitly acknowledging A7's counter-evidence for human-in-the-loop reframing.
  • Distinguish ADHD's primary context (LLM-to-LLM subroutine at decision points inside an agent loop) from A7's measured context (designers using LLMs to reframe their own problems).
  • Note honestly that the human-in-the-loop applicability of ADHD is not empirically supported and is a target for future evaluation.

This pre-empts the same critique at v1.0 from a more hostile reader.


Raised by u/mxriverlynn in adhd-application-to-han.md — a detailed external research review using Han's own /research skill.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationmethodologyMethodology critique or clarificationpaperPreprint paper updates

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions