Who Owns Creativity and Who Does the Work? Trade-offs in LLM-Supported Research Ideation.

Matthew Lease Matt Lease

Houjiang Liu Yujin Choi Sanjana Gautam Gabriel Jaffe Soo Young Rieh Matthew Lease Who Owns Creativity and Who Does the Work? Trade-offs in LLM-Supported Research Ideation. 2026 January abs/2601.12152 CoRR https://doi.org/10.48550/arXiv.2601.12152 db/journals/corr/corr2601.html#abs-2601-12152 streams/journals/corr

Soumyajit Gupta Venelin Kovatchev Anubrata Das 0001 Maria De-Arteaga Matthew Lease Finding Pareto trade-offs in fair and accurate detection of toxic speech. 123-141 2025 30 Inf. Res. iConf https://doi.org/10.47989/ir30iconf47572 db/journals/ires/ires30.html#GuptaKDDL25 streams/journals/ires

Houjiang Liu Jacek Gwizdka Matthew Lease Explgagement With Misinformation During a Health Crisis. 1-53 2025 9 Proc. ACM Hum. Comput. Interact. 7 https://doi.org/10.1145/3757479 db/journals/pacmhci/pacmhci9.html#LiuGL25

Houjiang Liu Jacek Gwizdka Matthew Lease Exploring Multidimensional Checkworthiness: Designing AI-assisted Claim Prioritization for Human Fact-checkers. 1-49 2025 9 Proc. ACM Hum. Comput. Interact. 7 https://doi.org/10.1145/3757473 db/journals/pacmhci/pacmhci9.html#LiuGL25a streams/journals/pacmhci

Gianluca Demartini Claudia Hauff Matthew Lease Stefano Mizzaro Kevin Roitero Mark Sanderson Falk Scholer Chirag Shah 0001 Damiano Spina Paul Thomas 0001 Arjen P. de Vries Guido Zuccon Preaching to the ChoIR: Lessons IR Should Share with AI. 78-91 2025 ICTIR https://doi.org/10.1145/3731120.3744612 conf/ictir/2025 db/conf/ictir/ictir2025.html#DemartiniHLMRSS25

Houjiang Liu Yiheng Su Matthew Lease Rhetorical XAI: Explaining AI's Benefits as well as its Use via Rhetorical Design. 2025 May abs/2505.09862 CoRR https://doi.org/10.48550/arXiv.2505.09862 db/journals/corr/corr2505.html#abs-2505-09862 streams/journals/corr

Gauri Kambhatla Sanjana Gautam Angela Zhang Alex Liu Ravi Srinivasan Junyi Jessy Li Matthew Lease Beyond Sociodemographic Prompting: Using Supervision to Align LLMs with Human Response Distributions. 2025 July abs/2507.00439 CoRR https://doi.org/10.48550/arXiv.2507.00439 db/journals/corr/corr2507.html#abs-2507-00439 streams/journals/corr

Chi-Yang Hsu Alexander Braylan Yiheng Su Omar Alonso Matthew Lease Instance-level Performance Prediction for Long-form Generation Tasks. 2025 September abs/2509.07309 CoRR https://doi.org/10.48550/arXiv.2509.07309 db/journals/corr/corr2509.html#abs-2509-07309 streams/journals/corr

Luis Oala Manil Maskey Lilith Bat-Leah Alicia Parrish Nezihe Merve Gürel Tzu-Sheng Kuo Yang Liu 0018 Rotem Dror Danilo Brajovic Xiaozhe Yao Max Bartolo William Gaviria Rojas Ryan Hileman Rainier Aliment Michael W. Mahoney Meg Risdal Matthew Lease Wojciech Samek Debojyoti Dutta Curtis G. Northcutt Cody Coleman Braden Hancock Bernard Koch Girmaw Abebe Tadesse Bojan Karlas Ahmed Alaa 0001 Adji Bousso Dieng Natasha F. Noy Vijay Janapa Reddi James Zou 0001 Praveen K. Paritosh Mihaela van der Schaar Kurt Bollacker Lora Aroyo Ce Zhang 0001 Joaquin Vanschoren Isabelle Guyon Peter Mattson DMLR: Data-centric Machine Learning Research - Past, Present and Future. 2024 (5):1-27 1 J. Data-centric Mach. Learn. Res. https://data.mlr.press/assets/pdf/v01-5.pdf db/journals/dmlr/dmlr1.html#OalaMBPGKLDBYBRHAMRLSDN24 streams/journals/dmlr

Houjiang Liu Anubrata Das 0001 Alexander Boltz Didi Zhou Daisy Pinaroc Matthew Lease Min Kyung Lee Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI. 1-44 2024 8 Proc. ACM Hum. Comput. Interact. CSCW2 https://doi.org/10.1145/3686962 db/journals/pacmhci/pacmhci8.html#Liu0BZPLL24

Alexander Braylan Madalyn Marabella Omar Alonso Matthew Lease A General Model for Aggregating Annotations AcrossSimple, Complex, and Multi-object Annotation Tasks (Abstract Reprint). 22693 2024 AAAI https://doi.org/10.1609/aaai.v38i20.30593 conf/aaai/2024 db/conf/aaai/aaai2024.html#BraylanMAL24 Gauri Kambhatla Matthew Lease Ashwin Rajadesingan Promoting Constructive Deliberation: Reframing for Receptiveness. 5110-5132 2024 EMNLP (Findings) https://doi.org/10.18653/v1/2024.findings-emnlp.294 https://aclanthology.org/2024.findings-emnlp.294 conf/emnlp/2024f db/conf/emnlp/emnlp2024f.html#KambhatlaLR24 Alex C. Williams Min Bai Jonathan Buck Tristan J. McKinney Amy Rechkemmer Koushik Kalyanaraman Matthew Lease Patrick Haffner Xiong Zhou Li Erran Li Snapper: Accelerating Bounding Box Annotation in Object Detection Tasks with Find-and-Snap Tooling. 471-488 2024 IUI https://doi.org/10.1145/3640543.3645162 conf/iui/2024 db/conf/iui/iui2024.html#WilliamsBBMRKLH24 Venelin Kovatchev Matthew Lease Benchmark Transparency: Measuring the Impact of Data on Evaluation. 1536-1551 2024 NAACL-HLT https://doi.org/10.18653/v1/2024.naacl-long.86 conf/naacl/2024 db/conf/naacl/naacl2024.html#KovatchevL24 Subramanian Chidambaram Alex C. Williams Min Bai Satyugjit Virk Patrick Haffner Matthew Lease Li Erran Li Annorama: Enabling Immersive At-Desk Annotation Experiences in Virtual Reality with 3D Point Cloud Dioramas. 16:1-16:13 2024 SUI https://doi.org/10.1145/3677386.3682081 conf/sui/2024 db/conf/sui/sui2024.html#ChidambaramWBVH24

Terrence Neumann Sooyong Lee Maria De-Arteaga Sina Fazelpour Matthew Lease Diverse, but Divisive: LLMs Can Exaggerate Gender Differences in Opinion Related to Harms of Misinformation. 2024 abs/2401.16558 CoRR https://doi.org/10.48550/arXiv.2401.16558 db/journals/corr/corr2401.html#abs-2401-16558

Venelin Kovatchev Matthew Lease Benchmark Transparency: Measuring the Impact of Data on Evaluation. 2024 abs/2404.00748 CoRR https://doi.org/10.48550/arXiv.2404.00748 db/journals/corr/corr2404.html#abs-2404-00748

Gauri Kambhatla Matthew Lease Ashwin Rajadesingan Promoting Constructive Deliberation: Reframing for Receptiveness. 2024 abs/2405.15067 CoRR https://doi.org/10.48550/arXiv.2405.15067 db/journals/corr/corr2405.html#abs-2405-15067

Soumyajit Gupta Venelin Kovatchev Maria De-Arteaga Matthew Lease Fairly Accurate: Optimizing Accuracy Parity in Fair Target-Group Detection. 2024 abs/2407.11933 CoRR https://doi.org/10.48550/arXiv.2407.11933 db/journals/corr/corr2407.html#abs-2407-11933 streams/journals/corr

Li Shi Houjiang Liu Yian Wong Utkarsh Mujumdar Dan Zhang 0004 Jacek Gwizdka Matthew Lease Argumentative Experience: Reducing Confirmation Bias on Controversial Issues through LLM-Generated Multi-Persona Debates. 2024 abs/2412.04629 CoRR https://doi.org/10.48550/arXiv.2412.04629 db/journals/corr/corr2412.html#abs-2412-04629 streams/journals/corr

Houjiang Liu Jacek Gwizdka Matthew Lease Exploring Multidimensional Checkworthiness: Designing AI-assisted Claim Prioritization for Human Fact-checkers. 2024 abs/2412.08185 CoRR https://doi.org/10.48550/arXiv.2412.08185 db/journals/corr/corr2412.html#abs-2412-08185 streams/journals/corr

Jie Yang 0028 Alessandro Bozzon Ujwal Gadiraju Matthew Lease Editorial: Human-centered AI: Crowd computing. 2023 February 6 Frontiers Artif. Intell. https://doi.org/10.3389/frai.2023.1161006 db/journals/frai/frai6.html#YangBGL23

Anubrata Das 0001 Houjiang Liu Venelin Kovatchev Matthew Lease The state of human-centered NLP technology for fact-checking. 103219 2023 60 Inf. Process. Manag. 2 https://doi.org/10.1016/j.ipm.2022.103219 db/journals/ipm/ipm60.html#0001LKL23

Alexander Braylan Madalyn Marabella Omar Alonso Matthew Lease A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks. 901-973 2023 78 J. Artif. Intell. Res. https://doi.org/10.1613/jair.1.14388 db/journals/jair/jair78.html#BraylanMAL23

Mehmet Deniz Türkmen Matthew Lease Mücahid Kutlu New Metrics to Encourage Innovation and Diversity in Information Retrieval Approaches. 239-254 2023 ECIR (2) https://doi.org/10.1007/978-3-031-28238-6_16 conf/ecir/2023-2 db/conf/ecir/ecir2023-2.html#TurkmenLK23 Vijay Keswani L. Elisa Celis Krishnaram Kenthapadi Matthew Lease Designing Closed-Loop Models for Task Allocation. 17-32 2023 HHAI https://doi.org/10.3233/FAIA230072 conf/hhai/2023 db/conf/hhai/hhai2023.html#KeswaniCKL23 Soumyajit Gupta Sooyong Lee Maria De-Arteaga Matthew Lease Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection. 3689-3700 2023 WWW https://doi.org/10.1145/3543507.3583290 conf/www/2023 db/conf/www/www2023.html#GuptaLDL23

Anubrata Das 0001 Houjiang Liu Venelin Kovatchev Matthew Lease The State of Human-centered NLP Technology for Fact-checking. 2023 abs/2301.03056 CoRR https://doi.org/10.48550/arXiv.2301.03056 db/journals/corr/corr2301.html#abs-2301-03056

Mehmet Deniz Türkmen Matthew Lease Mücahid Kutlu New Metrics to Encourage Innovation and Diversity in Information Retrieval Approaches. 2023 abs/2301.08062 CoRR https://doi.org/10.48550/arXiv.2301.08062 db/journals/corr/corr2301.html#abs-2301-08062

Ruijiang Gao Maytal Saar-Tsechansky Maria De-Arteaga Ligong Han Wei Sun 0031 Min Kyung Lee Matthew Lease Learning Complementary Policies for Human-AI Teams. 2023 abs/2302.02944 CoRR https://doi.org/10.48550/arXiv.2302.02944 db/journals/corr/corr2302.html#abs-2302-02944

Soumyajit Gupta Sooyong Lee Maria De-Arteaga Matthew Lease Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection. 2023 abs/2302.07372 CoRR https://doi.org/10.48550/arXiv.2302.07372 db/journals/corr/corr2302.html#abs-2302-07372

Vijay Keswani L. Elisa Celis Krishnaram Kenthapadi Matthew Lease Designing Closed-Loop Models for Task Allocation. 2023 abs/2305.19864 CoRR https://doi.org/10.48550/arXiv.2305.19864 db/journals/corr/corr2305.html#abs-2305-19864

Houjiang Liu Anubrata Das 0001 Alexander Boltz Didi Zhou Daisy Pinaroc Matthew Lease Min Kyung Lee Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI. 2023 abs/2308.07213 CoRR https://doi.org/10.48550/arXiv.2308.07213 db/journals/corr/corr2308.html#abs-2308-07213

Danula Hettiachchi Indigo Holcombe-James Stephanie Livingstone Anjalee de Silva Matthew Lease Flora D. Salim Mark Sanderson How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets. 2023 abs/2309.01288 CoRR https://doi.org/10.48550/arXiv.2309.01288 db/journals/corr/corr2309.html#abs-2309-01288

Yiheng Su Junyi Jessy Li Matthew Lease Interpretable by Design: Wrapper Boxes Combine Neural Performance with Faithful Explanations. 2023 abs/2311.08644 CoRR https://doi.org/10.48550/arXiv.2311.08644 db/journals/corr/corr2311.html#abs-2311-08644

Luis Oala Manil Maskey Lilith Bat-Leah Alicia Parrish Nezihe Merve Gürel Tzu-Sheng Kuo Yang Liu 0018 Rotem Dror Danilo Brajovic Xiaozhe Yao Max Bartolo William Gaviria Rojas Ryan Hileman Rainier Aliment Michael W. Mahoney Meg Risdal Matthew Lease Wojciech Samek Debojyoti Dutta Curtis G. Northcutt Cody Coleman Braden Hancock Bernard Koch Girmaw Abebe Tadesse Bojan Karlas Ahmed M. Alaa Adji Bousso Dieng Natasha F. Noy Vijay Janapa Reddi James Zou 0001 Praveen K. Paritosh Mihaela van der Schaar Kurt Bollacker Lora Aroyo Ce Zhang 0001 Joaquin Vanschoren Isabelle Guyon Peter Mattson DMLR: Data-centric Machine Learning Research - Past, Present and Future. 2023 abs/2311.13028 CoRR https://doi.org/10.48550/arXiv.2311.13028 db/journals/corr/corr2311.html#abs-2311-13028

Alexander Braylan Madalyn Marabella Omar Alonso Matthew Lease A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks. 2023 abs/2312.13437 CoRR https://doi.org/10.48550/arXiv.2312.13437 db/journals/corr/corr2312.html#abs-2312-13437

Vivek Krishna Pradhan Mike Schaekermann Matthew Lease In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers. 828187 2022 5 Frontiers Artif. Intell. https://doi.org/10.3389/frai.2022.828187 db/journals/frai/frai5.html#PradhanSL22

Lora Aroyo Matthew Lease Praveen K. Paritosh Mike Schaekermann Data excellence for AI: why should you care? 66-69 2022 29 Interactions 2 https://doi.org/10.1145/3517337 db/journals/interactions/interactions29.html#AroyoLPS22

Anubrata Das 0001 Chitrank Gupta Venelin Kovatchev Matthew Lease Junyi Jessy Li ProtoTEx: Explaining Model Decisions with Prototype Tensors. 2986-2997 2022 ACL (1) https://doi.org/10.18653/v1/2022.acl-long.213 https://aclanthology.org/2022.acl-long.213 https://www.wikidata.org/entity/Q131458510 conf/acl/2022-1 db/conf/acl/acl2022-1.html#0001GKLL22 Li Shi Nilavra Bhattacharya Anubrata Das 0001 Matt Lease Jacek Gwizdka The Effects of Interactive AI Design on User Behavior: An Eye-tracking Study of Fact-checking COVID-19 Claims. 315-320 2022 CHIIR https://doi.org/10.1145/3498366.3505786 https://www.wikidata.org/entity/Q130976297 conf/chiir/2022 db/conf/chiir/chiir2022.html#ShiB0LG22 Md. Mustafizur Rahman Mücahid Kutlu Matthew Lease Understanding and Predicting Characteristics of Test Collections in Information Retrieval. 136-148 2022 iConference (2) https://doi.org/10.1007/978-3-030-96960-8_10 conf/iconference/2022-2 db/conf/iconference/iconference2022-2.html#RahmanKL22 Prateek Chaudhry Matthew Lease You Are What You Tweet: Profiling Users by Past Tweets to Improve Hate Speech Detection. 195-203 2022 iConference (2) https://doi.org/10.1007/978-3-030-96960-8_13 conf/iconference/2022-2 db/conf/iconference/iconference2022-2.html#ChaudhryL22 Soumyajit Gupta Gurpreet Singh Raghu Bollapragada Matthew Lease Learning a neural Pareto manifold extractor with constraints. 749-758 2022 UAI https://proceedings.mlr.press/v180/gupta22a.html conf/uai/2022 db/conf/uai/uai2022.html#GuptaSBL22 Alexander Braylan Omar Alonso Matthew Lease Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks. 1720-1730 2022 WWW https://doi.org/10.1145/3485447.3512242 conf/www/2022 db/conf/www/www2022.html#BraylanAL22

Vijay Keswani Matthew Lease Krishnaram Kenthapadi Designing Closed Human-in-the-loop Deferral Pipelines. 2022 abs/2202.04718 CoRR https://arxiv.org/abs/2202.04718 db/journals/corr/corr2202.html#abs-2202-04718

Li Shi Nilavra Bhattacharya Anubrata Das 0001 Matthew Lease Jacek Gwizdka The Effects of Interactive AI Design on User Behavior: An Eye-tracking Study of Fact-checking COVID-19 Claims. 2022 abs/2202.08901 CoRR https://arxiv.org/abs/2202.08901 db/journals/corr/corr2202.html#abs-2202-08901

Anubrata Das 0001 Chitrank Gupta Venelin Kovatchev Matthew Lease Junyi Jessy Li ProtoTEx: Explaining Model Decisions with Prototype Tensors. 2022 abs/2204.05426 CoRR https://doi.org/10.48550/arXiv.2204.05426 db/journals/corr/corr2204.html#abs-2204-05426

Venelin Kovatchev Soumyajit Gupta Matthew Lease Fairly Accurate: Learning Optimal Accuracy vs. Fairness Tradeoffs for Hate Speech Detection. 2022 abs/2204.07661 CoRR https://doi.org/10.48550/arXiv.2204.07661 db/journals/corr/corr2204.html#abs-2204-07661

Venelin Kovatchev Trina Chatterjee Venkata Subrahmanyan Govindarajan Jifan Chen Eunsol Choi Gabriella Chronis Anubrata Das 0001 Katrin Erk Matthew Lease Junyi Jessy Li Yating Wu 0002 Kyle Mahowald longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. 2022 abs/2206.14729 CoRR https://doi.org/10.48550/arXiv.2206.14729 db/journals/corr/corr2206.html#abs-2206-14729

Alexander Braylan Omar Alonso Matthew Lease Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks. 2022 abs/2212.09503 CoRR https://doi.org/10.48550/arXiv.2212.09503 db/journals/corr/corr2212.html#abs-2212-09503

Caifan Du Matthew Lease Voices of Workers: Why a Worker-Centered Approach to Crowd Work Is Challenging. 2022 abs/2212.14471 CoRR https://doi.org/10.48550/arXiv.2212.14471 db/journals/corr/corr2212.html#abs-2212-14471

Danula Hettiachchi Mike Schaekermann Tristan McKinney Matthew Lease The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help. 332:1-332:26 2021 5 Proc. ACM Hum. Comput. Interact. CSCW2 https://doi.org/10.1145/3476073 db/journals/pacmhci/pacmhci5.html#HettiachchiSML21

Vijay Keswani Matthew Lease Krishnaram Kenthapadi Towards Unbiased and Accurate Deferral to Multiple Experts. 154-165 2021 AIES https://doi.org/10.1145/3461702.3462516 conf/aies/2021 db/conf/aies/aies2021.html#KeswaniLK21 Miriah Steiger Timir J. Bharucha Sukrit Venkatagiri Martin J. Riedl Matthew Lease The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support. 341:1-341:14 2021 CHI https://doi.org/10.1145/3411764.3445092 https://www.wikidata.org/entity/Q130966095 conf/chi/2021 db/conf/chi/chi2021.html#SteigerBVRL21 Danula Hettiachchi Mark Sanderson Jorge Gonçalves 0001 Simo Hosio Gabriella Kazai Matthew Lease Mike Schaekermann Emine Yilmaz Investigating and Mitigating Biases in Crowdsourced Data. 331-334 2021 CSCW Companion https://doi.org/10.1145/3462204.3481729 conf/cscw/2021c db/conf/cscw/cscw2021c.html#HettiachchiS0HK21 Soumyajit Gupta Gurpreet Singh Anubrata Das 0001 Matthew Lease Pareto Solutions vs Dataset Optima: Concepts and Methods for Optimizing Competing Objectives with Constraints in Retrieval. 43-52 2021 ICTIR https://doi.org/10.1145/3471158.3472248 conf/ictir/2021 db/conf/ictir/ictir2021.html#GuptaS0L21 Ruijiang Gao Maytal Saar-Tsechansky Maria De-Arteaga Ligong Han Min Kyung Lee Matthew Lease Human-AI Collaboration with Bandit Feedback. 1722-1728 2021 IJCAI https://doi.org/10.24963/ijcai.2021/237 conf/ijcai/2021 db/conf/ijcai/ijcai2021.html#GaoSDHLL21 Alexander Braylan Matthew Lease Aggregating Complex Annotations via Merging and Matching. 86-94 2021 KDD https://doi.org/10.1145/3447548.3467411 conf/kdd/2021 db/conf/kdd/kdd2021.html#BraylanL21 Md. Mustafizur Rahman Dinesh Balakrishnan Dhiraj Murthy Mücahid Kutlu Matt Lease An Information Retrieval Approach to Building Datasets for Hate Speech Detection. 2021 NeurIPS Datasets and Benchmarks https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/e00da03b685a0dd18fb6a08af0923de0-Abstract-round2.html conf/nips/2021db db/conf/nips/neurips2021db.html#RahmanBMKL21

Gurpreet Singh Soumyajit Gupta Matthew Lease Clint Dawson 0001 A Hybrid 2-stage Neural Optimization for Pareto Front Extraction. 2021 abs/2101.11684 CoRR https://arxiv.org/abs/2101.11684 db/journals/corr/corr2101.html#abs-2101-11684

Gurpreet Singh Soumyajit Gupta Matthew Lease Clint Dawson 0001 SCA-Net: A Self-Correcting Two-Layer Autoencoder for Hyper-spectral Unmixing. 2021 abs/2102.05713 CoRR https://arxiv.org/abs/2102.05713 db/journals/corr/corr2102.html#abs-2102-05713

Vijay Keswani Matthew Lease Krishnaram Kenthapadi Towards Unbiased and Accurate Deferral to Multiple Experts. 2021 abs/2102.13004 CoRR https://arxiv.org/abs/2102.13004 db/journals/corr/corr2102.html#abs-2102-13004

Danula Hettiachchi Mike Schaekermann Tristan McKinney Matthew Lease The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help. 2021 abs/2105.09457 CoRR https://arxiv.org/abs/2105.09457 db/journals/corr/corr2105.html#abs-2105-09457

Ruijiang Gao Maytal Saar-Tsechansky Maria De-Arteaga Ligong Han Min Kyung Lee Matthew Lease Human-AI Collaboration with Bandit Feedback. 2021 abs/2105.10614 CoRR https://arxiv.org/abs/2105.10614 db/journals/corr/corr2105.html#abs-2105-10614

Md. Mustafizur Rahman Dinesh Balakrishnan Dhiraj Murthy Mücahid Kutlu Matthew Lease An Information Retrieval Approach to Building Datasets for Hate Speech Detection. 2021 abs/2106.09775 CoRR https://arxiv.org/abs/2106.09775 db/journals/corr/corr2106.html#abs-2106-09775

Prakhar Singh Anubrata Das 0001 Junyi Jessy Li Matthew Lease The Case for Claim Difficulty Assessment in Automatic Fact Checking. 2021 abs/2109.09689 CoRR https://arxiv.org/abs/2109.09689 db/journals/corr/corr2109.html#abs-2109-09689

Soumyajit Gupta Gurpreet Singh Matthew Lease Scalable Uni-directional Pareto Optimality for Multi-Task Learning with Constraints. 2021 abs/2110.15442 CoRR https://arxiv.org/abs/2110.15442 db/journals/corr/corr2110.html#abs-2110-15442

Lora Aroyo Matthew Lease Praveen K. Paritosh Mike Schaekermann Data Excellence for AI: Why Should You Care. 2021 abs/2111.10391 CoRR https://arxiv.org/abs/2111.10391 db/journals/corr/corr2111.html#abs-2111-10391

Danula Hettiachchi Mark Sanderson Jorge Gonçalves 0001 Simo Hosio Gabriella Kazai Matthew Lease Mike Schaekermann Emine Yilmaz Proceedings of the CSCW 2021 Workshop - Investigating and Mitigating Biases in Crowdsourced Data. 2021 abs/2111.14322 CoRR https://arxiv.org/abs/2111.14322 db/journals/corr/corr2111.html#abs-2111-14322

Vivek Krishna Pradhan Mike Schaekermann Matthew Lease In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers. 2021 abs/2112.02255 CoRR https://arxiv.org/abs/2112.02255 db/journals/corr/corr2112.html#abs-2112-02255

Mücahid Kutlu Tyler McDonnell Tamer Elsayed Matthew Lease Annotator Rationales for Labeling Tasks in Crowdsourcing. 143-189 2020 69 J. Artif. Intell. Res. https://doi.org/10.1613/jair.1.12012 https://www.wikidata.org/entity/Q114367344 db/journals/jair/jair69.html#KutluMEL20

Anubrata Das 0001 Brandon Dang Matthew Lease Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content. 33-42 2020 HCOMP https://doi.org/10.1609/hcomp.v8i1.7461 conf/hcomp/2020 db/conf/hcomp/hcomp2020.html#0001DL20 Md. Mustafizur Rahman Mücahid Kutlu Tamer Elsayed Matthew Lease Efficient Test Collection Construction via Active Learning. 177-184 2020 ICTIR https://doi.org/10.1145/3409256.3409837 conf/ictir/2020 db/conf/ictir/ictir2020.html#RahmanKEL20 Alexander Braylan Matthew Lease Modeling and Aggregation of Complex Annotations via Annotation Distances. 1807-1818 2020 WWW https://doi.org/10.1145/3366423.3380250 conf/www/2020 db/conf/www/www2020.html#BraylanL20

Gurpreet Singh Soumyajit Gupta Matt Lease Clint N. Dawson TIME: A Transparent, Interpretable, Model-Adaptive and Explainable Neural Network for Dynamic Physical Processes. 2020 abs/2003.02426 CoRR https://arxiv.org/abs/2003.02426 db/journals/corr/corr2003.html#abs-2003-02426

Gurpreet Singh Soumyajit Gupta Matthew Lease Extracting Optimal Solution Manifolds using Constrained Neural Optimization. 2020 abs/2009.06024 CoRR https://arxiv.org/abs/2009.06024 db/journals/corr/corr2009.html#abs-2009-06024

Gurpreet Singh Soumyajit Gupta Matthew Lease Clint Dawson 0001 Streaming Singular Value Decomposition for Big Data Applications. 2020 abs/2010.14226 CoRR https://arxiv.org/abs/2010.14226 db/journals/corr/corr2010.html#abs-2010-14226

Prateek Chaudhry Matthew Lease You Are What You Tweet: Profiling Users by Past Tweets to Improve Hate Speech Detection. 2020 abs/2012.09090 CoRR https://arxiv.org/abs/2012.09090 db/journals/corr/corr2012.html#abs-2012-09090

Md. Mustafizur Rahman Mücahid Kutlu Matthew Lease Understanding and Predicting the Characteristics of Test Collections. 2020 abs/2012.13292 CoRR https://arxiv.org/abs/2012.13292 db/journals/corr/corr2012.html#abs-2012-13292

Alexandra Olteanu Jean Garcia-Gathright Maarten de Rijke Michael D. Ekstrand Adam Roegiest Aldo Lipani Alex Beutel Ana Lucic Ana-Andreea Stoica Anubrata Das 0001 Asia Biega Bart Voorn Claudia Hauff Damiano Spina David D. Lewis Douglas W. Oard Emine Yilmaz Faegheh Hasibi Gabriella Kazai Graham McDonald Hinda Haned Iadh Ounis Ilse van der Linden Joris Baan Kamuela N. Lau Krisztian Balog Mahmoud F. Sayed Maria Panteli Mark Sanderson Matthew Lease Preethi Lahoti Toshihiro Kamishima FACTS-IR: fairness, accountability, confidentiality, transparency, and safety in information retrieval. 20-43 2019 53 SIGIR Forum 2 https://doi.org/10.1145/3458553.3458556 db/journals/sigir/sigir53.html#OlteanuGRERLBOL19

Soumyajit Gupta Mücahid Kutlu Vivek Khetan Matthew Lease Correlation, Prediction and Ranking of Evaluation Metrics in Information Retrieval. 636-651 2019 ECIR (1) https://doi.org/10.1007/978-3-030-15712-8_41 conf/ecir/2019-1 db/conf/ecir/ecir2019-1.html#GuptaKKL19 An T. Nguyen 0001 Matthew Lease Byron C. Wallace Explainable modeling of annotations in crowdsourcing. 575-579 2019 IUI https://doi.org/10.1145/3301275.3302276 conf/iui/2019 db/conf/iui/iui2019.html#NguyenLW19 An Thanh Nguyen 0001 Matt Lease Byron C. Wallace Mash: software tools for developing interactive and transparent machine learning systems. 2019 IUI Workshops https://ceur-ws.org/Vol-2327/IUI19WS-ExSS2019-5.pdf conf/iui/2019w db/conf/iui/iui2019w.html#NguyenLW19a Md. Mustafizur Rahman Mücahid Kutlu Matthew Lease Constructing Test Collections using Multi-armed Bandits and Active Learning. 3158-3164 2019 WWW https://doi.org/10.1145/3308558.3313675 conf/www/2019 db/conf/www/www2019.html#RahmanKL19

Anubrata Das 0001 Kunjan Mehta Matthew Lease CobWeb: A Research Prototype for Exploring User Bias in Political Fact-Checking. 2019 abs/1907.03718 CoRR http://arxiv.org/abs/1907.03718 db/journals/corr/corr1907.html#abs-1907-03718

Anubrata Das 0001 Matthew Lease A Conceptual Framework for Evaluating Fairness in Search. 2019 abs/1907.09328 CoRR http://arxiv.org/abs/1907.09328 db/journals/corr/corr1907.html#abs-1907-09328

Lora Aroyo Anca Dumitrache Jeffrey V. Nickerson Matthew Lease Pietro Michelucci Reports of the Workshops Held at the Sixth AAAI Conference on Human Computation and Crowdsourcing. 2018 39 AI Mag. 4 https://doi.org/10.1609/aimag.v39i4.2834 https://www.wikidata.org/entity/Q63844637 db/journals/aim/aim39.html#AroyoDNLM18 57-63

Mücahid Kutlu Tamer Elsayed Matthew Lease Intelligent topic selection for low-cost information retrieval evaluation: A New perspective on deep vs. shallow judging. 37-59 2018 54 Inf. Process. Manag. 1 https://doi.org/10.1016/j.ipm.2017.09.002 db/journals/ipm/ipm54.html#KutluEL18

Kezban Dilek Onal Ye Zhang Ismail Sengor Altingovde Md. Mustafizur Rahman Pinar Karagoz Alex Braylan Brandon Dang Heng-Lu Chang Henna Kim Quinten McNamara Aaron Angert Edward Banner Vivek Khetan Tyler McDonnell An Thanh Nguyen 0001 Dan Xu Byron C. Wallace Maarten de Rijke Matthew Lease Neural information retrieval: at the end of the early years. 111-182 2018 21 Inf. Retr. J. 2-3 https://doi.org/10.1007/s10791-017-9321-y https://www.wikidata.org/entity/Q59615721 db/journals/ir/ir21.html#OnalZARKBDCKMAB18

An T. Nguyen 0001 Aditya Kharosekar Matthew Lease Byron C. Wallace An Interpretable Joint Graphical Model for Fact-Checking From Crowds. 2018 AAAI https://doi.org/10.1609/aaai.v32i1.11487 conf/aaai/2018 db/conf/aaai/aaai2018.html#NguyenKLW18 1511-1518 Mücahid Kutlu Tamer Elsayed Maram Hasanain Matthew Lease When Rank Order Isn't Enough: New Statistical-Significance-Aware Correlation Measures. 397-406 2018 CIKM https://doi.org/10.1145/3269206.3271751 conf/cikm/2018 db/conf/cikm/cikm2018.html#KutluEHL18 Mücahid Kutlu Tyler McDonnell Aashish Sheshadri Tamer Elsayed Matthew Lease Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably. 42-46 2018 DESIRES https://ceur-ws.org/Vol-2167/paper10.pdf conf/desires/2018 db/conf/desires/desires2018.html#KutluMSEL18 Matthew Lease Fact Checking and Information Retrieval. 97-98 2018 DESIRES https://ceur-ws.org/Vol-2167/short1.pdf conf/desires/2018 db/conf/desires/desires2018.html#Lease18 Tanya Goyal Tyler McDonnell Mücahid Kutlu Tamer Elsayed Matthew Lease Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations. 41-49 2018 HCOMP https://doi.org/10.1609/hcomp.v6i1.13331 conf/hcomp/2018 db/conf/hcomp/hcomp2018.html#GoyalMKEL18 Mücahid Kutlu Tyler McDonnell Yassmine Barkallah Tamer Elsayed Matthew Lease Crowd vs. Expert: What Can Relevance Judgment Rationales Teach Us About Assessor Disagreement? 805-814 2018 SIGIR https://doi.org/10.1145/3209978.3210033 conf/sigir/2018 db/conf/sigir/sigir2018.html#KutluMBEL18 An T. Nguyen 0001 Aditya Kharosekar Saumyaa Krishnan Siddhesh Krishnan Elizabeth Tate Byron C. Wallace Matthew Lease Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking. 189-199 2018 UIST https://doi.org/10.1145/3242587.3242666 conf/uist/2018 db/conf/uist/uist2018.html#NguyenKKKTWL18 Matthew Lease Omar Alonso Crowdsourcing and Human Computation: Introduction. 2018 Encyclopedia of Social Network Analysis and Mining. 2nd Ed. https://doi.org/10.1007/978-1-4939-7131-2_107 reference/snam/2018 db/reference/snam/snam2018.html#LeaseA18

Md. Mustafizur Rahman Mücahid Kutlu Tamer Elsayed Matthew Lease Efficient Test Collection Construction via Active Learning. 2018 abs/1801.05605 CoRR http://arxiv.org/abs/1801.05605 db/journals/corr/corr1801.html#abs-1801-05605

Mücahid Kutlu Vivek Khetan Matthew Lease Correlation and Prediction of Evaluation Metrics in Information Retrieval. 2018 abs/1802.00323 CoRR http://arxiv.org/abs/1802.00323 db/journals/corr/corr1802.html#abs-1802-00323

Brandon Dang Martin J. Riedl Matthew Lease Toward Safer Crowdsourced Content Moderation. 2018 abs/1804.10999 CoRR http://arxiv.org/abs/1804.10999 db/journals/corr/corr1804.html#abs-1804-10999

Mücahid Kutlu Tyler McDonnell Aashish Sheshadri Tamer Elsayed Matthew Lease Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably. 2018 abs/1806.00755 CoRR http://arxiv.org/abs/1806.00755 db/journals/corr/corr1806.html#abs-1806-00755

Ye Zhang Matthew Lease Byron C. Wallace Active Discriminative Text Representation Learning. 3386-3392 2017 AAAI https://doi.org/10.1609/aaai.v31i1.10962 conf/aaai/2017 db/conf/aaai/aaai2017.html#ZhangLW17 Ye Zhang Matthew Lease Byron C. Wallace Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization. 155-160 2017 ACL (2) https://doi.org/10.18653/v1/P17-2024 conf/acl/2017-2 db/conf/acl/acl2017-2.html#ZhangLW17 An Thanh Nguyen 0001 Byron C. Wallace Junyi Jessy Li Ani Nenkova Matthew Lease Aggregating and Predicting Sequence Labels from Crowd Annotations. 299-309 2017 ACL (1) https://doi.org/10.18653/v1/P17-1028 https://www.wikidata.org/entity/Q44785376 conf/acl/2017-1 db/conf/acl/acl2017-1.html#NguyenWLNL17 Xi Zheng 0001 Akanksha Bansal Matthew Lease Bullseye: structured passage retrieval and document highlighting for scholarly search. 32:1-32:4 2017 ACSW https://doi.org/10.1145/3014812.3014846 conf/acsw/2017 db/conf/acsw/acsw2017.html#ZhengBL17 Tyler McDonnell Mücahid Kutlu Tamer Elsayed Matthew Lease The Many Benefits of Annotator Rationales for Relevance Judgments. 4909-4913 2017 IJCAI https://doi.org/10.24963/ijcai.2017/692 conf/ijcai/2017 db/conf/ijcai/ijcai2017.html#McDonnellKEL17

Mücahid Kutlu Tamer Elsayed Matthew Lease Learning to Effectively Select Topics For Information Retrieval Test Collections. 2017 abs/1701.07810 CoRR http://arxiv.org/abs/1701.07810 db/journals/corr/corr1701.html#KutluEL17

Ye Zhang Matthew Lease Byron C. Wallace Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization. 2017 abs/1702.02535 CoRR http://arxiv.org/abs/1702.02535 db/journals/corr/corr1702.html#ZhangLW17a

Akash Mankar Riddhi J. Shah Matthew Lease Design Activism for Minimum Wage Crowd Work. 2017 abs/1706.10097 CoRR http://arxiv.org/abs/1706.10097 db/journals/corr/corr1706.html#MankarSL17

Tyler McDonnell Matthew Lease Mücahid Kutlu Tamer Elsayed Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. 139-148 2016 HCOMP https://doi.org/10.1609/hcomp.v4i1.13287 conf/hcomp/2016 db/conf/hcomp/hcomp2016.html#McDonnellLKE16 An Thanh Nguyen 0001 Matthew Halpern Byron C. Wallace Matthew Lease Probabilistic Modeling for Crowdsourcing Partially-Subjective Ratings. 149-158 2016 HCOMP https://doi.org/10.1609/hcomp.v4i1.13274 conf/hcomp/2016 db/conf/hcomp/hcomp2016.html#NguyenHWL16 Reem Suwaileh Mücahid Kutlu Nihal Fathima Tamer Elsayed Matthew Lease ArabicWeb16: A New Crawl for Today's Arabic Web. 673-676 2016 SIGIR https://doi.org/10.1145/2911451.2914677 conf/sigir/2016 db/conf/sigir/sigir2016.html#SuwailehKFEL16 An T. Nguyen 0001 Byron C. Wallace Matthew Lease A Correlated Worker Model for Grouped, Imbalanced and Multitask Data. 2016 UAI http://auai.org/uai2016/proceedings/papers/235.pdf conf/uai/2016 db/conf/uai/uai2016.html#NguyenWL16 Arpita Ghosh Matthew Lease Proceedings of the Fourth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2016, 30 October - 3 November, 2016, Austin, Texas, USA. HCOMP 2016 AAAI Press 978-1-57735-774-2 https://ojs.aaai.org/index.php/HCOMP/issue/view/312 db/conf/hcomp/hcomp2016.html

Brandon Dang Miles Hutson Matthew Lease MmmTurkey: A Crowdsourcing Framework for Deploying Tasks and Recording Worker Behavior on Amazon Mechanical Turk. 2016 abs/1609.00945 CoRR http://arxiv.org/abs/1609.00945 db/journals/corr/corr1609.html#DangHL16

Yalin Sun Pengxiang Cheng 0001 Shengwei Wang Hao Lyu Matthew Lease Iain James Marshall Byron C. Wallace Crowdsourcing Information Extraction for Biomedical Systematic Reviews. 2016 abs/1609.01017 CoRR http://arxiv.org/abs/1609.01017 db/journals/corr/corr1609.html#SunCWLLMW16

Ye Zhang Md. Mustafizur Rahman Alex Braylan Brandon Dang Heng-Lu Chang Henna Kim Quinten McNamara Aaron Angert Edward Banner Vivek Khetan Tyler McDonnell An Thanh Nguyen 0001 Dan Xu Byron C. Wallace Matthew Lease Neural Information Retrieval: A Literature Review. 2016 abs/1611.06792 CoRR http://arxiv.org/abs/1611.06792 db/journals/corr/corr1611.html#ZhangRBDCKMABKM16

Hyun Joon Jung Matthew Lease A Discriminative Approach to Predicting Assessor Accuracy. 159-171 2015 ECIR https://doi.org/10.1007/978-3-319-16354-3_17 conf/ecir/2015 db/conf/ecir/ecir2015.html#JungL15 Hyun Joon Jung Matthew Lease Modeling Temporal Crowd Work Quality with Limited Supervision. 83-91 2015 HCOMP https://doi.org/10.1609/hcomp.v3i1.13227 conf/hcomp/2015 db/conf/hcomp/hcomp2015.html#JungL15 An Thanh Nguyen 0001 Byron C. Wallace Matthew Lease Combining Crowd and Expert Labels Using Decision Theoretic Active Learning. 120-129 2015 HCOMP https://doi.org/10.1609/hcomp.v3i1.13225 conf/hcomp/2015 db/conf/hcomp/hcomp2015.html#NguyenWL15

Tatiana Josephy Matt Lease Praveen K. Paritosh Markus Krause Mihai Georgescu Michael Tjalve Daniela Braga Workshops Held at the First AAAI Conference on Human Computation and Crowdsourcing: A Report. 75-78 2014 35 AI Mag. 2 https://doi.org/10.1609/aimag.v35i2.2537 db/journals/aim/aim35.html#JosephyLPKGTB14

Hyun Joon Jung Yubin Park Matthew Lease Predicting Next Label Quality: A Time-Series Model of Crowdwork. 2014 HCOMP https://doi.org/10.1609/hcomp.v2i1.13165 conf/hcomp/2014 db/conf/hcomp/hcomp2014.html#JungPL14 87-95 Yinglong Zhang Jin Zhang Matthew Lease Jacek Gwizdka Multidimensional relevance modeling via psychometrics and crowdsourcing. 435-444 2014 SIGIR https://doi.org/10.1145/2600428.2609577 conf/sigir/2014 db/conf/sigir/sigir2014.html#ZhangZLG14a Matthew Lease Omar Alonso Crowdsourcing and Human Computation, Introduction. 304-315 2014 Encyclopedia of Social Network Analysis and Mining https://doi.org/10.1007/978-1-4614-6170-8_107 db/reference/snam/snam2014.html#LeaseA14

Ethan Petuchowski Matthew Lease TurKPF: TurKontrol as a Particle Filter. 2014 abs/1404.5078 CoRR http://arxiv.org/abs/1404.5078 db/journals/corr/corr1404.html#PetuchowskiL14

Xi Zheng 0001 Akanksha Bansal Matthew Lease Bullseye: Structured Passage Retrieval and Document Highlighting for Scholarly Search. 2014 abs/1406.1875 CoRR http://arxiv.org/abs/1406.1875 db/journals/corr/corr1406.html#ZhengBL14

Matthew Lease Emine Yilmaz Crowdsourcing for information retrieval: introduction to the special issue. 91-100 2013 16 Inf. Retr. 2 https://doi.org/10.1007/s10791-013-9222-7 https://www.wikidata.org/entity/Q60143181 db/journals/ir/ir16.html#LeaseY13

Aniket Kittur Jeffrey V. Nickerson Michael S. Bernstein Elizabeth Gerber Aaron D. Shaw John Zimmerman Matt Lease John J. Horton The future of crowd work. 1301-1318 2013 CSCW https://doi.org/10.1145/2441776.2441923 https://www.wikidata.org/entity/Q124248108 conf/cscw/2013 db/conf/cscw/cscw2013.html#KitturNBGSZLH13 Aashish Sheshadri Matthew Lease SQUARE: A Benchmark for Research on Computing Crowd Consensus. 2013 HCOMP https://doi.org/10.1609/hcomp.v1i1.13088 conf/hcomp/2013 db/conf/hcomp/hcomp2013.html#SheshadriL13 156-164 Ripon K. Saha Matthew Lease Sarfraz Khurshid Dewayne E. Perry Improving bug localization using structured information retrieval. 345-355 2013 ASE https://doi.org/10.1109/ASE.2013.6693093 https://www.wikidata.org/entity/Q60143202 conf/kbse/2013 db/conf/kbse/ase2013.html#SahaLKP13 Aashish Sheshadri Matthew Lease SQUARE: Benchmarking Crowd Consensus at MediaEval. 2013 MediaEval conf/mediaeval/2013 db/conf/mediaeval/mediaeval2013.html#SheshadriL13 https://ceur-ws.org/Vol-1043/mediaeval2013_submission_62.pdf Mark D. Smucker Gabriella Kazai Matthew Lease Overview of the TREC 2013 Crowdsourcing Track. 2013 TREC http://trec.nist.gov/pubs/trec22/papers/CROWD.OVERVIEW.pdf conf/trec/2013 db/conf/trec/trec2013.html#SmuckerKRL13

Donna Vakharia Matthew Lease Beyond AMT: An Analysis of Crowd Work Platforms. 2013 CoRR http://arxiv.org/abs/1310.1672 abs/1310.1672 db/journals/corr/corr1310.html#VakhariaL13

Hyun Joon Jung Matthew Lease Crowdsourced Task Routing via Matrix Factorization. 2013 CoRR http://arxiv.org/abs/1310.5142 abs/1310.5142 db/journals/corr/corr1310.html#JungL13a

Di Liu Randolph G. Bias Matthew Lease Rebecca Kuipers Crowdsourcing for usability testing. 1-10 2012 ASIST https://doi.org/10.1002/meet.14504901100 https://www.wikidata.org/entity/Q114389927 conf/asist/2012 db/conf/asist/asist2012.html#LiuBLK12 Hyun Joon Jung Matthew Lease Improving Quality of Crowdsourced Labels via Probabilistic Matrix Factorization. 2012 HCOMP@AAAI http://www.aaai.org/ocs/index.php/WS/AAAIW12/paper/view/5258 conf/hcomp/2012 db/conf/hcomp/hcomp2012.html#JungL12 Hohyon Ryu Matthew Lease Nicholas Woodward Finding and exploring memes in social media. 295-304 2012 HT https://doi.org/10.1145/2309996.2310044 https://www.wikidata.org/entity/Q60143186 conf/ht/2012 db/conf/ht/ht2012.html#RyuLW12 Hyun Joon Jung Matthew Lease Inferring missing relevance judgments from crowd workers via probabilistic matrix factorization. 1095-1096 2012 SIGIR https://doi.org/10.1145/2348283.2348486 conf/sigir/2012 db/conf/sigir/sigir2012.html#JungL12 Shilpa Shukla Matthew Lease Ambuj Tewari Parallelizing ListNet training using spark. 1127-1128 2012 SIGIR https://doi.org/10.1145/2348283.2348502 https://www.wikidata.org/entity/Q60143187 conf/sigir/2012 db/conf/sigir/sigir2012.html#ShuklaLT12 Matthew Lease Omar Alonso Crowdsourcing for search evaluation and social-algorithmic search. 1180 2012 SIGIR https://doi.org/10.1145/2348283.2348530 https://www.wikidata.org/entity/Q60143183 conf/sigir/2012 db/conf/sigir/sigir2012.html#LeaseA12 Mark D. Smucker Gabriella Kazai Matthew Lease Overview of the TREC 2012 Crowdsourcing Track. 2012 TREC http://trec.nist.gov/pubs/trec21/papers/CROWD12.overview.pdf conf/trec/2012 db/conf/trec/trec2012.html#SmuckerKL12

Di Liu Matthew Lease Rebecca Kuipers Randolph G. Bias Crowdsourcing for Usability Testing http://arxiv.org/abs/1203.1468 2012 CoRR abs/1203.1468 db/journals/corr/corr1203.html#abs-1203-1468

Abhimanu Kumar Jason Baldridge Matthew Lease Joydeep Ghosh Dating Texts without Explicit Temporal Cues http://arxiv.org/abs/1211.2290 2012 CoRR abs/1211.2290 db/journals/corr/corr1211.html#abs-1211-2290

Hyun Joon Jung Matthew Lease Evaluating Classifiers Without Expert Labels http://arxiv.org/abs/1212.0960 2012 CoRR abs/1212.0960 db/journals/corr/corr1212.html#abs-1212-0960

Matthew Lease Vitor R. Carvalho Emine Yilmaz Crowdsourcing for search and data mining. 18-24 2011 45 SIGIR Forum 1 https://doi.org/10.1145/1988852.1988856 db/journals/sigir/sigir45.html#LeaseCY11

Matthew Lease Emine Yilmaz Crowdsourcing for information retrieval. 66-75 2011 45 SIGIR Forum 2 https://doi.org/10.1145/2093346.2093356 https://www.wikidata.org/entity/Q60143182 db/journals/sigir/sigir45.html#LeaseY11

Hyun Joon Jung Matthew Lease Improving Consensus Accuracy via Z-Score and Weighted Voting. 2011 Human Computation http://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/view/3905 conf/aaai/2011hc db/conf/aaai/hc2011.html#JungL11 Matthew Lease On Quality Control and Machine Learning in Crowdsourcing. 2011 Human Computation http://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/view/3906 conf/aaai/2011hc db/conf/aaai/hc2011.html#Lease11 Jörn Klinger Matthew Lease Enabling trust in crowd labor relations through identity sharing. 1-4 2011 ASIST https://doi.org/10.1002/meet.2011.14504801257 https://www.wikidata.org/entity/Q114389921 conf/asist/2011 db/conf/asist/asist2011.html#KlingerL11 Hohyon Ryu Matthew Lease Crowdworker filtering with support vector machine. 1-4 2011 ASIST https://doi.org/10.1002/meet.2011.14504801245 https://www.wikidata.org/entity/Q60143184 conf/asist/2011 db/conf/asist/asist2011.html#RyuL11 Stephen M. Wolfson Matthew Lease Look before you leap: Legal pitfalls of crowdsourcing. 1-10 2011 ASIST https://doi.org/10.1002/meet.2011.14504801135 https://www.wikidata.org/entity/Q114389922 conf/asist/2011 db/conf/asist/asist2011.html#WolfsonL11 Abhimanu Kumar Matthew Lease Jason Baldridge Supervised language modeling for temporal resolution of texts. 2069-2072 2011 CIKM https://doi.org/10.1145/2063576.2063892 https://www.wikidata.org/entity/Q60143195 conf/cikm/2011 db/conf/cikm/cikm2011.html#KumarLB11 Yongyi Zhou Ramona Broussard Matthew Lease Mobile options for online public access catalogs. 598-605 2011 iConference https://doi.org/10.1145/1940761.1940842 https://www.wikidata.org/entity/Q60143194 conf/iconference/2011 db/conf/iconference/iconference2011.html#ZhouBL11 Aibo Tian Matthew Lease Active learning to maximize accuracy vs. effort in interactive information retrieval. 145-154 2011 SIGIR https://doi.org/10.1145/2009916.2009939 https://www.wikidata.org/entity/Q60143188 conf/sigir/2011 db/conf/sigir/sigir2011.html#TianL11 Abhimanu Kumar Matthew Lease Learning to rank from a noisy crowd. 1221-1222 2011 SIGIR https://doi.org/10.1145/2009916.2010129 https://www.wikidata.org/entity/Q60143193 conf/sigir/2011 db/conf/sigir/sigir2011.html#KumarL11 Omar Alonso Matthew Lease Crowdsourcing for information retrieval: principles, methods, and applications. 1299-1300 2011 SIGIR https://doi.org/10.1145/2009916.2010170 https://www.wikidata.org/entity/Q60143190 conf/sigir/2011 db/conf/sigir/sigir2011.html#AlonsoL11 Omar Alonso Matthew Lease Crowdsourcing 101: putting the WSDM of crowds to work for you. 1-2 2011 WSDM https://doi.org/10.1145/1935826.1935831 https://www.wikidata.org/entity/Q60143189 conf/wsdm/2011 db/conf/wsdm/wsdm2011.html#AlonsoL11 Vitor R. Carvalho Matthew Lease Emine Yilmaz Crowdsourcing for search and data mining. 5-6 2011 WSDM https://doi.org/10.1145/1935826.1935828 https://www.wikidata.org/entity/Q60143192 conf/wsdm/2011 db/conf/wsdm/wsdm2011.html#CarvalhoLY11 Matthew Lease Beyond keywords: finding information more accurately and easily using natural language. Brown University, USA 2010 https://cs.brown.edu/research/pubs/theses/phd/2010/lease.pdf

Vitor R. Carvalho Matthew Lease Emine Yilmaz Crowdsourcing for search evaluation. 17-22 2010 44 SIGIR Forum 2 https://doi.org/10.1145/1924475.1924481 db/journals/sigir/sigir44.html#CarvalhoLY10

Ramona Broussard Yongyi Zhou Matthew Lease Mobile phone search for library catalogs. 1-4 2010 ASIST https://doi.org/10.1002/meet.14504701128 conf/asist/2010 db/conf/asist/asist2010.html#BroussardZL10 Ramona Broussard Yongyi Zhou Matthew Lease University of Texas mobile library search. 1-2 2010 ASIST https://doi.org/10.1002/meet.14504701385 conf/asist/2010 db/conf/asist/asist2010.html#BroussardZL10a Saeedeh Momtazi Matthew Lease Dietrich Klakow Effective Term Weighting for Sentence Retrieval. 482-485 2010 ECDL https://doi.org/10.1007/978-3-642-15464-5_62 conf/ercimdl/2010 db/conf/ercimdl/ecdl2010.html#MomtaziLK10 Catherine Grady Matthew Lease Crowdsourcing Document Relevance Assessment with Mechanical Turk. 172-179 2010 Mturk@HLT-NAACL https://aclanthology.org/W10-0727/ conf/naacl/2010mturk db/conf/naacl/mturk2010.html#GradyL10 Matthew Lease James Allan 0001 W. Bruce Croft Regression Rank: Learning to Meet the Opportunity of Descriptive Queries. 90-101 2009 ECIR https://doi.org/10.1007/978-3-642-00958-7_11 conf/ecir/2009 db/conf/ecir/ecir2009.html#LeaseAC09 Matthew Lease An improved markov random field model for supporting verbose queries. 476-483 2009 SIGIR https://doi.org/10.1145/1571941.1572023 https://www.wikidata.org/entity/Q60143197 conf/sigir/2009 db/conf/sigir/sigir2009.html#Lease09 Marc-Allen Cartright Jangwon Seo Matthew Lease UMass Amherst and UT Austin @ the TREC 2009 Relevance Feedback Track. 2009 TREC http://trec.nist.gov/pubs/trec18/papers/umass-amhearst.RF.pdf conf/trec/2009 db/conf/trec/trec2009.html#CartrightSL09 Matthew Lease Incorporating Relevance and Pseudo-relevance Feedback in the Markov Random Field Model. 2008 TREC http://trec.nist.gov/pubs/trec17/papers/brownu.rf.rev.pdf conf/trec/2008 db/conf/trec/trec2008.html#Lease08 Matthew Lease Natural language processing for information retrieval: the time is ripe (again). 1-8 2007 conf/cikm/2007pikm PIKM https://doi.org/10.1145/1316874.1316876 https://www.wikidata.org/entity/Q60143198 db/conf/cikm/pikm2007.html#Lease07 Matthew Lease Eugene Charniak A Dirichlet-Smoothed Bigram Model for Retrieving Spontaneous Speech. 687-694 2007 CLEF https://doi.org/10.1007/978-3-540-85760-0_87 https://www.wikidata.org/entity/Q60143196 conf/clef/2007 db/conf/clef/clef2007.html#LeaseC07 Matthew Lease Eugene Charniak Brown at CL-SR'07: Retrieving Conversational Speech in English and Czech. 2007 CLEF (Working Notes) https://ceur-ws.org/Vol-1173/CLEF2007wn-CLSR-LeaseEt2007.pdf conf/clef/2007w db/conf/clef/clef2007w.html#LeaseC07a

Matthew Lease Mark Johnson 0001 Eugene Charniak Recognizing disfluencies in conversational speech. 1566-1573 2006 14 IEEE Trans. Speech Audio Process. 5 https://doi.org/10.1109/TASL.2006.878269 https://www.wikidata.org/entity/Q57262085 db/journals/taslp/taslp14.html#LeaseJC06

Matthew Lease Eugene Charniak Mark Johnson 0001 David McClosky A Look at Parsing and Its Applications. 2006 conf/aaai/2006 AAAI db/conf/aaai/aaai2006.html#LeaseCJM06 http://www.aaai.org/Library/AAAI/2006/aaai06-270.php 1642-1645 John Hale Izhak Shafran Lisa Yung Bonnie J. Dorr Mary P. Harper Anna Krasnyanskaya Matthew Lease Yang Liu 0004 Brian Roark Matthew G. Snover Robin Stewart PCFGs with Syntactic and Prosodic Indicators of Speech Repairs. 2006 conf/acl/2006 ACL https://aclanthology.org/P06-1021/ https://doi.org/10.3115/1220175.1220196 https://www.wikidata.org/entity/Q60143200 db/conf/acl/acl2006.html#HaleSYDHKLLRSS06 Brian Roark Yang Liu 0004 Mary P. Harper Robin Stewart Matthew Lease Matthew G. Snover Izhak Shafran Bonnie J. Dorr John Hale Anna Krasnyanskaya Lisa Yung Reranking for Sentence Boundary Detection in Conversational Speech. 545-548 2006 ICASSP (1) https://doi.org/10.1109/ICASSP.2006.1660078 conf/icassp/2006 db/conf/icassp/icassp2006.html#RoarkLHSLSSDHKY06 Brian Roark Mary P. Harper Eugene Charniak Bonnie J. Dorr Mark Johnson 0001 Jeremy G. Kahn Yang Liu 0004 Mari Ostendorf John Hale Anna Krasnyanskaya Matthew Lease Izhak Shafran Matthew G. Snover Robin Stewart Lisa Yung SParseval: Evaluation Metrics for Parsing Speech. 333-338 2006 LREC http://www.lrec-conf.org/proceedings/lrec2006/summaries/116.html conf/lrec/2006 db/conf/lrec/lrec2006.html#RoarkHCDJK0OHKL06 Ann Bies Stephanie M. Strassel Haejoong Lee Kazuaki Maeda Seth Kulick Yang Liu 0004 Mary P. Harper Matthew Lease Linguistic Resources for Speech Parsing. 629-634 2006 LREC http://www.lrec-conf.org/proceedings/lrec2006/summaries/755.html conf/lrec/2006 db/conf/lrec/lrec2006.html#BiesSLMK0HL06 Matthew Lease Mark Johnson 0001 Early Deletion of Fillers In Processing Conversational Speech. 2006 conf/naacl/2006 HLT-NAACL https://aclanthology.org/N06-2019/ db/conf/naacl/naacl2006.html#LeaseJ06 Matthew Lease Eugene Charniak Mark Johnson 0001 Parsing and its applications for conversational speech. 961-964 2005 ICASSP (5) https://doi.org/10.1109/ICASSP.2005.1416465 https://www.wikidata.org/entity/Q57262094 conf/icassp/2005 db/conf/icassp/icassp2005.html#LeaseCJ05 Matthew Lease Eugene Charniak Parsing Biomedical Literature. 58-69 2005 conf/ijcnlp/2005 IJCNLP https://doi.org/10.1007/11562214_6 db/conf/ijcnlp/ijcnlp2005.html#LeaseC05 Jeremy G. Kahn Matthew Lease Eugene Charniak Mark Johnson 0001 Mari Ostendorf Effective Use of Prosody in Parsing Conversational Speech. 2005 conf/naacl/2005 HLT/EMNLP https://aclanthology.org/H05-1030/ db/conf/naacl/naacl2005.html#KahnLCJO05 233-240 Matthew Lease Guy Eddon SmartElevator: Revitalizing A Legacy Device through Inexpensive Augmentation. 254-259 2003 conf/icdcsw/2003 ICDCS Workshops https://doi.org/10.1109/ICDCSW.2003.1203563 https://doi.ieeecomputersociety.org/10.1109/ICDCSW.2003.1203563 db/conf/icdcsw/icdcsw2003.html#LeaseE03 Anthony LaMarca Waylon Brunette David Koizumi Matthew Lease Stefan B. Sigurdsson Kevin Sikorski Dieter Fox Gaetano Borriello PlantCare: An Investigation in Practical Ubiquitous Systems. 316-332 2002 conf/huc/2002 UbiComp https://doi.org/10.1007/3-540-45809-3_25 db/conf/huc/ubicomp2002.html#LaMarcaBKLSSFB02 Anthony LaMarca Waylon Brunette David Koizumi Matthew Lease Stefan B. Sigurdsson Kevin Sikorski Dieter Fox Gaetano Borriello Making Sensor Networks Practical with Robots. 152-166 2002 conf/pervasive/2002 Pervasive https://doi.org/10.1007/3-540-45866-2_13 https://www.wikidata.org/entity/Q60143201 db/conf/pervasive/pervasive2002.html#LaMarcaBKLSSFB02 Ira J. Kalet Jonn Wu Matthew Lease Mary M. Austin-Seymour James F. Brinkley Cornelius Rosse Anatomical information in radiation treatment planning. 1999 AMIA https://knowledge.amia.org/amia-55142-a1999a-1.591078/t-001-1.593353/f-001-1.593354/a-058-1.593793/a-059-1.593790 conf/amia/1999 db/conf/amia/amia1999.html#KaletWLABR99 Girmaw AbebeGirmaw Abebe Tadesse Ahmed Alaa 0001Ahmed M. Alaa Rainier Aliment James Allan 0001 Omar Alonso Ismail Sengör AltingövdeIsmail Sengor Altingovde Aaron Angert Lora Aroyo Mary M. Austin-Seymour Joris Baan Min Bai Dinesh Balakrishnan Jason Baldridge Krisztian Balog Edward Banner Akanksha Bansal Yassmine Barkallah Max Bartolo Lilith Bat-Leah Michael S. Bernstein Alex Beutel Timir J. Bharucha Nilavra Bhattacharya Randolph G. Bias Asia J. BiegaAsia Biega Ann Bies Kurt Bollacker Raghu Bollapragada Alexander Boltz Gaetano Borriello Alessandro Bozzon Daniela Braga Danilo Brajovic Alexander BraylanAlex Braylan James F. Brinkley Ramona Broussard Waylon Brunette Jonathan Buck Marc-Allen Cartright Vitor R. Carvalho L. Elisa Celis Heng-Lu Chang Eugene Charniak Trina Chatterjee Prateek Chaudhry Jifan Chen Pengxiang Cheng 0001 Subramanian Chidambaram Eunsol Choi Yujin Choi Gabriella Chronis Cody Coleman W. Bruce Croft Brandon Dang Anubrata Das 0001 Clint Dawson 0001Clint N. Dawson Maria De-Arteaga Gianluca Demartini Adji B. DiengAdji Bousso Dieng Bonnie J. Dorr Rotem Dror Caifan Du Anca Dumitrache Debojyoti Dutta Guy Eddon Michael D. Ekstrand Tamer Elsayed Katrin Erk Nihal Fathima Sina Fazelpour Dieter Fox Ujwal Gadiraju Ruijiang Gao Jean Garcia-Gathright Sanjana Gautam Mihai Georgescu Elizabeth Gerber Arpita Ghosh Joydeep Ghosh Jorge Gonçalves 0001 Venkata Subrahmanyan Govindarajan Tanya Goyal Catherine Grady Chitrank Gupta Soumyajit Gupta Nezihe Merve Gürel Isabelle Guyon Jacek Gwizdka Patrick Haffner John Hale Matthew Halpern Ligong Han Braden Hancock Hinda Haned Mary P. Harper Maram Hasanain Faegheh Hasibi Claudia Hauff Danula Hettiachchi Ryan Hileman Indigo Holcombe-James John Joseph HortonJohn J. Horton Simo Hosio Chi-Yang Hsu Miles Hutson Gabriel Jaffe Mark Johnson 0001 Tatiana Josephy Hyun Joon Jung Jeremy G. Kahn Ira J. Kalet Koushik Kalyanaraman Gauri Kambhatla Toshihiro Kamishima Pinar KaragözPinar Karagoz Bojan Karlas Gabriella Kazai Krishnaram Kenthapadi Vijay Keswani Aditya Kharosekar Vivek Khetan Sarfraz Khurshid Henna Kim Aniket Kittur Dietrich Klakow Jörn Klinger Bernard Koch David Koizumi Venelin Kovatchev Anna Krasnyanskaya Markus Krause Saumyaa Krishnan Siddhesh Krishnan Rebecca Kuipers Seth Kulick Abhimanu Kumar Tzu-Sheng Kuo Mucahid KutluMücahid Kutlu Preethi Lahoti Anthony LaMarca Kamuela N. Lau Haejoong Lee Min Kyung Lee Sooyong Lee David D. Lewis Junyi Jessy Li Li Erran Li Ilse van der Linden Aldo Lipani Alex Liu Di Liu Houjiang Liu Yang Liu 0004 Yang Liu 0018 Stephanie Livingstone Ana Lucic Hao Lyu Kazuaki Maeda Michael W. Mahoney Kyle Mahowald Akash Mankar Madalyn Marabella Iain James Marshall Manil Maskey Peter Mattson David McClosky Graham McDonald Tyler McDonnell Tristan McKinneyTristan J. McKinney Quinten McNamara Kunjan Mehta Pietro Michelucci Stefano Mizzaro Saeedeh Momtazi Utkarsh Mujumdar Dhiraj Murthy Ani Nenkova Terrence Neumann An T. Nguyen 0001An Thanh Nguyen 0001 Jeffrey V. Nickerson Curtis G. Northcutt Natasha F. Noy Luis Oala Douglas W. Oard Alexandra Olteanu Kezban Dilek Onal Mari Ostendorf Iadh Ounis Maria Panteli Praveen K. Paritosh Yubin Park Alicia Parrish Dewayne E. Perry Ethan Petuchowski Daisy Pinaroc Vivek Krishna Pradhan Md. Mustafizur Rahman Ashwin Rajadesingan Amy Rechkemmer Vijay Janapa Reddi Martin J. Riedl Soo Young Rieh Maarten de Rijke Megan RisdalMeg Risdal Brian Roark Adam Roegiest Kevin Roitero William Gaviria Rojas Cornelius Rosse Hohyon Ryu Maytal Saar-Tsechansky Ripon K. Saha Flora D. Salim Wojciech Samek Mark Sanderson Mahmoud F. Sayed Mihaela van der Schaar Mike Schaekermann Falk Scholer Jangwon Seo Izhak Shafran Chirag Shah 0001 Riddhi J. Shah Aaron D. Shaw Aashish Sheshadri Li Shi Shilpa Shukla Stefan B. Sigurdsson Kevin Sikorski Anjalee de Silva Gurpreet Singh Prakhar Singh Mark D. Smucker Matthew G. Snover Damiano Spina Ravi Srinivasan Miriah Steiger Robin Stewart Ana-Andreea Stoica Stephanie M. Strassel Yiheng Su Wei Sun 0031 Yalin Sun Reem Suwaileh Elizabeth Tate Ambuj Tewari Paul Thomas 0001 Aibo Tian Michael Tjalve Mehmet Deniz Türkmen Donna Vakharia Joaquin Vanschoren Sukrit Venkatagiri Satyugjit Virk Bart Voorn Arjen P. de Vries Byron C. Wallace Shengwei Wang Alex C. Williams Stephen M. Wolfson Yian Wong Nicholas Joel WoodwardNicholas Woodward Jonn Wu Yating Wu 0002 Dan Xu Jie Yang 0028 Xiaozhe Yao Emine Yilmaz Lisa Yung Angela Zhang Ce Zhang 0001 Dan Zhang 0004 Jin Zhang Ye Zhang Yinglong Zhang James Xi ZhengXi Zheng 0001 Didi Zhou Xiong Zhou Yongyi Zhou John Zimmerman James Zou 0001 Guido Zuccon