Association for Computational Linguistics. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10352–10371, Toronto, Canada. Which Examples Should be Multiply Annotated? Active Learning When Annotators May Disagree. Anthology ID: 2023.findings-acl.658 Volume: Findings of the Association for Computational Linguistics: ACL 2023 Month: July Year: 2023 Address: Toronto, Canada Venue: Findings SIG: Publisher: Association for Computational Linguistics Note: Pages: 10352–10371 Language: URL: DOI: 10.18653/v1/2023.findings-acl.658 Bibkey: baumler-etal-2023-examples Cite (ACL): Connor Baumler, Anna Sotnikova, and Hal Daumé III. We find that traditional uncertainty-based active learning underperforms simple passive learning on tasks with high levels of disagreement, but that our active learning approach is able to successfully improve on passive and active baselines, reducing the number of annotations required by at least 24% on average across several datasets. Because we cannot know the true entropy of annotations on unlabeled examples, we estimate a model that predicts annotator entropy trained using very few multiply-labeled examples. Fortunately, for many tasks, not all examples are equally controversial we develop an active learning approach, Disagreement Aware Active Learning (DAAL) that concentrates annotations on examples where model entropy and annotator entropy are the most different. ![]() However, capturing disagreement can increase annotation time and expense. In such situations, preserving this disagreement through the machine learning pipeline can be important for downstream use cases. If you’re not sure what kind of annotations you need, consult your assignment guidelines or ask your instructor.Abstract Linguistic annotations, especially for controversial topics like hate speech detection, are frequently contested due to annotator backgrounds and positionalities. You’ll usually write either descriptive, evaluative, or reflective annotations. They shouldn’t go into too much depth quoting or discussing minor details from the source, but aim to write about it in broad terms. MLA states that annotations can describe or evaluate sources, or do both. Descriptive, evaluative, or reflective annotations? If in doubt, aim to keep your annotations short, but use multiple paragraphs if longer annotations are required for your assignment. However, it’s acceptable to write multiple-paragraph annotations if you need to. MLA states that annotations usually aim to be concise and thus are only one paragraph long.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |