2023

Lorandi, Michela ORCID: 0000-0002-6131-8763 and Belz, Anya ORCID: 0000-0002-0552-8096 (2023) Data-to-text generation for severely under-resourced languages with GPT-3.5: a bit of help needed from Google Translate. In: 16th International Natural Language Generation Conference - Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge, 11-12 Sep 2023, Prague, Czech Republic.

Popović, Maja ORCID: 0000-0001-8234-8745, Arvan, Mohammad, Parde, Natalie ORCID: 0000-0003-0072-7499 and Belz, Anya ORCID: 0000-0002-0552-8096 (2023) Exploring variation of results from different experimental conditions. In: ACL 2023, 9 - 14 July 2023, Toronto, Canada.

Lorandi, Michela and Belz, Anya (2023) How to control sentiment in text generation: a survey of the state-of-the-art in sentiment-control techniques. In: 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, 14 July 2023, Toronto, Canada.

Belz, Anya ORCID: 0000-0002-0552-8096, Thomson, Craig, Reiter, Ehud ORCID: 0000-0002-7548-9504, Abercrombie, Gavin ORCID: 0000-0002-6546-3562, Alonso-Moral, Jose M. ORCID: 0000-0003-3673-421X, Arvan, Mohammad, Cheung, Jackie, Cieliebak, Mark ORCID: 0009-0007-3059-8516, Clark, Elizabeth, van Deemter, Kees, Kelleher, John D. ORCID: 0000-0001-6462-3248 and Klubička, Filip ORCID: 0000-0001-9712-6141 (2023) Missing information, unresponsive authors, experimental flaws: the impossibility of assessing the reproducibility of previous human evaluations in NLP. In: Fourth Workshop on Insights from Negative Results in NLP, 2 May 2023, Dubrovnik, Croatia. ISBN 978-1-959429-49-4

Mille, Simon ORCID: 0000-0002-8852-2764, Ricci, Josep, Shvets, Alexander ORCID: 0000-0002-8370-2109 and Belz, Anya ORCID: 0000-0002-0552-8096 (2023) A pipeline for extracting abstract dependency templates for data-to-text natural language generation. In: Seventh International Conference on Dependency Linguistics (Depling, GURT/SyntaxFest 2023), 9-12 Mar 2023, Washington, D.C..

2022

Popović, Maja ORCID: 0000-0001-8234-8745 and Belz, Anya ORCID: 0000-0002-0552-8096 (2022) On reporting scores and agreement for error annotation tasks. In: 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), 7 Dec 2022, Abu Dhabi, United Arab Emirates & Online.

Huidrom, Rudali ORCID: 0000-0003-0630-3603 and Belz, Anya ORCID: 0000-0002-0552-8096 (2022) A survey of recent error annotation schemes for automatically generated text. In: 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), 7 Dec 2022, Abu Dhabi, UAE and Online.

Belz, Anya ORCID: 0000-0002-0552-8096 (2022) A metrological perspective on reproducibility in nlp. Computational Linguistics, 48 (4). pp. 1125-1135. ISSN 0891-2017

Savkov, Aleksandar ORCID: 0009-0009-6831-5563, Moramarco, Francesco, Korfiatis, Alex Papadopoulos, Perera, Mark, Belz, Anya ORCID: 0000-0002-0552-8096 and Reiter, Ehud ORCID: 0000-0002-7548-9504 (2022) Consultation checklists: standardising the human evaluation of medical note generation. In: EMNLP 2022 Industry Track, 9-11 Dec 2022, Abu Dhabi, UAE.

Belz, Anya ORCID: 0000-0002-0552-8096, Shimorina, Anastasia, Popović, Maja ORCID: 0000-0001-8234-8745 and Reiter, Ehud ORCID: 0000-0002-7548-9504 (2022) The 2022 ReproGen shared task on reproducibility of evaluations in NLG: overview and results. In: 15th International Conference on Natural Language Generation: Generation Challenges, 17-22 July 2022, Waterville, ME, USA.

Popović, Maja ORCID: 0000-0001-8234-8745, Castilho, Sheila ORCID: 0000-0002-8416-6555, Huidrom, Rudali ORCID: 0000-0003-0630-3603 and Belz, Anya ORCID: 0000-0002-0552-8096 (2022) Reproducing a manual evaluation of the simplicity of text simplification system outputs. In: 15th International Conference on Natural Language Generation: Generation Challenges, 17-22 July 2022, Waterville, ME, USA.

Huidrom, Rudali ORCID: 0000-0003-0630-3603, Dušek, Ondřej ORCID: 0000-0002-1415-1702, Kasner, Zdeněk ORCID: 0000-0002-5753-5538, Castro Ferrera, Thiago ORCID: 0000-0003-0200-3646 and Belz, Anya ORCID: 0000-0002-0552-8096 (2022) Two reproductions of a human-assessed comparative evaluation of a semantic error detection system. In: International Natural Language Generation Conference, 18-22 July 2022, Waterville, Maine, USA + Online.

Knoll, Tom, Moramarco, Francesco, Korfiatis, Alex Papadopoulos, Young, Rachel, Ruffini, Claudia, Perera, Mark, Perstl, Christian, Reiter, Ehud, Belz, Anya ORCID: 0000-0002-0552-8096 and Savkov, Aleksandar ORCID: 0009-0009-6831-5563 (2022) User-driven development of a medical note generation system. In: 20th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL'22), 10-15 July 2022, Seattle, Washington, USA.

Shimorina, Anastasia and Belz, Anya ORCID: 0000-0002-0552-8096 (2022) The human evaluation datasheet: a template for recording details of human evaluation experiments in NLP. In: 2nd Workshop on Human Evaluation of NLP Systems, 27 May 2022, Dublin, Ireland. ISBN 9781713867395

Belz, Anya ORCID: 0000-0002-0552-8096, Popović, Maja ORCID: 0000-0001-8234-8745 and Mille, Simon ORCID: 0000-0002-8852-2764 (2022) Quantified reproducibility assessment of NLP results. In: 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 22-27 May 2022, Dublin, Ireland.

Moramarco, Francesco, Korfiatis, Alex Papadopoulos, Perera, Mark, Juric, Damir, Flann, Jack, Reiter, Ehud, Belz, Anya ORCID: 0000-0002-0552-8096 and Savkov, Aleksandar ORCID: 0009-0009-6831-5563 (2022) Human evaluation and correlation with automatic metrics in consultation note generation. In: 60th Annual Meeting of the Association for Computational Linguistics, 22-27 May 2022, Dublin, Ireland.

2021

Popović, Maja ORCID: 0000-0001-8234-8745 and Belz, Anya ORCID: 0000-0002-0552-8096 (2021) A reproduction study of an annotation-based human evaluation of MT outputs. In: 14th International Conference on Natural Language Generation, 20-24 Sept 2021, Aberdeen, Scotland.

Mille, Simon ORCID: 0000-0002-8852-2764, Castro Ferreira, Thiago ORCID: 0000-0003-0200-3646, Davis, Brian ORCID: 0000-0002-5759-2655 and Belz, Anya ORCID: 0000-0002-0552-8096 (2021) Another PASS: a reproduction study of the human evaluation of a football report generation system. In: 14th International Conference on Natural Language Generation (INLG 2021), 22-27 May 2022, Aberdeen, Scotland.

Belz, Anya ORCID: 0000-0002-0552-8096, Agarwal, Shubham, Shimorina, Anastasia and Reiter, Ehud (2021) A systematic review of reproducibility research in natural language processing. In: 16th Conference of the European Chapter of the Association for Computational Linguistics: EACL'21, 19 - 23 Apr 2021, Online. ISBN 978-1-954085-02-2

2020

Howcroft, David ORCID: 0000-0002-0810-9065, Belz, Anya ORCID: 0000-0002-0552-8096, Gkatzia, Dimitra, Clinciu, Miruna, Hasan, Sadid, Mahamood, Saad ORCID: 0000-0003-2332-8749, Mille, Simon ORCID: 0000-0002-8852-2764, van Miltenburg, Emiel ORCID: 0000-0002-7143-8961, Santhanam, Sashank ORCID: 0000-0002-9412-3495 and Rieser, Verena ORCID: 0000-0001-6117-4395 (2020) Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In: 13th International Natural Language Generation Conference 2020 (INLG'20), 15-18 Dec 2020, Dublin, Ireland.

Belz, Anya ORCID: 0000-0002-0552-8096, Mille, Simon ORCID: 0000-0002-8852-2764 and Howcroft, David M. ORCID: 0000-0002-0810-9065 (2020) Disentangling the properties of human evaluation methods: a classification system to support comparability, meta-evaluation and reproducibility testing. In: 13th International Natural Language Generation Conference 2020 (INLG'20), 15-18 Dec 2020, Dublin, Ireland.

2019

Belz, Anya ORCID: 0000-0002-0552-8096, Hoile, Richard ORCID: 0000-0001-5003-1655, Ford, Elizabeth ORCID: 0000-0001-5613-8509 and Mullick, Azam (2019) Conceptualisation and annotation of drug nonadherence Information for knowledge extraction from patient-generated texts. In: 5th Workshop on Noisy User-generated Text (W-NUT 2019), 4 Nov 2019, Hong Kong, China.

2018

Birmingham, Brandon ORCID: 0000-0002-3006-3526, Muscat, Adrian ORCID: 0000-0002-9157-2818 and Belz, Anya ORCID: 0000-0002-0552-8096 (2018) Adding the third dimension to spatial relation detection in 2d images. In: 11th International Conference on Natural Language Generation, 5-8 Nov 2018, Tilburg, The Netherlands.

Mille, Simon ORCID: 0000-0002-8852-2764, Belz, Anya ORCID: 0000-0002-0552-8096, Bohnet, Bernd and Wanner, Leo ORCID: 0000-0002-9446-3748 (2018) Underspecified universal dependency structures as inputs for multilingual surface realisation. In: 11th International Conference on Natural Language Generation, 5-8 Nov 2018, Tilburg, The Netherlands.

2010

Belz, Anya, White, Mike, van Genabith, Josef ORCID: 0000-0003-1322-7944, Hogan, Deirdre and Stent, Amanda (2010) Finding common ground: towards a surface realisation shared task. In: INLG 2010 - 6th International Natural Language Generation Conference, 7-9 July 2010, Trim, Ireland.

2006

Belz, Anya ORCID: 0000-0002-0552-8096 and Kilgarriff, Adam (2006) Shared-task evaluations in HLT. In: Fourth International Natural Language Generation Conference on-INLG'06, 15-16 July 2006, Sydney, Australia. ISBN 9781617384981