%0 Journal Article %T Assessing inter-annotator agreement from collaborative annotation campaign in marine bioacoustics %+ Sorbonne Université (SU) %+ Laboratoire Géosciences Océan (LGO) %+ Equipe Observations Signal & Environnement (Lab-STICC_OSE) %+ École Nationale Supérieure de Techniques Avancées Bretagne (ENSTA Bretagne) %+ Institute of Sound and Vibration Research (ISVR) %+ DGA Techniques navales %+ Université Paris-Saclay %A Nguyen Hong Duc, Paul %A Torterotot, Maëlle %A Samaran, Flore %A White, Paul R. %A Gérard, Odile %A Adam, Olivier %A Cazau, Dorian %Z This work was supported by the CominLabs7 through the innovation action Tech4Whales, DREC Agence Française de la Biodiversité8 and ISblue.9 Paul Nguyen Hong Duc was supported by the French Defence Procurement Agency (Direction Générale des Armements) and Maëlle Torterotot was supported by a PhD fellowship from the University of Brest. %< avec comité de lecture %@ 1574-9541 %J Ecological Informatics %I Elsevier %V 61 %P 101185-1 - 101185-12 %8 2021-03 %D 2021 %R 10.1016/j.ecoinf.2020.101185 %K Efficiency measurement %K Detection method %K Data set %K Data processing %K DCLDE 2015 low frequency %K Multi-annotator agreement %K Marine bioacoustics %K Reliability %Z Life Sciences [q-bio]/Ecology, environment %Z Physics [physics]/Mechanics [physics]/Acoustics [physics.class-ph] %Z Life Sciences [q-bio]/Animal biologyJournal articles %X It is currently widely recognized that automated methods are crucial to help processing long-term recordings of marine bioacoustics. To evaluate the efficiency of such methods, it is essential to develop large-scale annotated datasets. However, besides being laborious and resource intensive, recent studies have suggested that such a task could also be highly subjective with the generation of annotator specific errors. In this work, we investigate the question of inter-annotator agreement from a multi-annotator annotation campaign performed on a marine bioacoustics dataset. After providing quantitative evidence of inter-annotator variability, we investigate potential sources on both the user annotation practice and the annotation data and task to better understand why and how such variability occurs. Our study reveals that the acoustic event type, the Signal-to-Noise Ratio of the acoustic event and the annotator profile are three examples of critical factors impacting the annotation results of a multi-annotator campaign. %G English %2 https://ensta-bretagne.hal.science/hal-03028306/document %2 https://ensta-bretagne.hal.science/hal-03028306/file/manuscript_wo_journal.pdf %L hal-03028306 %U https://ensta-bretagne.hal.science/hal-03028306 %~ UNIV-BREST %~ INSTITUT-TELECOM %~ ENSTA-BRETAGNE %~ CNRS %~ UNIV-UBS %~ IFREMER %~ ENSTA-BRETAGNE-STIC %~ IUEM %~ THESES_IUEM %~ GIP-BE %~ ENIB %~ LAB-STICC %~ UNIV-PARIS-SACLAY %~ LGO %~ SORBONNE-UNIVERSITE %~ SORBONNE-UNIV %~ INSTITUTS-TELECOM %~ TEST-HALCNRS %~ ISBLUE %~ UNIVERSITE-PARIS-SACLAY %~ SU-TI %~ ANR %~ LAB-STICC_OSE %~ LAB-STICC_IAOCEAN %~ ALLIANCE-SU %~ GEO-OCEAN