Since 2001, hundreds of thousands of hours of underwater acoustic recordings have been made throughout the Southern Ocean south of 60° S. Detailed analysis of the occurrence of marine mammal sounds in these circumpolar recordings could provide novel insights into their ecology, but manual inspection of the entirety of all recordings would be prohibitively time consuming and expensive. Automated signal processing methods have now developed to the point that they can be applied to these data in a cost-effective manner. However training and evaluating the efficacy of these automated signal processing methods still requires a representative annotated library of sounds to identify the true presence and absence of different sound types. This work presents such a library of annotated recordings for the purpose of training and evaluating automated detectors of Antarctic blue and fin whale calls. Creation of the library has focused on the annotation of a representative sample of recordings to ensure that automated algorithms can be developed and tested across a broad range of instruments, locations, environmental conditions, and years. To demonstrate the utility of the library, we characterise the performance of two automated detection algorithms that have been commonly used to detect stereotyped calls of blue and fin whales. The availability of this library will facilitate development of improved detectors for the acoustic presence of Southern Ocean blue and fin whales. It can also be expanded upon to facilitate standardization of subsequent analysis of spatiotemporal trends in call-density of these circumpolar species.