As AI-based critical systems are expected to operate in dynamic environments, it is crucial to ensure their reliability under various operational conditions. In computer vision, one way to achieve this is by testing the system's robustness to input image perturbations. However, while many methods have been proposed and evaluated in academic settings, their effectiveness and applicability in practice remain largely unknown. In this paper, we report the results of testing the robustness of an industrial case of an AI-based road object detection system, in a black-box setting. By defining relevant perturbations and metrics, we analyse the system's response to changes in its hardware and software environment, and identify areas for improvement through retraining with data augmentation. We address the key challenges encountered during this evaluation and provide insights that may help practitioners in performing similar tests and guide future research on robustness testing of AI-based object detection systems.