Deep learning-based multiclass wild animal detection and behavioral classification in camera trap images: An ensemble framework with systematic review

Abstract

The automated image analysis of wildlife images captured by camera traps is crucial for conservation efforts, ecological research and anti-poaching initiatives. But in large-scale camera trap datasets, species detection accuracy is still difficult to achieve, and the classification of the species is difficult to do concurrently with the classification of the captured behavior. However, it is still challenging to be able to accurately detect the species and at the same time classify the behavior in large-scale camera trap data due to high intra-class variation, complex backgrounds, occlusion and class imbalance. In this paper, we present a novel ensemble deep learning framework that combines ResNet-50, EfficientNet-B7 and Vision Transformer (ViT-B/16) models with a multi-scale feature fusion layer for the simultaneous classification of species and behavioral states of ten wildlife species in the African savanna region. We propose a new camera-trap database with 31,400 images from ten species covered with fine-grained annotations of their behavior. The extensive experiments illustrated that our proposed ensemble model exhibits 95.8% top-1 accuracy and 95.2% macro F1-score which outperforms existing state-of-the-art methods such as YOLOv8 based detectors (93.0%), the standalone EfficientNet-B7 (91.4%) and Vision Transformer based baseline methods (92.1%). The framework achieves an average AUC of 0.991 for all species and can yield real-time inference at 34.7ms per image. A detailed systematic literature review of 87 papers from 2018 to 2024 is provided, showing the various approaches: traditional machine learning methods, CNN-based, transformer based, multi-task frameworks, and behavioural analysis based methods. This work provides a sturdy baseline for automated wildlife monitoring systems that can be deployed in a resource constrained field setting, and a publicly released dataset (CTW-10) for reproducible benchmarking by the conservation informatics community. The ensemble approach is able to utilize the complementary spatial characteristics from different backbone families and achieve excellent fine-grained species discrimination