Non-invasive extraction of white shark swimming kinematics from unoccupied aircraft system (UAS) imagery
Alexandra E. DiGiacomo
A * , Ann Marie Abraham A and Barbara A. Block A
A
Abstract
Consumer-grade unoccupied aircraft systems (UAS) are increasingly being used by both scientists and hobbyists in the coastal environment. Marine megafauna are observed via UAS as part of monitoring programs, recreational interests, and scientific research, amassing aerial imagery datasets. Because manual documentation of these datasets is infeasible at scale, efficient approaches leveraging computer vision and deep learning have emerged to detect and classify marine megafauna.
This study provides a workflow to quantitatively estimate swimming kinematics tailbeat frequency (TBF) and tailbeat amplitude (TBA) of white sharks (Carcharodon carcharias) from aerial UAS video data.
Body pose estimation was performed using computer vision model DeepLabCut to track six key white shark body parts across UAS videos. The relative positions of these body part coordinates were used to compute tail position over time and quantify TBF and TBA across a population of white sharks in Monterey Bay, California.
With a training set of just 52 images, the deep residual neural network reaches human-level labeling accuracy of body parts (root mean square error of <1.3 cm). This workflow is applied to 76 focal follows representing 34 individuals to produce TBF (0.43 ± 0.07 Hz) and TBA (0.24 ± 0.10 BL) values similar to those derived from biologging devices previously deployed on individuals in this population.
The results indicated that body pose estimation via DeepLabCut can allow for the rapid extraction of quantitative kinematics such as TBF and TBA in juvenile white shark populations that aggregate in coastal habitats.
This approach provides a non-invasive, scalable method to understanding megafauna kinematics in sensitive species that overcomes the logistical barriers of traditional biologging approaches.
Keywords: behaviour, computer vision, deep learning, drones, ecology, locomotion, unoccupied aircraft systems (UAS), vertebrates.
References
Akbari Y, Almaadeed N, Al-maadeed S, Elharrouss O (2021) Applications, databases and open computer vision research from drone videos and images: a survey. Artificial Intelligence Review 54, 3887-3938.
| Crossref | Google Scholar |
Anderson JM, Spurgeon E, Stirling BS, May J, Rex PT, Hyla B, McCullough S, Thompson M, Lowe CG (2022) High resolution acoustic telemetry reveals swim speeds and inferred field metabolic rates in juvenile white sharks Carcharodon carcharias. PLoS ONE 17, e0268914.
| Crossref | Google Scholar |
Bierlich KC, Hewitt J, Schick RS, Pallin L, Dale J, Friedlaender AS, Christiansen F, Sprogis KR, Dawn AH, Bird CN, Larsen GD, Nichols R, Shero MR, Goldbogen J, Read AJ, Johnston DW (2022) Seasonal gain in body condition of foraging humpback whales along the Western Antarctic Peninsula. Frontiers in Marine Science 9, 1036860.
| Crossref | Google Scholar |
Blank JM, Morrissette JM, Farwell CJ, Price M, Schallert RJ, Block BA (2007) Temperature effects on metabolic rate of juvenile Pacific bluefin tuna Thunnus orientalis. Journal of Experimental Biology 210, 4254-4261.
| Crossref | Google Scholar | PubMed |
Block BA, Dewar H, Blackwell SB, Williams T, Prince E, Boustany AM, Farwell C, Dau DJ, Seitz A (2001) Archival and pop-up satellite tagging of Atlantic bluefin tuna. In ‘Electronic tagging and tracking in marine fisheries: proceedings of the symposium on tagging and tracking marine fish with electronic devices’, 7–11 February, 2000, East-West Center, University of Hawaii. (Eds JR Sibert, JL Nielsen) pp. 65–88. (Springer Netherlands: Dordrecht, Netherlands) 10.1007/978-94-017-1402-0_3
Block BA, Jonsen ID, Jorgensen SJ, Winship AJ, Shaffer SA, Bograd SJ, Hazen EL, Foley DG, Breed GA, Harrison A-L, Ganong JE, Swithenbank A, Castleton M, Dewar H, Mate BR, Shillinger GL, Schaefer KM, Benson SR, Weise MJ, Henry RW, Costa DP (2011) Tracking apex marine predator movements in a dynamic ocean. Nature 475, 86-90.
| Crossref | Google Scholar | PubMed |
Boustany AM, Davis SF, Pyle P, Anderson SD, Le Boeuf BJ, Block BA (2002) Expanded niche for white sharks. Nature 415, 35-36.
| Crossref | Google Scholar | PubMed |
Burnett JD, Lemos L, Barlow D, Wing MG, Chandler T, Torres LG (2019) Estimating morphometric attributes of baleen whales with photogrammetry from small UASs: a case study with blue and gray whales. Marine Mammal Science 35, 108-139.
| Crossref | Google Scholar |
Chapple TK, Jorgensen SJ, Anderson SD, Kanive PE, Klimley AP, Botsford LW, Block BA (2011) A first estimate of white shark, Carcharodon carcharias, abundance off central California. Biology Letters 7, 581-583.
| Crossref | Google Scholar | PubMed |
Chapple TK, Gleiss AC, Jewell OJD, Wikelski M, Block BA (2015) Tracking sharks without teeth: a non-invasive rigid tag attachment for large predatory sharks. Animal Biotelemetry 3, 14.
| Crossref | Google Scholar |
Chikodili NB, Abdulmalik MD, Abisoye OA, Bashir SA (2021) Outlier detection in multivariate time series data using a fusion of K-Medoid, standardized Euclidean distance and z-score. In ‘Information and communication technology and applications’. (Eds S Misra, B Muhammad-Bello) pp. 259–271. (Springer International Publishing: Cham, Switzerland) 10.1007/978-3-030-69143-1_21
Clark TD, Farwell CJ, Rodriguez LE, Brandt WT, Block BA (2013) Heart rate responses to temperature in free-swimming Pacific bluefin tuna (Thunnus orientalis). Journal of Experimental Biology 216, 3208-3214.
| Crossref | Google Scholar | PubMed |
Colefax AP, Kelaher BP, Pagendam DE, Butcher PA (2020) Assessing white shark (Carcharodon carcharias) behavior along coastal beaches for conservation-focused shark mitigation. Frontiers in Marine Science 7, 268.
| Crossref | Google Scholar |
Di Santo V, Goerig E, Wainwright DK, Akanyeti O, Liao JC, Castro-Santos T, Lauder GV (2021) Convergence of undulatory swimming kinematics across a diversity of fishes. Proceedings of the National Academy of Sciences 118, e2113206118.
| Crossref | Google Scholar |
Donley JM, Sepulveda CA, Konstantinidis P, Gemballa S, Shadwick RE (2004) Convergent evolution in mechanical design of lamnid sharks and tunas. Nature 429, 61-65.
| Crossref | Google Scholar | PubMed |
Ferretti F, Worm B, Britten GL, Heithaus MR, Lotze HK (2010) Patterns and ecosystem consequences of shark declines in the ocean. Ecology Letters 13, 1055-1071.
| Crossref | Google Scholar | PubMed |
Fong V, Hoffmann SL, Pate JH (2022) Using drones to assess volitional swimming kinematics of manta ray behaviors in the wild. Drones 6, 111.
| Crossref | Google Scholar |
Gorssen W, Winters C, Meyermans R, D’Hooge R, Janssens S, Buys N (2022) Estimating genetics of body dimensions and activity levels in pigs using automated pose estimation. Scientific Reports 12, 15384.
| Crossref | Google Scholar |
Hammerschlag N, Gallagher AJ, Lazarre DM (2011) A review of shark satellite tagging studies. Journal of Experimental Marine Biology and Ecology 398, 1-8.
| Crossref | Google Scholar |
Heim V, Lüscher D, Hottinger J, Ebert D (2022) Development and validation of a drill attachment for faster and safer deployments of fin-mounted geolocators in large-bodied sharks. Animal Biotelemetry 10, 33.
| Crossref | Google Scholar |
Hoffmann SL, Warren SM, Porter ME (2017) Regional variation in undulatory kinematics of two hammerhead species: the bonnethead (Sphyrna tiburo) and the scalloped hammerhead (Sphyrna lewini). Journal of Experimental Biology 220, 3336-3343.
| Crossref | Google Scholar | PubMed |
Jewell OJD, Gleiss AC, Jorgensen SJ, Andrzejaczek S, Moxley JH, Beatty SJ, Wikelski M, Block BA, Chapple TK (2019) Cryptic habitat use of white sharks in kelp forest revealed by animal-borne video. Biology Letters 15, 20190085.
| Crossref | Google Scholar |
Jewell OJD, Chapple TK, Jorgensen SJ, Kanive P, Moxley JH, Tweedley JR, Anderson S, Block BA, Gleiss AC (2024) Diverse habitats shape the movement ecology of a top marine predator, the white shark Carcharodon carcharias. Ecosphere 15, e4825.
| Crossref | Google Scholar |
Jorgensen SJ, Reeb CA, Chapple TK, Anderson S, Perle C, Van Sommeran SR, Fritz-Cope C, Brown AC, Klimley AP, Block BA (2010) Philopatry and migration of Pacific white sharks. Proceedings of the Royal Society B: Biological Sciences 277, 679-688.
| Crossref | Google Scholar |
Kanive PE, Rotella JJ, Chapple TK, Anderson SD, White TD, Block BA, Jorgensen SJ (2021) Estimates of regional annual abundance and population growth rates of white sharks off central California. Biological Conservation 257, 109104.
| Crossref | Google Scholar |
Kock AA, Photopoulou T, Durbach I, Mauff K, Meÿer M, Kotze D, Griffiths CL, O’Riain MJ (2018) Summer at the beach: spatio-temporal patterns of white shark occurrence along the inshore areas of False Bay, South Africa. Movement Ecology 6, 7.
| Crossref | Google Scholar |
Kuhlane L, Brown D, Boby A (2023) Exploring the incremental improvements of YOLOv5 on tracking and identifying great white sharks in Cape Town. In ‘Intelligent computing’. (Ed. K Arai) pp. 1455–1473. (Springer Nature Switzerland: Cham, Switzerland) 10.1007/978-3-031-37963-5_98
Letessier TB, Bouchet PJ, Meeuwig JJ (2017) Sampling mobile oceanic fishes and sharks: implications for fisheries and conservation planning. Biological Reviews 92, 627-646.
| Crossref | Google Scholar | PubMed |
Li J, Kang F, Zhang Y, Liu Y, Yu X (2023) Research on tracking and identification of typical protective behavior of cows based on DeepLabCut. Applied Sciences 13, 1141.
| Crossref | Google Scholar |
Lowe CG (1996) Kinematics and Critical Swimming Speed of Juvenile Scalloped Hammerhead Sharks. Journal of Experimental Biology 199, 2605-2610.
| Crossref | Google Scholar | PubMed |
Lowe CG, Goldman KJ (2001) Thermal and bioenergetics of elasmobranchs: bridging the gap. Environmental Biology of Fishes 60, 251-266.
| Crossref | Google Scholar |
Martin BD, Emery KO (1967) Geology of Monterey Canyon, California. AAPG Bulletin 51, 2281-2304.
| Crossref | Google Scholar |
Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, Bethge M (2018) DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience 21, 1281-1289.
| Crossref | Google Scholar | PubMed |
Moxley JH, Nicholson TE, Van Houtan KS, Jorgensen SJ (2019) Non-trophic impacts from white sharks complicate population recovery for sea otters. Ecology and Evolution 9, 6378-6388.
| Crossref | Google Scholar | PubMed |
Muhling BA, Snyder S, Hazen EL, Whitlock RE, Dewar H, Park J-Y, Stock CA, Block BA (2022) Risk and reward in foraging migrations of North Pacific albacore determined from estimates of energy intake and movement costs. Frontiers in Marine Science 9, 730428.
| Crossref | Google Scholar |
Porter ME, Ruddy BT, Kajiura SM (2020) Volitional swimming kinematics of blacktip sharks, Carcharhinus limbatus, in the Wild. Drones 4, 78.
| Crossref | Google Scholar |
Ramp SR, Paduan JD, Shulman I, Kindle J, Bahr FL, Chavez F (2005) Observations of upwelling and relaxation events in the northern Monterey Bay during August 2000. Journal of Geophysical Research: Oceans 110, C07013.
| Crossref | Google Scholar |
Rex PT, May JHM, III, Pierce EK, Lowe CG (2023) Patterns of overlapping habitat use of juvenile white shark and human recreational water users along southern California beaches. PLoS ONE 18, e0286575.
| Crossref | Google Scholar |
Runde BJ, Buckel JA, Bacheler NM, Tharp RM, Rudershausen PJ, Harms CA, Ben-Horin T (2022) Evaluation of six methods for external attachment of electronic tags to fish: assessment of tag retention, growth and fish welfare. Journal of Fish Biology 101, 419-430.
| Crossref | Google Scholar | PubMed |
Sánchez-Rodríguez J, Raufaste C, Argentina M (2023) Scaling the tail beat frequency and swimming speed in underwater undulatory swimming. Nature Communications 14, 5569.
| Crossref | Google Scholar |
Sato K, Watanuki Y, Takahashi A, Miller PJO, Tanaka H, Kawabe R, Ponganis PJ, Handrich Y, Akamatsu T, Watanabe Y, Mitani Y, Costa DP, Bost C-A, Aoki K, Amano M, Trathan P, Shapiro A, Naito Y (2006) Stroke frequency, but not swimming speed, is related to body size in free-ranging seabirds, pinnipeds and cetaceans. Proceedings of the Royal Society B: Biological Sciences 274, 471-477.
| Crossref | Google Scholar |
Semmens JM, Kock AA, Watanabe YY, Shepard CM, Berkenpas E, Stehfest KM, Barnett A, Payne NL (2019) Preparing to launch: biologging reveals the dynamics of white shark breaching behaviour. Marine Biology 166, 95.
| Crossref | Google Scholar |
Steinhausen MF, Steffensen JF, Andersen NG (2005) Tail beat frequency as a predictor of swimming speed and oxygen consumption of saithe (Pollachius virens) and whiting (Merlangius merlangus) during forced swimming. Marine Biology 148, 197-204.
| Crossref | Google Scholar |
Tanaka KR, Van Houtan KS, Mailander E, Dias BS, Galginaitis C, O’Sullivan J, Lowe CG, Jorgensen SJ (2021) North Pacific warming shifts the juvenile range of a marine apex predator. Scientific Reports 11, 3373.
| Crossref | Google Scholar |
Torres WI, Bierlich K (2020) MorphoMetriX: a photogrammetric measurement GUI for morphometric analysis of megafauna. Journal of Open Source Software 5, 1825.
| Crossref | Google Scholar |
Torres LG, Nieukirk SL, Lemos L, Chandler TE (2018) Drone Up! Quantifying whale behavior from a new perspective improves observational capacity. Frontiers in Marine Science 5, 319.
| Crossref | Google Scholar |
Towner AV, Kock AA, Stopforth C, Hurwitz D, Elwen SH (2023) Direct observation of killer whales preying on white sharks and evidence of a flight response. Ecology 104, e3875.
| Crossref | Google Scholar |
Virtanen P, Gommers R, Oliphant TE, Haberland M, Reddy T, Cournapeau D, Burovski E, Peterson P, Weckesser W, Bright J, van der Walt SJ, Brett M, Wilson J, Millman KJ, Mayorov N, Nelson ARJ, Jones E, Kern R, Larson E, Carey CJ, Polat İ, Feng Y, Moore EW, VanderPlas J, Laxalde D, Perktold J, Cimrman R, Henriksen I, Quintero EA, Harris CR, Archibald AM, Ribeiro AH, Pedregosa F, van Mulbregt P, et al. (2020) SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods 17, 261-272.
| Crossref | Google Scholar | PubMed |
Watanabe YY, Papastamatiou YP (2023) Biologging and biotelemetry: tools for understanding the lives and environments of marine animals. Annual Review of Animal Biosciences 11, 247-267.
| Crossref | Google Scholar | PubMed |
Watanabe YY, Payne NL, Semmens JM, Fox A, Huveneers C (2019) Hunting behaviour of white sharks recorded by animal-borne accelerometers and cameras. Marine Ecology Progress Series 621, 221-227.
| Crossref | Google Scholar |
Weng KC, Boustany AM, Pyle P, Anderson SD, Brown A, Block BA (2007) Migration and habitat of white sharks (Carcharodon carcharias) in the eastern Pacific Ocean. Marine Biology 152, 877-894.
| Crossref | Google Scholar |
Wilga CD, Lauder GV (2000) Three-dimensional kinematics and wake structure of the pectoral fins during locomotion in leopard sharks Triakis semifasciata. Journal of Experimental Biology 203, 2261-2278.
| Crossref | Google Scholar | PubMed |
Wiltshire C, Lewis-Cheetham J, Komedová V, Matsuzawa T, Graham KE, Hobaiter C (2023) DeepWild: application of the pose estimation tool DeepLabCut for behaviour tracking in wild chimpanzees and bonobos. Journal of Animal Ecology 92, 1560-1574.
| Crossref | Google Scholar | PubMed |
Wolf PR, Dewitt BA, Wilkinson BE (2014) Elements of photogrammetry with applications in GIS. 4th edn. McGraw-Hill Education. Available at https://www.accessengineeringlibrary.com/content/book/9780071761123 [accessed 5 November 2024]
