The evaluation toolkit

You can download the evaluation toolkit and sample results in two ways.

First : You can download the evaluation toolkit(WATB_Benchmark_toolkit.rar) and tracking results(WATB_tracking_results.rar) in Google Drive: Second : You can download the evaluation toolkit(WATB_Benchmark_toolkit.rar) and tracking results(WATB_tracking_results.rar) in Baidu Netdisk:

Evaluated trackers

In order to evaluate existing trackers’ performance on WATB, we tested 38 tracking methods including CF and deep learning based trackers For CF trackers and deep trackers,we select the tracking methods with hand-crafted features and the recent popular Siamese trackers. Details are shown in the table below:

Trackers Description Language Trackers Description Language
Stark ICCV21 Python CSK ECCV12 Matlab
STMTrack CVPR21 Python KCF PAMI14 Matlab
TransT CVPR21 Python SAMF ECCVW14 Matlab
Dimp50 ICCV19 Python DSST BMVA14 Matlab
Dimp18 ICCV19 Python fDSST PAMI16 Matlab
SiamCAR CVPR20 Python CF2 ICCV15 Matlab
SiamAttn CVPR20 Python SRDCF ICCV15 Matlab
ATOM CVPR19 Python Staple CVPR16 Matlab
SiamRPN++ CVPR19 Python CFWCR ICCVW17 Matlab
SiamRPN CVPR18 Python BACF ICCV17 Matlab
SiamMask CVPR19 Python ECO CVPR17 Matlab
SiamGAT CVPR21 Python STRCF CVPR18 Matlab
SiamBAN CVPR20 Python SKSCF PAMI18 Matlab
HiFT ICCV21 Python DSARCF TIP19 Matlab
SiamFC ECCV16 Python LADCF TIP19 Matlab
ACSDCF_HC IJCV21 Matlab ASRCF CVPR19 Matlab
MRCF TIE22 Matlab GFSDCF ICCV19 Matlab
MSCF ICRA21 Matlab ARCF ICCV19 Matlab
AutoTrack CVPR20 Matlab DRCF TGRS20 Matlab

Overall all Performance

We extensively tested 38 trackers on WATB. Each tracker is run individually without any modification and re-training. We employ the precision measure,normalized precision measure and success measure for all the tested trackers and draw corresponding plots to compare their performance which are shown in figure below.

precision normalized precision success measures

Species-based Evaluation

We divided the WATB into 8 categories according to species, by the way, evaluated these 8 categories as follows.

species precision normalized precision success measures
Bird
Fish
Mammals
Arthropod
Amphibian
Reptile
Mollusc
Coelenterate

Attribute-based Evaluation

In order to detect the strengths and limitations of existing trackers on wild animal tracking, we provide attribute based evaluation under the 13 challenging attributes of WATB.

attribute precision normalized precision success measures
IV
OPR
IPR
DEF
FM
SV
CM
OV
POC
FOC
LR
SOB
MB

Test results

The normalized precision scores of the evaluated trackers over 13 attributes are given in following table. The results of deep trackers and CF trackers are separated and listed in different parts of the table. For deep trackers, the best three results are shown in. red, blue and green, respectively, while the best three CF trackers are shown in italic style using similar colors to deep trackers.

Trackers SV OPR POC FM DEF MB CM LR SO FOC IPR IV OV
Stark 0.567 0.544 0.531 0.510 0.635 0.517 0.524 0.528 0.537 0.418 0.557 0.503 0.524
STMTrack 0.535 0.535 0.488 0.433 0.598 0.451 0.428 0.496 0.559 0.346 0.538 0.510 0.456
TransT 0.532 0.522 0.494 0.460 0.590 0.459 0.453 0.476 0.548 0.371 0.538 0.483 0.484
Dimp50 0.507 0.492 0.471 0.444 0.572 0.445 0.437 0.480 0.527 0.349 0.506 0.469 0.469
SiamAttn 0.501 0.495 0.464 0.402 0.536 0.426 0.409 0.422 0.566 0.336 0.516 0.464 0.423
SiamBAN 0.496 0.496 0.456 0.403 0.534 0.432 0.402 0.421 0.551 0.322 0.488 0.446 0.439
Diamp18 0.495 0.487 0.463 0.427 0.546 0.450 0.414 0.466 0.510 0.347 0.534 0.459 0.449
ATOM 0.473 0.457 0.433 0.408 0.516 0.425 0.427 0.427 0.520 0.333 0.505 0.467 0.426
SiamRPN++ 0.457 0.457 0.428 0.363 0.493 0.400 0.373 0.407 0.512 0.305 0.471 0.452 0.394
SiamCAR 0.455 0.466 0.428 0.371 0.486 0.411 0.356 0.416 0.534 0.311 0.492 0.409 0.419
SiamRPN 0.435 0.440 0.417 0.356 0.466 0.380 0.355 0.382 0.497 0.310 0.451 0.432 0.404
SiamMask 0.430 0.436 0.395 0.341 0.470 0.365 0.344 0.393 0.532 0.277 0.458 0.397 0.376
SiamFC 0.359 0.339 0.326 0.309 0.376 0.297 0.288 0.341 0.430 0.241 0.390 0.350 0.326
SiamGAT 0.312 0.308 0.283 0.252 0.350 0.245 0.279 0.259 0.330 0.197 0.372 0.298 0.255
HiFT 0.251 0.246 0.224 0.195 0.278 0.193 0.232 0.209 0.293 0.142 0.305 0.265 0.193
GFSDCF 0.417 0.417 0.395 0.356 0.427 0.357 0.342 0.382 0.494 0.299 0.454 0.424 0.357
ASRCF 0.329 0.326 0.318 0.258 0.336 0.275 0.243 0.303 0.431 0.236 0.364 0.343 0.307
CF2 0.304 0.295 0.275 0.257 0.310 0.267 0.249 0.265 0.372 0.216 0.317 0.277 0.288
ARCF 0.296 0.286 0.274 0.249 0.296 0.270 0.238 0.273 0.376 0.210 0.321 0.268 0.291
ECO 0.295 0.270 0.268 0.257 0.274 0.249 0.233 0.277 0.336 0.218 0.322 0.281 0.280
LADCF 0.292 0.273 0.271 0.247 0.295 0.259 0.228 0.271 0.361 0.205 0.326 0.282 0.281
STRCF 0.286 0.275 0.268 0.240 0.284 0.240 0.220 0.255 0.369 0.213 0.288 0.302 0.240
MRCF 0.286 0.272 0.271 0.237 0.287 0.257 0.226 0.267 0.367 0.211 0.310 0.269 0.277
MSCF 0.285 0.279 0.271 0.234 0.290 0.248 0.236 0.254 0.373 0.207 0.297 0.294 0.264
CFWCR 0.281 0.268 0.253 0.239 0.274 0.251 0.236 0.268 0.342 0.206 0.277 0.298 0.249
ACSDCF_HC 0.275 0.272 0.256 0.223 0.279 0.226 0.216 0.257 0.381 0.210 0.310 0.285 0.228
AutoTrack 0.274 0.264 0.252 0.224 0.266 0.240 0.218 0.247 0.349 0.196 0.287 0.258 0.271
BACF 0.272 0.270 0.254 0.210 0.259 0.243 0.194 0.250 0.375 0.202 0.248 0.285 0.264
Staple 0.265 0.271 0.252 0.207 0.267 0.228 0.217 0.237 0.368 0.184 0.304 0.283 0.247
DRCF 0.257 0.247 0.255 0.209 0.247 0.226 0.190 0.230 0.318 0.187 0.254 0.211 0.254
SAMF 0.244 0.242 0.232 0.186 0.232 0.216 0.198 0.200 0.332 0.176 0.252 0.236 0.245
SRDCF 0.229 0.220 0.215 0.201 0.210 0.204 0.197 0.206 0.295 0.177 0.215 0.231 0.231
DSST 0.224 0.225 0.213 0.179 0.225 0.182 0.172 0.213 0.317 0.180 0.213 0.226 0.193
DSARCF 0.222 0.219 0.214 0.190 0.201 0.204 0.182 0.196 0.293 0.174 0.203 0.214 0.230
KCF 0.212 0.209 0.206 0.180 0.208 0.180 0.173 0.187 0.305 0.171 0.244 0.220 0.197
SKSCF 0.209 0.215 0.206 0.176 0.207 0.184 0.188 0.171 0.285 0.167 0.217 0.224 0.196
fDSST 0.205 0.198 0.203 0.168 0.184 0.169 0.170 0.173 0.271 0.168 0.206 0.225 0.202
CSK 0.164 0.166 0.162 0.137 0.157 0.135 0.136 0.171 0.251 0.133 0.179 0.187 0.172
Copyright @ WATB Team 2022