Abstract
Most existing performance evaluation methods concentrate on defining various metrics over a wide range of conditions and generating standard benchmarking video sequences to examine the effectiveness of a video tracking system. It is a common practice to incorporate a robustness margin or factor into the system/algorithm design. However, these methods, deterministic approaches, often lead to overdesign, thus increasing costs, or underdesign, causing frequent system failures. In order to overcome the aforementioned limitations, we propose an alternative framework to analyze the physics of the failure process via the concept of reliability. In comparison with existing approaches where system performance is evaluated based on a given benchmarking sequence, the advantage of our proposed framework lies in that a unified and statistical index is used to evaluate the performance of an automated video surveillance system independent of input sequences. Meanwhile, based on our proposed framework, the uncertainty problem of a failure process caused by the system’s complexity, imprecise measurements of the relevant physical constants and variables, and the indeterminate nature of future events can be addressed accordingly.
[1] Bashir F., Porikli F., Performance evaluation of object detection and tracking systems, In: Proceedings of the 9th IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS 06), New York, NY, USA, 2006 Search in Google Scholar
[2] Collins R.T., Lipton A.J., Kanade, T., Introduction to the special section on video surveillance, IEEE T. Pattern Anal., 22(8), 745–746, 2000 http://dx.doi.org/10.1109/TPAMI.2000.86867610.1109/TPAMI.2000.868676Search in Google Scholar
[3] Calderara S., Prati A., Vezzani R., Cucchiara R., Consistent labeling for multi-camera object tracking, In: The 13th International Conference on Image Analysis and Processing, Roli, F., Vitulano, S. (Eds.), Springer, LNCS 3617, 1206–1214, 2005 10.1007/11553595_148Search in Google Scholar
[4] Chau D.P., Bremond F., Thonnat M. Online evaluation of tracking algorithm performance, In: The Int. Conf. on Imaging for Crime Detection and Prevention, 1–6, 2009 Search in Google Scholar
[5] Cui Y., Samarasekera S., Huang Q., Greiffenhagen M., Indoor monitoring via the collaboration between a peripheral sensor and a foveal sensor, In: IEEE Workshop on Visual Surveillance, 2–9, 1998 Search in Google Scholar
[6] Doermann D. Mihalcik D., Tools and techniques for video performance evaluation, In: The 15th Int. Conf. on Pattern Recognition, vol. 4, 167–170, 2000 http://dx.doi.org/10.1109/ICPR.2000.90288810.1109/ICPR.2000.902888Search in Google Scholar
[7] Dodson B. Nolan D. Reliability engineering handbook, CRC Press, 1999 Search in Google Scholar
[8] Dai Y.-S., Xie M., Log Q., Ng S.-H., Uncertainty analysis in software reliability modeling by Bayesian approach with maximum-entropy principle, IEEE T. Software Eng., 33(11), 781–795, 2007 http://dx.doi.org/10.1109/TSE.2007.7073910.1109/TSE.2007.70739Search in Google Scholar
[9] Ebeling C.E., An introduction to reliability and maintainability engineering, McGraw-Hill, 1997 Search in Google Scholar
[10] Erdem C., Tekalp A., Sankur B., Metrics for performance evaluation of video object segmentation and tracking without ground-truth, IEEE Image Proc., 2, 69–72, 2001 Search in Google Scholar
[11] Jaynes C., Webb S., Steele R. M., Xiong Q., An open development environment for evaluation of video surveillance systems, In: The 3rd Int. Workshop on Performance Evaluation of Tracking and Surveillance, 2002 Search in Google Scholar
[12] Kapur J., Maximum-entropy models in science and engineering, John Wiley & Sons, 1989 Search in Google Scholar
[13] Kasturi R., Goldgof D., Soundararajan P., Manohar V., Garofolo J., Bowers R., Boonstra M., Korzhova V., Zhang J., Framework for performance evaluation of face, text, and vehicle detection and tracking in video: data, metrics, and protocol, IEEE T. Pattern Anal., 31(2), 319–336, 2009 http://dx.doi.org/10.1109/TPAMI.2008.5710.1109/TPAMI.2008.57Search in Google Scholar PubMed
[14] Lazarevic-McManus N., Renno J., Jones G. A., Performance evaluation in visual surveillance using the F-measure, In: The 4th ACM Int. Workshop on Video Surveillance and Sensor-Networks, 45–52, 2006 10.1145/1178782.1178790Search in Google Scholar
[15] List, T., Bins, J., Vazquez, J., Fisher, R.B., Performance evaluating the evaluator, In: Proc. 2nd Joint IEEE Int. Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, (VS-PETS), Beijing, 129–136, 2005 Search in Google Scholar
[16] Lei R., Xu L.-Q., Real-time outdoor video surveillance with robust foreground extraction and object tracking via multi-state transition management, Pattern Recogn. Lett., 27(15), 1816–1825, 2006 http://dx.doi.org/10.1016/j.patrec.2006.02.01710.1016/j.patrec.2006.02.017Search in Google Scholar
[17] Nawaz T., Cavallaro, A., PFT: A protocol for evaluating video trackers, IEEE Image Proc. 2011 10.1109/ICIP.2011.6116105Search in Google Scholar
[18] Schlögl T., Beleznai C., Winter M., Bischof H., Performance evaluation metrics for motion detection and tracking, In: The 17th Int. Conf. on Pattern Recognition, 4, 519–522, 2004 10.1109/ICPR.2004.1333825Search in Google Scholar
[19] Pan P., Porikli F., Schonfeld D., A new method for tracking performance evaluation based on a reflective model and perturbation analysis, IEEE ICASSP, 3529–3532, 2009 10.1109/ICASSP.2009.4960387Search in Google Scholar
[20] Perera A. G. A., Hooqs A., Srnivas C., Brooksby G., Wensheng H., Evaluation of algorithms for tracking multiple objects in video, In: The 35th IEEE Applied Imagery and pattern Recognition Workshop, 35–35, 2006 10.1109/AIPR.2006.23Search in Google Scholar
[21] Popoola J., Amer A., Performance evaluation for tracking algorithms using object labels, Int. Conf. Acoust. Spee., 733–736, 2008 10.1109/ICASSP.2008.4517714Search in Google Scholar
[22] Wackerly D.D., Mendenhall III W., Scheaffer R.L., Mathematical statistics with applications, 2nd edition, Duxbury Press, 2002 Search in Google Scholar
[23] Yilmaz A., Javed O., Shah M., Object tracking: a survey, ACM Comput. Surv. 38(4), 13, 2006 http://dx.doi.org/10.1145/1177352.117735510.1145/1177352.1177355Search in Google Scholar
[24] Zhang M., Chen K., Guo Y.Y., Online parameter based Kalman filter precision evaluation method for video tracking, In: IEEE Int. Conference on Multimedia Technology, 598–601, 2011 Search in Google Scholar
© 2011 Versita Warsaw
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.
