Real-Time Detection of Object

Missing and New Object Placement

in Video

Ayush Jha

April 28, 2025

Real-Time Object Detection

Contents

1 Executive Summary 2

2 System Architecture 2
2.1 Core Components.....................................................................................................................2
2.2 Detection Pipeline....................................................................................................................2

3 Performance Metrics 2
3.1 FPS Achievement..................................................................................................................2
3.2 Detection Accuracy..................................................................................................................2

4 Hardware Configuration 3

5 Implementation Techniques & Optimizations 3

5.1 Model Selection & Optimization.............................................................................................3
5.2 Processing Optimizations.........................................................................................................3
5.2.1 Dual Resolution Processing........................................................................................3
5.2.2 Pre-allocation of Tensors............................................................................................3
5.2.3 Asynchronous Video Writing.....................................................................................3
5.2.4 Frame Skipping Option.............................................................................................3
5.3 Tracking Algorithm Enhancements........................................................................................3
5.3.1 State Management......................................................................................................3
5.3.2 IoU-based Matching....................................................................................................4
5.3.3 Track History..............................................................................................................4

6 Output Visualization 4
6.1 Color-coded bounding boxes..................................................................................................4
6.2 Text labels................................................................................................................................4
6.3 Real-time FPS counter..........................................................................................................4

7 Sample Output Frames 5

8 Challenges & Solutions 6

8.1 Challenge 1: Balancing Speed vs. Accuracy.........................................................................6
8.2 Challenge 2: False Positives in Object State Changes........................................................6
8.3 Challenge 3: CPU/GPU Memory Management................................................................6
8.4 Challenge 4: Video Writing Performance Impact................................................................6

9 Additional Features 6
9.1 Command-line Interface..........................................................................................................6
9.2 Performance Statistics.............................................................................................................6
9.3 Scalable Processing..................................................................................................................6

10 Future Improvements 7

11 Conclusion 7

Page 1
 Real-Time Object Detection

1 Executive Summary
This report details the implementation of a real-time video analytics system capable of detecting
both missing objects and new object placement in video streams. The system achieves high
performance on modest hardware using YOLOv8 for object detection coupled with a custom
ByteTrack-inspired algorithm for object tracking. The implementation focuses on optimizing
both detection accuracy and processing speed to enable real-time analysis.

2 System Architecture
2.1 Core Components
1. Object Detection: YOLOv8n model from Ultralytics
2. Object Tracking: Custom ByteTrack-inspired implementation
3. State Management: Track history monitoring for object state changes
4. Asynchronous Processing: Threading for video writing operations

2.2 Detection Pipeline

The system follows this workflow:
1. Video frame acquisition
2. Preprocessing and resizing
3. YOLOv8 inference
4. ByteTrack algorithm for tracking and state determination
5. Classification of objects as new, tracked, or missing
6. Visualization and output generation

3 Performance Metrics
3.1 FPS Achievement
The system achieved the following performance metrics:

• Average FPS: 42.7

• Minimum FPS: 36.2
• Maximum FPS: 46.5
• Target FPS achieved: Yes (target: 35+)

3.2 Detection Accuracy

The system successfully detects:
• New objects appearing in the scene (highlighted in green)
• Tracked objects (highlighted in yellow)
• Missing/lost objects (highlighted in red for up to 15 frames)

Page 2
 Real-Time Object Detection

4 Hardware Configuration
Testing was conducted on the following hardware:
• CPU: Intel Core i5 11th Generation
• RAM: 8GB
• GPU: NVIDIA GeForce GTX 1650 (4GB VRAM)
• CUDA: Version 11.6

5 Implementation Techniques & Optimizations

5.1 Model Selection & Optimization
• YOLOv8n: Selected for its balance of speed and accuracy
• Half-precision (FP16): Reduced memory usage and improved inference speed
• TensorRT acceleration: Attempted implementation for additional speed boost
• CUDA optimization: Utilized CUDA 11.6 with cudnn benchmarking enabled

5.2 Processing Optimizations

5.2.1 Dual Resolution Processing
• Input Resolution: 640x480
• Processing Resolution: 384x384
• This approach balances visual quality with processing speed

5.2.2 Pre-allocation of Tensors

• Pre-allocated CUDA tensors to reduce memory allocation overhead during inference

5.2.3 Asynchronous Video Writing

• Implementation of threaded video writer to prevent FPS drops during disk operations

5.2.4 Frame Skipping Option

• Configurable frame skipping to further increase FPS when needed

5.3 Tracking Algorithm Enhancements

The custom ByteTracker implementation includes:

5.3.1 State Management

• New objects (first detected)
• Tracked objects (consistently detected)
• Lost objects (recently disappeared)
• Removed objects (missing for extended period)

Page 3
 Real-Time Object Detection

5.3.2 IoU-based Matching

• Greedy matching algorithm optimized for speed

• Configurable IoU threshold (default: 0.3)

5.3.3 Track History

• Maintained history of each object’s state

• Used for visualization and state determination

6 Output Visualization
The system visualizes detection results with:

6.1 Color-coded bounding boxes

• Green: Newly detected objects

• Yellow: Continuously tracked objects

• Red: Missing/lost objects

6.2 Text labels

Showing:

• Object class

• Track ID

• State indicator (”NEW” or ”MISSING”)

6.3 Real-time FPS counter

With color indicator:

• Green: ≥37 FPS

• Orange: 30-36 FPS

• Red: <30 FPS

Page 4
Real-Time Object Detection

7 Sample Output Frames

Figure 1: System detecting a new object (green bounding box)

Figure 2: System identifying a missing object (red bounding box)

Page 5
Real-Time Object Detection

Figure 3: System tracking multiple objects simultaneously

8 Challenges & Solutions

8.1 Challenge 1: Balancing Speed vs. Accuracy
Solution: Implemented dual-resolution approach and half-precision inference

8.2 Challenge 2: False Positives in Object State Changes

Solution: Implemented minimum hits (3) before confirming an object as tracked

8.3 Challenge 3: CPU/GPU Memory Management

Solution: Used tensor pre-allocation and CUDA optimization techniques

8.4 Challenge 4: Video Writing Performance Impact

Solution: Implemented asynchronous video writing in a separate thread

9 Additional Features
9.1 Command-line Interface
• Extensive configuration options for input source, model, resolution, etc.

• Runtime adjustments for confidence threshold and processing parameters

9.2 Performance Statistics

• Detailed performance metrics displayed at completion

• Real-time FPS monitoring

9.3 Scalable Processing

• Configuration options to adapt to different hardware capabilities

Page 6
 Real-Time Object Detection

10 Future Improvements
1. Model Quantization: Further optimize model size and inference speed

2. Multi-scene Support: Enhance algorithm to handle scene changes

3. Improved State Classification: Add more sophisticated state determination logic

4. UI Enhancement: Develop a graphical interface for non-technical users

5. External API Integration: Enable cloud-based analytics and notifications

11 Conclusion
The implemented system successfully meets the requirements for real-time detection of missing
objects and new object placement in video. With an average FPS of 42.7 on modest hardware,
the system demonstrates that high-performance real-time object tracking is achievable with
carefully optimized algorithms and appropriate hardware utilization. The ByteTrack-inspired
implementation provides robust tracking performance while maintaining the speed required for
real-time applications.

Note: This implementation does not include Docker containerization as specified in the original
requirements.

Page 7