This project implements a real-time object detection pipeline using a YOLO v8 model. A Rust-based gRPC service (using Tonic and Ort) handles inference, while an Axum-based middleware streams video from a webcam, collects predictions, and serves them via WebSocket. Additionally, a simple HTML page is provided for live visualization of the detections, and a predict_image POST endpoint allows direct inference on JPEG images.
- Tonic, Ort for running YOLO model inference via gRPC using ONNX runtime
- Axum, Tonic for handling video streaming and client Http requests
- OpenCV for processing video frames in the middleware
- HTML + JavaScript for real-time visualization
To run the project, simply execute the following command at the root directory:
make allThis will download the necessary images, runs the service and automatically open a webpage for visualization.
If you have set up CUDA correctly, you can run the project with GPU acceleration by executing:
make all-cuda
The system is designed for real-time processing using parallel inference sessions to maximize throughput.
-
TensorRT: The time taken to generate predictions for a single frame is typically around 42ms (median) and 90ms (99th percentile). This means the detections you see correspond to the video frame from approximately 42/90ms prior.
-
CPU: The time taken to generate predictions for a single frame is typically around 250ms (median) and 450ms (99th percentile). This means the detections you see correspond to the video frame from approximately 250/450ms prior.
This, of course, depends on the hardware you run and the results obtained here are on a RTX2060 with 8gb of VRAM for the TensorRT part and an Intel® Core™ i5-9600K × 6 for the CPU part.
Currently, the project only works on Linux as it relies on mounting the /dev/video1 device.
If you are using a different video input, you may need to modify the source code accordingly.
This project utilizes the YOLOv8m model, which is distributed under the AGPL-3.0 license. Please refer to the Ultralytics website for the full license details.
This project is intended for educational and demonstration purposes. Its licensing is subject to the licensing terms of the YOLOv8m model.
Disclaimer: Users are responsible for ensuring their compliance with the YOLOv8m model's license.