Is it possible to use SAM2 for real-time video applications?

[giulio333]

I am interested in using SAM2 for real-time video processing applications. Could you please let me know if this is possible with the current implementation?

[thad75]

I have run SAM2 tiny, small on a100 without any issue for real time application (25 fps +). Make sure flash-attention is installed though

[giulio333]

Thanks, what is flash-attention?

[rishabh297]

I have run SAM2 tiny, small on a100 without any issue for real time application (25 fps +). Make sure flash-attention is installed though

Hey, @thad75 can you please share how you were able to build SAM2 on a real-time application? Is there any Github that you have for this project?

[ipmLessing]

I have run SAM2 tiny, small on a100 without any issue for real time application (25 fps +). Make sure flash-attention is installed though

Are you using the video predictor or just a single image predictor? I'm also looking for the way to make the sam2 in real 'real-time' with small input sample and track the stuff that I want in the live video (images) stream.

[Gy920]

I have implemented a demo of building SAM2 on a real-time application, which can be found at https://github.com/Gy920/segment-anything-2-real-time. However, there are still some issues (such as only being able to set the object to be segmented in the first frame).

I have run SAM2 tiny, small on a100 without any issue for real time application (25 fps +). Make sure flash-attention is installed though

Are you using the video predictor or just a single image predictor? I'm also looking for the way to make the sam2 in real 'real-time' with small input sample and track the stuff that I want in the live video (images) stream.

[pingchesu]

@Gy920 Thanks for implementing Sam2CameraPredictor, I have a question is that I will dynamicly add objects points in each frame, and use Sam2CameraPredictor to track each object that I added. Is that possible for Sam2CameraPredictor to meet this requirement?

[Gy920]

@Gy920 Thanks for implementing Sam2CameraPredictor, I have a question is that I will dynamicly add objects points in each frame, and use Sam2CameraPredictor to track each object that I added. Is that possible for Sam2CameraPredictor to meet this requirement?

You need to reset Sam2CameraPredictor when adding new objects, and then re-add your object points in that frame. This may cause some performance overhead.

[sachinsshetty]

https://github.com/IDEA-Research/Grounded-SAM-2
Can be expanded to use real-time.

https://github.com/IDEA-Research/Grounded-SAM-2/blob/main/grounded_sam2_tracking_demo.py

Really cool work by @IDEA-Research

I making some experiments to use it for real time video at

https://github.com/sachinsshetty/segment-anything-2/tree/experiments

its a WIP,

[mano3-1]

You need to reset Sam2CameraPredictor when adding new objects, and then re-add your object points in that frame. This may cause some performance overhead.

Thanks for the implementation @Gy920 . I've been working on tracking object using your code. But I noticed a drop in performance, as in when I use SAM's video predictor notebooks, the segmentations are accurate..but when I compare it with yours, there is a drop in performance..I would like to know if there is a logic change from original SAM's implementation and yours. Attaching the videos that I obtained from using video predictor notebook and camera sam also.

Video Predictor: here

Camera predictor: here

As you may see, the results from video predictor are consistent than camera predictor.
Note: the prompt points are same for both videos.

[Gy920]

You need to reset Sam2CameraPredictor when adding new objects, and then re-add your object points in that frame. This may cause some performance overhead.

Thanks for the implementation @Gy920 . I've been working on tracking object using your code. But I noticed a drop in performance, as in when I use SAM's video predictor notebooks, the segmentations are accurate..but when I compare it with yours, there is a drop in performance..I would like to know if there is a logic change from original SAM's implementation and yours. Attaching the videos that I obtained from using video predictor notebook and camera sam also.

Video Predictor: here

Camera predictor: here

As you may see, the results from video predictor are consistent than camera predictor. Note: the prompt points are same for both videos.

I noticed the colors are different between the two videos (R and B channels are swapped), as mentioned in this issue. The unsatisfactory result might be caused by this bug, which I'll fix later.

[mano3-1]

I have converted the frames from BGR to RGB before feeding them to predictor.load_first_frame or predictor.track. Still the results are same.

[Gy920]

I have converted the frames from BGR to RGB before feeding them to predictor.load_first_frame or predictor.track. Still the results are same.

I suspect it might be related to the conditions in part. I simplified the logic in this part during implementation, which could potentially lead to a decrease in tracking performance.

I'm a bit swamped at the moment, but I'll carve out some time to dig into this issue and get it sorted as soon as possible.

[lihaowei1999]

@Gy920 Hi. I think I found the problem... Do not know if it is the same case on your side but it definitely worth a try.
Add

cv2.imwrite("Temp/temp.jpg", img)
img = cv2.imread("Temp/temp.jpg")

to the front of perpare_data and the segmentation will be much more stable.
Or simply apply:

img = cv2.imdecode(np.frombuffer(cv2.imencode('.jpg', img)[1], np.uint8), cv2.IMREAD_COLOR)

will be the same.

I think SAM2 is sensitive to the noise generated by jpg. If you change to png, the result will be not be as good.