Polygon tracking feature with XMem tracker #7829

omerferhatt · 2024-04-30T09:19:43Z

Motivation and context

Polygon tracking is a crucial part of computer vision annotation tasks. It helps the required consecutive frame annotation, shortens the required annotation time, creates more precise annotation and moreover simplifies all HITL (Human in the loop) processes. Since the CVAT doesn't support this feature, it seems that adding this functionality with side benefits will be good for the sake of this project.

It aims to solve the below problems:

Using polygon tracking algorithms natively inside the CVAT to use it with semi-automatic annotations.
Tracking multiple polygon objects without losing the common track options (enable/disable, modify the last tracked object and continue from it, etc.)
Creating trackable polygon objects through interactor's (which allows to use SAM with any kind of polygon tracking algorithm)

According to the previous issues, it:

How it looks

initial_demo_video.webm

SAM runs on the CPU and XMem tracker runs on the GPU

Resource usage

For 5 object tracking on 1280x720 video will use approx. 2GB VRAM at peak.

Known issues and further optimization cases:

When maxWorker is more than 1 in function.yaml, the entire tracker works again in a duplicate manner. Since the problem is not solved, it is set to 1 for now.
Since tracking works on individual objects, it can be improved with batch inference.
As explained in step 2, this increases the request size since model states are kept separately for each input. Therefore, the process outside the model slows down as the number of objects tracked increases.

How has this been tested?

Tested and developed according to CVAT documentation

Checklist

I submit my changes into the develop branch
I have created a changelog fragment
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (see GitHub docs)
I have increased versions of npm packages if it is necessary
(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

Summary by CodeRabbit

New Features
- Added object type selection functionality to enhance user control over shape and polygon annotations.
- Introduced new serverless functions for advanced object segmentation using a PyTorch-based model with GPU support.
Enhancements
- Improved filtering logic to support both rectangles and polygons more effectively in the annotation interface.
Documentation
- Updated component documentation to reflect new functionalities and logic adjustments.
Performance
- Optimized image processing and object tracking within the serverless PyTorch model environment.

coderabbitai · 2024-04-30T09:19:49Z

Walkthrough

The recent updates involve integrating advanced object segmentation capabilities into the CVAT tool, enhancing its ability to handle complex video annotation tasks. This includes the addition of the XMem model for long-term video object segmentation and updates to the UI for better handling of object types during annotation. These changes aim to improve the efficiency and versatility of the annotation process.

Changes

File Path	Change Summary
`cvat-ui/.../tools-control.tsx`	Updated UI components for object type selection and handling in annotations.
`serverless/pytorch/omerferhatt/xmem/nuclio/...`	Introduced serverless configurations and handlers for the XMem model, enabling advanced video object segmentation.

Assessment against linked issues

Objective	Addressed	Explanation
Enhancement: XMem mask video propagation as CVAT AI Tools polygon tracker (#5465)	✅
Feature Idea: Incorporate "Segment Anything" (#5984)	❌	The changes do not mention integration or adaptation of the "Segment Anything" model.

Poem

🐰 Oh hop! A leap in tech so grand,
With shapes and masks at our command.
From frame to frame, the objects glide,
Segmented neatly, side by side.
A rabbit's touch in lines of code,
In CVAT, our efforts showed. 🌟
Let's celebrate, for progress made,
In AI's shade, our tools upgrade.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

…ased

coderabbitai

Actionable comments posted: 2

serverless/pytorch/omerferhatt/xmem/nuclio/main.py

serverless/pytorch/omerferhatt/xmem/nuclio/model_handler.py

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

PMazarovich · 2024-05-17T09:28:26Z

Hello, @bsekachev , do you think this MR will be reviewed/merged soon?
This sounds very helpful for annotators
@omerferhatt thanks!

bsekachev · 2024-05-22T11:53:51Z

I will try to review it on this week.

bsekachev · 2024-05-30T11:41:37Z

cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx

@@ -1358,6 +1387,7 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
                <Tabs type='card' tabBarGutter={8}>
                    <Tabs.TabPane key='interactors' tab='Interactors'>
                        {this.renderMasksConvertingBlock()}
+                        {convertMasksToPolygons ? this.renderObjectTypeBlock() : null}


xmem is a tracker, not interactor. The difference between them is exactly that interactors produce shapes on one frame, when tracker produces shapes on multiple frames (tracks).
Why are you modifying the block responsible for interactors?

bsekachev · 2024-05-30T11:45:15Z

cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx

+        const objectTypes = Object.values(ObjectType);
+        objectTypes.splice(objectTypes.indexOf(ObjectType.TAG), 1);


If you mean that this code supports only shape and track, you should write exactly this.
Otherwise, when we will add one more object type, this part will be automatically broken and nobody will know about it

bsekachev · 2024-05-30T11:46:58Z

serverless/pytorch/omerferhatt/xmem/nuclio/function-gpu.yaml

+        - kind: WORKDIR
+          value: /opt/nuclio
+        - kind: RUN
+          value: git clone --branch main https://github.com/omerferhatt/XMem xmem


Why not to use original repository?

Also it is better to use specific tag. main branch may be changed and new version will not compatible with old dependencies, for example.

bsekachev · 2024-05-30T11:48:51Z

serverless/pytorch/omerferhatt/xmem/nuclio/function-gpu.yaml

+          value: |-
+            pip install torch torchvision
+        - kind: RUN
+          value: wget 'https://www.dropbox.com/scl/fi/5m1l747p15qzgq023e0q9/xmem.pth?rlkey=ss2kjaq4qlvvk5juucyvtmrh8&dl=0' -O '/xmem.pth'


I do not think we should put the file to root /.
Better to put to current working directory.

I do not think dropbox is reliable enough.

Also it is not clear who is the weights owner, not clear weights license.
Is there original place?

bsekachev · 2024-05-30T11:50:07Z

serverless/pytorch/omerferhatt/xmem/nuclio/function-gpu.yaml

+      workerAvailabilityTimeoutMilliseconds: 10000
+      attributes:
+        # Set value from the calculation of tracking of 100 objects at the same time on a 4k image
+        maxRequestBodySize: 1073741824 # 1GB


Is there a reason to send 1 Gb body, within HTTP request?

bsekachev · 2024-05-30T11:53:47Z

serverless/pytorch/omerferhatt/xmem/nuclio/function-gpu.yaml

+          value: pip install opencv-python-headless jsonpickle
+        - kind: RUN
+          value: |-
+            pip install torch torchvision


Usually it is a good idea to fix dependencies version. Otherwise sometimes it will turn around, that the image cannot be build anymore.

bsekachev · 2024-06-19T05:47:33Z

Hello @omerferhatt

Do you have any plans regarding the pull request?

bsekachev · 2024-06-24T08:36:00Z

I will close the pull request now.
Please, do not hesitate to reopen if you are ready to finish it.

omerferhatt · 2024-07-02T08:37:39Z

I'm going to try to finish this week, thanks.

jtziebarth · 2024-07-30T20:42:50Z

@omerferhatt @bsekachev
I tried building CVAT with this feature and it works great overall.

However I notice that it takes a while to run inference on each frame (and in fact you can see it in the demo video omerferhatt posted above). Does anyone know what the source of this slow down is? When I run my own XMem inference script (adapted from the XMem author's example in their README), I can rip through inference on video at around 10 frames per second. With CVAT running on that same hardware, it takes several seconds to step through XMem tracking on a single frame. Is there something inherent in running XMem inference as a serverless function that causes the observed slowdown? And, if so, is there a way to speed it up? Otherwise, it will be very tedious to step through long videos frame by frame to track the objects. And this would be true of any model, not just XMem as used in this feature branch.

omerferhatt · 2024-07-30T20:57:21Z

@omerferhatt @bsekachev I tried building CVAT with this feature and it works great overall.

However I notice that it takes a while to run inference on each frame (and in fact you can see it in the demo video omerferhatt posted above). Does anyone know what the source of this slow down is? When I run my own XMem inference script (adapted from the XMem author's example in their README), I can rip through inference on video at around 10 frames per second. With CVAT running on that same hardware, it takes several seconds to step through XMem tracking on a single frame. Is there something inherent in running XMem inference as a serverless function that causes the observed slowdown? And, if so, is there a way to speed it up? Otherwise, it will be very tedious to step through long videos frame by frame to track the objects. And this would be true of any model, not just XMem as used in this feature branch.

Hi @jtziebarth ,
Thanks for the recap. While I was working on it, I noticed that the reason it was running slow was the request time when importing and exporting states, but I didn't have much time to look into it. I'm currently working on SAM 2 because it's better than SAM + XMem++, so I don't think I can get back to this topic quickly. Sorry about that.

jtziebarth · 2024-07-30T21:37:19Z

@omerferhatt thanks for such a quick reply. Can you clarify what you mean by "request time when importing and exporting states?" Where in the code are you referring to? If you can point me in the right direction I can try to take a look at it. Thanks.

omerferhatt added 5 commits January 18, 2024 08:29

XMem and polygon tracking

3eaa96f

General updates

c61403b

Merge branch 'develop' into feature/xmem

e6bf576

.patch file removed

62f4b4f

Revert docker-compose.dev.yml

da065eb

omerferhatt and others added 4 commits April 30, 2024 12:44

EOF fix

afc3f12

Base images updated, pip installation order fixed and body size incre…

c4e2fb6

…ased

Merge branch 'develop' into feature/xmem

8c1a668

Update launch.json to remove unused debug configuration

312e709

omerferhatt marked this pull request as ready for review April 30, 2024 12:45

omerferhatt requested a review from bsekachev as a code owner April 30, 2024 12:45

coderabbitai bot reviewed Apr 30, 2024

View reviewed changes

serverless/pytorch/omerferhatt/xmem/nuclio/main.py Outdated Show resolved Hide resolved

serverless/pytorch/omerferhatt/xmem/nuclio/model_handler.py Outdated Show resolved Hide resolved

omerferhatt and others added 2 commits April 30, 2024 15:53

Update serverless/pytorch/omerferhatt/xmem/nuclio/main.py

bb24cfa

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

Update serverless/pytorch/omerferhatt/xmem/nuclio/model_handler.py

df826f4

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

omerferhatt changed the title ~~[WiP] Polygon tracking feature with XMem tracker~~ Polygon tracking feature with XMem tracker Apr 30, 2024

omerferhatt and others added 2 commits April 30, 2024 16:24

Fix bug in ToolsControlComponent for tracking rectangles

46000b1

Merge branch 'develop' into feature/xmem

791ce59

bsekachev reviewed May 30, 2024

View reviewed changes

bsekachev closed this Jun 24, 2024

aashutoshpy mentioned this pull request Aug 6, 2024

[WIP] Polygon tracking feature with Cutie #8261

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Polygon tracking feature with XMem tracker #7829

Polygon tracking feature with XMem tracker #7829

omerferhatt commented Apr 30, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Apr 30, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

coderabbitai bot left a comment

PMazarovich commented May 17, 2024 •

edited

Loading

bsekachev commented May 22, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev commented Jun 19, 2024

bsekachev commented Jun 24, 2024

omerferhatt commented Jul 2, 2024

jtziebarth commented Jul 30, 2024

omerferhatt commented Jul 30, 2024

jtziebarth commented Jul 30, 2024

		const objectTypes = Object.values(ObjectType);
		objectTypes.splice(objectTypes.indexOf(ObjectType.TAG), 1);

Polygon tracking feature with XMem tracker #7829

Polygon tracking feature with XMem tracker #7829

Conversation

omerferhatt commented Apr 30, 2024 • edited by coderabbitai bot Loading

Motivation and context

How it looks

Resource usage

Known issues and further optimization cases:

How has this been tested?

Checklist

License

Summary by CodeRabbit

coderabbitai bot commented Apr 30, 2024 • edited Loading

Walkthrough

Changes

Assessment against linked issues

Poem

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

coderabbitai bot left a comment

Choose a reason for hiding this comment

PMazarovich commented May 17, 2024 • edited Loading

bsekachev commented May 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsekachev commented Jun 19, 2024

bsekachev commented Jun 24, 2024

omerferhatt commented Jul 2, 2024

jtziebarth commented Jul 30, 2024

omerferhatt commented Jul 30, 2024

jtziebarth commented Jul 30, 2024

omerferhatt commented Apr 30, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Apr 30, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)

PMazarovich commented May 17, 2024 •

edited

Loading