Skip to content

Commit d4129f2

Browse files
Maxim Zhiltsovbsekachev
Maxim Zhiltsov
andauthored
Add MOTS png format (#2198)
* Add mots format * fix upload * update docs * update changelog * Update datumaro dependency * fix header * update dm dependency * Support importing with outside property in mot and mots * fix track exporting Co-authored-by: Boris Sekachev <[email protected]>
1 parent d957d6a commit d4129f2

File tree

9 files changed

+261
-41
lines changed

9 files changed

+261
-41
lines changed

CHANGELOG.md

+4-2
Original file line numberDiff line numberDiff line change
@@ -27,16 +27,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2727
- Ability to prepare meta information manually (<https://github.com/openvinotoolkit/cvat/pull/2217>)
2828
- Ability to upload prepared meta information along with a video when creating a task (<https://github.com/openvinotoolkit/cvat/pull/2217>)
2929
- Optional chaining plugin for cvat-canvas and cvat-ui (<https://github.com/openvinotoolkit/cvat/pull/2249>)
30+
- MOTS png mask format support (<https://github.com/openvinotoolkit/cvat/pull/2198>)
3031

3132
### Changed
32-
3333
- UI models (like DEXTR) were redesigned to be more interactive (<https://github.com/opencv/cvat/pull/2054>)
3434
- Used Ubuntu:20.04 as a base image for CVAT Dockerfile (<https://github.com/opencv/cvat/pull/2101>)
3535
- Right colors of label tags in label mapping when a user runs automatic detection (<https://github.com/openvinotoolkit/cvat/pull/2162>)
3636
- Nuclio became an optional component of CVAT (<https://github.com/openvinotoolkit/cvat/pull/2192>)
3737
- A key to remove a point from a polyshape [Ctrl => Alt] (<https://github.com/openvinotoolkit/cvat/pull/2204>)
3838
- Updated `docker-compose` file version from `2.3` to `3.3`(<https://github.com/openvinotoolkit/cvat/pull/2235>)
3939
- Added auto inference of url schema from host in CLI, if provided (<https://github.com/openvinotoolkit/cvat/pull/2240>)
40+
- Track frames in skips between annotation is presented in MOT and MOTS formats are marked `outside` (<https://github.com/openvinotoolkit/cvat/pull/2198>)
4041

4142
### Deprecated
4243

@@ -47,7 +48,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4748
-
4849

4950
### Fixed
50-
5151
- Fixed multiple errors which arises when polygon is of length 5 or less (<https://github.com/opencv/cvat/pull/2100>)
5252
- Fixed task creation from PDF (<https://github.com/opencv/cvat/pull/2141>)
5353
- Fixed CVAT format import for frame stepped tasks (<https://github.com/openvinotoolkit/cvat/pull/2151>)
@@ -60,6 +60,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
6060
- Fixed use case when logs could be saved twice or more times #2202 (<https://github.com/openvinotoolkit/cvat/pull/2203>)
6161
- Fixed issues from #2112 (<https://github.com/openvinotoolkit/cvat/pull/2217>)
6262
- Git application name (renamed to dataset_repo) (<https://github.com/openvinotoolkit/cvat/pull/2243>)
63+
- A problem in exporting of tracks, where tracks could be truncated (<https://github.com/openvinotoolkit/cvat/issues/2129>)
64+
6365

6466
### Security
6567

cvat/apps/dataset_manager/annotation.py

+1-4
Original file line numberDiff line numberDiff line change
@@ -729,7 +729,6 @@ def interpolate(shape0, shape1):
729729
if track.get("interpolated_shapes"):
730730
return track["interpolated_shapes"]
731731

732-
# TODO: should be return an iterator?
733732
shapes = []
734733
curr_frame = track["shapes"][0]["frame"]
735734
prev_shape = {}
@@ -747,9 +746,7 @@ def interpolate(shape0, shape1):
747746
curr_frame = shape["frame"]
748747
prev_shape = shape
749748

750-
# TODO: Need to modify a client and a database (append "outside" shapes for polytracks)
751-
if not prev_shape["outside"] and (prev_shape["type"] == ShapeType.RECTANGLE
752-
or prev_shape["type"] == ShapeType.POINTS or prev_shape["type"] == ShapeType.CUBOID):
749+
if not prev_shape["outside"]:
753750
shape = copy(prev_shape)
754751
shape["frame"] = end_frame
755752
shapes.extend(interpolate(prev_shape, shape))

cvat/apps/dataset_manager/formats/README.md

+35-2
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
- [CVAT](#cvat)
1414
- [LabelMe](#labelme)
1515
- [MOT](#mot)
16+
- [MOTS](#mots)
1617
- [COCO](#coco)
1718
- [PASCAL VOC and mask](#voc)
1819
- [YOLO](#yolo)
@@ -708,8 +709,8 @@ Downloaded file: a zip archive of the following structure:
708709
``` bash
709710
taskname.zip/
710711
├── img1/
711-
| ├── imgage1.jpg
712-
| └── imgage2.jpg
712+
| ├── image1.jpg
713+
| └── image2.jpg
713714
└── gt/
714715
├── labels.txt
715716
└── gt.txt
@@ -742,6 +743,38 @@ taskname.zip/
742743

743744
- supported annotations: Rectangle tracks
744745

746+
### [MOTS PNG](https://www.vision.rwth-aachen.de/page/mots)<a id="mots" />
747+
748+
#### MOTS PNG Dumper
749+
750+
Downloaded file: a zip archive of the following structure:
751+
752+
``` bash
753+
taskname.zip/
754+
└── <any_subset_name>/
755+
| images/
756+
| ├── image1.jpg
757+
| └── image2.jpg
758+
└── instances/
759+
├── labels.txt
760+
├── image1.png
761+
└── image2.png
762+
763+
# labels.txt
764+
cat
765+
dog
766+
person
767+
...
768+
```
769+
770+
- supported annotations: Rectangle and Polygon tracks
771+
772+
#### MOTS PNG Loader
773+
774+
Uploaded file: a zip archive of the structure above
775+
776+
- supported annotations: Polygon tracks
777+
745778
### [LabelMe](http://labelme.csail.mit.edu/Release3.0)<a id="labelme" />
746779

747780
#### LabelMe Dumper

cvat/apps/dataset_manager/formats/mot.py

+14
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,20 @@ def _import(src_file, task_data):
7979
for track in tracks.values():
8080
# MOT annotations do not require frames to be ordered
8181
track.shapes.sort(key=lambda t: t.frame)
82+
83+
# insert outside=True in skips between the frames track is visible
84+
prev_shape_idx = 0
85+
prev_shape = track.shapes[0]
86+
for shape in track.shapes[1:]:
87+
has_skip = task_data.frame_step < shape.frame - prev_shape.frame
88+
if has_skip and not prev_shape.outside:
89+
prev_shape = prev_shape._replace(outside=True,
90+
frame=prev_shape.frame + task_data.frame_step)
91+
prev_shape_idx += 1
92+
track.shapes.insert(prev_shape_idx, prev_shape)
93+
prev_shape = shape
94+
prev_shape_idx += 1
95+
8296
# Append a shape with outside=True to finish the track
8397
last_shape = track.shapes[-1]
8498
if last_shape.frame + task_data.frame_step <= \
+102
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Copyright (C) 2019 Intel Corporation
2+
#
3+
# SPDX-License-Identifier: MIT
4+
5+
from tempfile import TemporaryDirectory
6+
7+
from pyunpack import Archive
8+
9+
from cvat.apps.dataset_manager.bindings import (CvatTaskDataExtractor,
10+
find_dataset_root, match_dm_item)
11+
from cvat.apps.dataset_manager.util import make_zip_archive
12+
from datumaro.components.extractor import AnnotationType, Transform
13+
from datumaro.components.project import Dataset
14+
15+
from .registry import dm_env, exporter, importer
16+
17+
18+
class KeepTracks(Transform):
19+
def transform_item(self, item):
20+
return item.wrap(annotations=[a for a in item.annotations
21+
if 'track_id' in a.attributes])
22+
23+
@exporter(name='MOTS PNG', ext='ZIP', version='1.0')
24+
def _export(dst_file, task_data, save_images=False):
25+
extractor = CvatTaskDataExtractor(task_data, include_images=save_images)
26+
envt = dm_env.transforms
27+
extractor = extractor.transform(KeepTracks) # can only export tracks
28+
extractor = extractor.transform(envt.get('polygons_to_masks'))
29+
extractor = extractor.transform(envt.get('boxes_to_masks'))
30+
extractor = extractor.transform(envt.get('merge_instance_segments'))
31+
extractor = Dataset.from_extractors(extractor) # apply lazy transforms
32+
with TemporaryDirectory() as temp_dir:
33+
dm_env.converters.get('mots_png').convert(extractor,
34+
save_dir=temp_dir, save_images=save_images)
35+
36+
make_zip_archive(temp_dir, dst_file)
37+
38+
@importer(name='MOTS PNG', ext='ZIP', version='1.0')
39+
def _import(src_file, task_data):
40+
with TemporaryDirectory() as tmp_dir:
41+
Archive(src_file.name).extractall(tmp_dir)
42+
43+
dataset = dm_env.make_importer('mots')(tmp_dir).make_dataset()
44+
masks_to_polygons = dm_env.transforms.get('masks_to_polygons')
45+
dataset = dataset.transform(masks_to_polygons)
46+
47+
tracks = {}
48+
label_cat = dataset.categories()[AnnotationType.label]
49+
50+
root_hint = find_dataset_root(dataset, task_data)
51+
52+
for item in dataset:
53+
frame_number = task_data.abs_frame_id(
54+
match_dm_item(item, task_data, root_hint=root_hint))
55+
56+
for ann in item.annotations:
57+
if ann.type != AnnotationType.polygon:
58+
continue
59+
60+
track_id = ann.attributes['track_id']
61+
shape = task_data.TrackedShape(
62+
type='polygon',
63+
points=ann.points,
64+
occluded=ann.attributes.get('occluded') == True,
65+
outside=False,
66+
keyframe=True,
67+
z_order=ann.z_order,
68+
frame=frame_number,
69+
attributes=[],
70+
source='manual',
71+
)
72+
73+
# build trajectories as lists of shapes in track dict
74+
if track_id not in tracks:
75+
tracks[track_id] = task_data.Track(
76+
label_cat.items[ann.label].name, 0, 'manual', [])
77+
tracks[track_id].shapes.append(shape)
78+
79+
for track in tracks.values():
80+
track.shapes.sort(key=lambda t: t.frame)
81+
82+
# insert outside=True in skips between the frames track is visible
83+
prev_shape_idx = 0
84+
prev_shape = track.shapes[0]
85+
for shape in track.shapes[1:]:
86+
has_skip = task_data.frame_step < shape.frame - prev_shape.frame
87+
if has_skip and not prev_shape.outside:
88+
prev_shape = prev_shape._replace(outside=True,
89+
frame=prev_shape.frame + task_data.frame_step)
90+
prev_shape_idx += 1
91+
track.shapes.insert(prev_shape_idx, prev_shape)
92+
prev_shape = shape
93+
prev_shape_idx += 1
94+
95+
# Append a shape with outside=True to finish the track
96+
last_shape = track.shapes[-1]
97+
if last_shape.frame + task_data.frame_step <= \
98+
int(task_data.meta['task']['stop_frame']):
99+
track.shapes.append(last_shape._replace(outside=True,
100+
frame=last_shape.frame + task_data.frame_step)
101+
)
102+
task_data.add_track(track)

cvat/apps/dataset_manager/formats/registry.py

+1
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ def make_exporter(name):
8787
import cvat.apps.dataset_manager.formats.labelme
8888
import cvat.apps.dataset_manager.formats.mask
8989
import cvat.apps.dataset_manager.formats.mot
90+
import cvat.apps.dataset_manager.formats.mots
9091
import cvat.apps.dataset_manager.formats.pascal_voc
9192
import cvat.apps.dataset_manager.formats.tfrecord
9293
import cvat.apps.dataset_manager.formats.yolo

cvat/apps/dataset_manager/tests/test_annotation.py

+47-20
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,17 @@
88

99

1010
class TrackManagerTest(TestCase):
11+
def _check_interpolation(self, track):
12+
interpolated = TrackManager.get_interpolated_shapes(track, 0, 7)
13+
14+
self.assertEqual(len(interpolated), 6)
15+
self.assertTrue(interpolated[0]["keyframe"])
16+
self.assertFalse(interpolated[1]["keyframe"])
17+
self.assertTrue(interpolated[2]["keyframe"])
18+
self.assertTrue(interpolated[3]["keyframe"])
19+
self.assertFalse(interpolated[4]["keyframe"])
20+
self.assertFalse(interpolated[5]["keyframe"])
21+
1122
def test_point_interpolation(self):
1223
track = {
1324
"frame": 0,
@@ -32,14 +43,18 @@ def test_point_interpolation(self):
3243
"occluded": False,
3344
"outside": True
3445
},
46+
{
47+
"frame": 4,
48+
"attributes": [],
49+
"points": [3.0, 4.0, 5.0, 6.0],
50+
"type": "points",
51+
"occluded": False,
52+
"outside": False
53+
},
3554
]
3655
}
3756

38-
interpolated = TrackManager.get_interpolated_shapes(track, 0, 2)
39-
40-
self.assertEqual(len(interpolated), 3)
41-
self.assertTrue(interpolated[0]["keyframe"])
42-
self.assertFalse(interpolated[1]["keyframe"])
57+
self._check_interpolation(track)
4358

4459
def test_polygon_interpolation(self):
4560
track = {
@@ -65,14 +80,18 @@ def test_polygon_interpolation(self):
6580
"occluded": False,
6681
"outside": True
6782
},
83+
{
84+
"frame": 4,
85+
"attributes": [],
86+
"points": [3.0, 4.0, 5.0, 6.0, 7.0, 6.0, 4.0, 5.0],
87+
"type": "polygon",
88+
"occluded": False,
89+
"outside": False
90+
},
6891
]
6992
}
7093

71-
interpolated = TrackManager.get_interpolated_shapes(track, 0, 2)
72-
73-
self.assertEqual(len(interpolated), 3)
74-
self.assertTrue(interpolated[0]["keyframe"])
75-
self.assertFalse(interpolated[1]["keyframe"])
94+
self._check_interpolation(track)
7695

7796
def test_bbox_interpolation(self):
7897
track = {
@@ -98,14 +117,18 @@ def test_bbox_interpolation(self):
98117
"occluded": False,
99118
"outside": True
100119
},
120+
{
121+
"frame": 4,
122+
"attributes": [],
123+
"points": [3.0, 4.0, 5.0, 6.0],
124+
"type": "rectangle",
125+
"occluded": False,
126+
"outside": False
127+
},
101128
]
102129
}
103130

104-
interpolated = TrackManager.get_interpolated_shapes(track, 0, 2)
105-
106-
self.assertEqual(len(interpolated), 3)
107-
self.assertTrue(interpolated[0]["keyframe"])
108-
self.assertFalse(interpolated[1]["keyframe"])
131+
self._check_interpolation(track)
109132

110133
def test_line_interpolation(self):
111134
track = {
@@ -131,11 +154,15 @@ def test_line_interpolation(self):
131154
"occluded": False,
132155
"outside": True
133156
},
157+
{
158+
"frame": 4,
159+
"attributes": [],
160+
"points": [3.0, 4.0, 5.0, 6.0],
161+
"type": "polyline",
162+
"occluded": False,
163+
"outside": False
164+
},
134165
]
135166
}
136167

137-
interpolated = TrackManager.get_interpolated_shapes(track, 0, 2)
138-
139-
self.assertEqual(len(interpolated), 3)
140-
self.assertTrue(interpolated[0]["keyframe"])
141-
self.assertFalse(interpolated[1]["keyframe"])
168+
self._check_interpolation(track)

cvat/apps/dataset_manager/tests/test_formats.py

+3
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,7 @@ def test_export_formats_query(self):
267267
'Datumaro 1.0',
268268
'LabelMe 3.0',
269269
'MOT 1.1',
270+
'MOTS PNG 1.0',
270271
'PASCAL VOC 1.1',
271272
'Segmentation mask 1.1',
272273
'TFRecord 1.0',
@@ -282,6 +283,7 @@ def test_import_formats_query(self):
282283
'CVAT 1.1',
283284
'LabelMe 3.0',
284285
'MOT 1.1',
286+
'MOTS PNG 1.0',
285287
'PASCAL VOC 1.1',
286288
'Segmentation mask 1.1',
287289
'TFRecord 1.0',
@@ -316,6 +318,7 @@ def test_empty_images_are_exported(self):
316318
('Datumaro 1.0', 'datumaro_project'),
317319
('LabelMe 3.0', 'label_me'),
318320
# ('MOT 1.1', 'mot_seq'), # does not support
321+
# ('MOTS PNG 1.0', 'mots_png'), # does not support
319322
('PASCAL VOC 1.1', 'voc'),
320323
('Segmentation mask 1.1', 'voc'),
321324
('TFRecord 1.0', 'tf_detection_api'),

0 commit comments

Comments
 (0)