Skip to content

Commit 8caa169

Browse files
authored
Replace mask format support with Datumaro (#1163)
* Add box to mask transform * Fix 'source' labelmap mode in voc converter * Import groups * Replace mask format support * Update mask format documentation * codacy * Fix tests * Fix dataset * Fix segments grouping * Merge instances in mask export
1 parent 80d3f97 commit 8caa169

File tree

14 files changed

+301
-160
lines changed

14 files changed

+301
-160
lines changed

README.md

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -38,18 +38,17 @@ Format selection is possible after clicking on the Upload annotation / Dump anno
3838
[Datumaro](datumaro/README.md) dataset framework allows additional dataset transformations
3939
via its command line tool.
4040

41-
| Annotation format | Dumper | Loader |
42-
| ---------------------------------------------------------------------------------- | ------ | ------ |
43-
| [CVAT XML v1.1 for images](cvat/apps/documentation/xml_format.md#annotation) | X | X |
44-
| [CVAT XML v1.1 for a video](cvat/apps/documentation/xml_format.md#interpolation) | X | X |
45-
| [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) | X | X |
46-
| [YOLO](https://pjreddie.com/darknet/yolo/) | X | X |
47-
| [MS COCO Object Detection](http://cocodataset.org/#format-data) | X | X |
48-
| PNG mask | X | |
49-
| PNG instance mask | X | |
50-
| [TFrecord](https://www.tensorflow.org/tutorials/load_data/tf_records) | X | X |
51-
| [MOT](https://motchallenge.net/) | X | X |
52-
| [LabelMe](http://labelme.csail.mit.edu/Release3.0) | X | X |
41+
| Annotation format | Dumper | Loader |
42+
| ------------------------------------------------------------------------------------------ | ------ | ------ |
43+
| [CVAT XML v1.1 for images](cvat/apps/documentation/xml_format.md#annotation) | X | X |
44+
| [CVAT XML v1.1 for a video](cvat/apps/documentation/xml_format.md#interpolation) | X | X |
45+
| [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) | X | X |
46+
| [YOLO](https://pjreddie.com/darknet/yolo/) | X | X |
47+
| [MS COCO Object Detection](http://cocodataset.org/#format-data) | X | X |
48+
| PNG class mask + instance mask as in [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) | X | X |
49+
| [TFrecord](https://www.tensorflow.org/tutorials/load_data/tf_records) | X | X |
50+
| [MOT](https://motchallenge.net/) | X | X |
51+
| [LabelMe](http://labelme.csail.mit.edu/Release3.0) | X | X |
5352

5453
## Links
5554
- [Intel AI blog: New Computer Vision Tool Accelerates Annotation of Digital Images and Video](https://www.intel.ai/introducing-cvat)

cvat/apps/annotation/README.md

Lines changed: 37 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -506,18 +506,48 @@ python create_pascal_tf_record.py --data_dir <path to VOCdevkit> --set train --y
506506
- downloaded file: a zip archive with the following structure:
507507
```bash
508508
taskname.zip
509-
├── frame_000001.png
510-
├── frame_000002.png
511-
├── frame_000003.png
512-
├── ...
513-
└── colormap.txt
509+
├── labelmap.txt # optional, required for non-VOC labels
510+
├── ImageSets/
511+
│   └── Segmentation/
512+
│   └── default.txt # list of image names without extension
513+
├── SegmentationClass/ # merged class masks
514+
│   └── image1.png
515+
│   └── image2.png
516+
└── SegmentationObject/ # merged instance masks
517+
└── image1.png
518+
└── image2.png
514519
```
515520
Mask is a png image with several (RGB) channels where each pixel has own color which corresponds to a label.
516521
Color generation correspond to the Pascal VOC color generation
517522
[algorithm](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/htmldoc/devkit_doc.html#sec:voclabelcolormap).
518523
(0, 0, 0) is used for background.
519-
`colormap.txt` file contains the values of the used colors in RGB format.
524+
`labelmap.txt` file contains the values of the used colors in RGB format. The file structure:
525+
```bash
526+
# label:color_rgb:parts:actions
527+
background:0,128,0::
528+
aeroplane:10,10,128::
529+
bicycle:10,128,0::
530+
bird:0,108,128::
531+
boat:108,0,100::
532+
bottle:18,0,8::
533+
bus:12,28,0::
534+
```
520535
- supported shapes - Rectangles, Polygons
521536

522537
#### Mask loader description
523-
Not supported
538+
- uploaded file: a zip archive of the following structure:
539+
```bash
540+
name.zip
541+
├── labelmap.txt # optional, required for non-VOC labels
542+
├── ImageSets/
543+
│   └── Segmentation/
544+
│   └── <any_subset_name>.txt
545+
├── SegmentationClass/
546+
│   └── image1.png
547+
│   └── image2.png
548+
└── SegmentationObject/
549+
└── image.png
550+
└── image2.png
551+
```
552+
- supported shapes: Polygons
553+
- additional comments: the CVAT task should be created with the full label set that may be in the annotation files

cvat/apps/annotation/mask.py

Lines changed: 48 additions & 118 deletions
Original file line numberDiff line numberDiff line change
@@ -6,130 +6,60 @@
66
"name": "MASK",
77
"dumpers": [
88
{
9-
"display_name": "{name} (by class) {format} {version}",
9+
"display_name": "{name} {format} {version}",
1010
"format": "ZIP",
11-
"version": "1.0",
12-
"handler": "dump_by_class"
11+
"version": "1.1",
12+
"handler": "dump",
1313
},
14+
],
15+
"loaders": [
1416
{
15-
"display_name": "{name} (by instance) {format} {version}",
17+
"display_name": "{name} {format} {version}",
1618
"format": "ZIP",
17-
"version": "1.0",
18-
"handler": "dump_by_instance"
19+
"version": "1.1",
20+
"handler": "load",
1921
},
2022
],
21-
"loaders": [
22-
],
2323
}
2424

25-
MASK_BY_CLASS = 0
26-
MASK_BY_INSTANCE = 1
27-
28-
def convert_box_to_polygon(shape):
29-
xtl = shape.points[0]
30-
ytl = shape.points[1]
31-
xbr = shape.points[2]
32-
ybr = shape.points[3]
33-
34-
return [xtl, ytl, xbr, ytl, xbr, ybr, xtl, ybr]
35-
36-
def create_mask_colorizer(annotations, colorize_type):
37-
import numpy as np
38-
from collections import OrderedDict
39-
40-
class MaskColorizer:
41-
42-
def __init__(self, annotations, colorize_type):
43-
44-
if colorize_type == MASK_BY_CLASS:
45-
self.colors = self.gen_class_mask_colors(annotations)
46-
elif colorize_type == MASK_BY_INSTANCE:
47-
self.colors = self.gen_instance_mask_colors()
48-
49-
def generate_pascal_colormap(self, size=256):
50-
# RGB format, (0, 0, 0) used for background
51-
colormap = np.zeros((size, 3), dtype=int)
52-
ind = np.arange(size, dtype=int)
53-
54-
for shift in reversed(range(8)):
55-
for channel in range(3):
56-
colormap[:, channel] |= ((ind >> channel) & 1) << shift
57-
ind >>= 3
58-
59-
return colormap
60-
61-
def gen_class_mask_colors(self, annotations):
62-
colormap = self.generate_pascal_colormap()
63-
labels = [label[1]["name"] for label in annotations.meta["task"]["labels"] if label[1]["name"] != 'background']
64-
labels.insert(0, 'background')
65-
label_colors = OrderedDict((label, colormap[idx]) for idx, label in enumerate(labels))
66-
67-
return label_colors
68-
69-
def gen_instance_mask_colors(self):
70-
colormap = self.generate_pascal_colormap()
71-
# The first color is black
72-
instance_colors = OrderedDict((idx, colormap[idx]) for idx in range(len(colormap)))
73-
74-
return instance_colors
75-
76-
return MaskColorizer(annotations, colorize_type)
77-
78-
def dump(file_object, annotations, colorize_type):
79-
80-
from zipfile import ZipFile, ZIP_STORED
81-
import numpy as np
82-
import os
83-
from pycocotools import mask as maskUtils
84-
import matplotlib.image
85-
import io
86-
87-
colorizer = create_mask_colorizer(annotations, colorize_type=colorize_type)
88-
if colorize_type == MASK_BY_CLASS:
89-
save_dir = "SegmentationClass"
90-
elif colorize_type == MASK_BY_INSTANCE:
91-
save_dir = "SegmentationObject"
92-
93-
with ZipFile(file_object, "w", ZIP_STORED) as output_zip:
94-
for frame_annotation in annotations.group_by_frame():
95-
image_name = frame_annotation.name
96-
annotation_name = "{}.png".format(os.path.splitext(os.path.basename(image_name))[0])
97-
width = frame_annotation.width
98-
height = frame_annotation.height
99-
100-
shapes = frame_annotation.labeled_shapes
101-
# convert to mask only rectangles and polygons
102-
shapes = [shape for shape in shapes if shape.type == 'rectangle' or shape.type == 'polygon']
103-
if not shapes:
104-
continue
105-
shapes = sorted(shapes, key=lambda x: int(x.z_order))
106-
img_mask = np.zeros((height, width, 3))
107-
buf_mask = io.BytesIO()
108-
for shape_index, shape in enumerate(shapes):
109-
points = shape.points if shape.type != 'rectangle' else convert_box_to_polygon(shape)
110-
rles = maskUtils.frPyObjects([points], height, width)
111-
rle = maskUtils.merge(rles)
112-
mask = maskUtils.decode(rle)
113-
idx = (mask > 0)
114-
# get corresponding color
115-
if colorize_type == MASK_BY_CLASS:
116-
color = colorizer.colors[shape.label] / 255
117-
elif colorize_type == MASK_BY_INSTANCE:
118-
color = colorizer.colors[shape_index+1] / 255
119-
120-
img_mask[idx] = color
121-
122-
# write mask
123-
matplotlib.image.imsave(buf_mask, img_mask, format='png')
124-
output_zip.writestr(os.path.join(save_dir, annotation_name), buf_mask.getvalue())
125-
# Store color map for each class
126-
labels = '\n'.join('{}:{}'.format(label, ','.join(str(i) for i in color)) for label, color in colorizer.colors.items())
127-
output_zip.writestr('colormap.txt', labels)
128-
129-
def dump_by_class(file_object, annotations):
130-
131-
return dump(file_object, annotations, MASK_BY_CLASS)
132-
133-
def dump_by_instance(file_object, annotations):
13425

135-
return dump(file_object, annotations, MASK_BY_INSTANCE)
26+
def dump(file_object, annotations):
27+
from cvat.apps.dataset_manager.bindings import CvatAnnotationsExtractor
28+
from cvat.apps.dataset_manager.util import make_zip_archive
29+
from datumaro.components.project import Environment, Dataset
30+
from tempfile import TemporaryDirectory
31+
32+
env = Environment()
33+
polygons_to_masks = env.transforms.get('polygons_to_masks')
34+
boxes_to_masks = env.transforms.get('boxes_to_masks')
35+
merge_instance_segments = env.transforms.get('merge_instance_segments')
36+
id_from_image = env.transforms.get('id_from_image_name')
37+
38+
extractor = CvatAnnotationsExtractor('', annotations)
39+
extractor = extractor.transform(polygons_to_masks)
40+
extractor = extractor.transform(boxes_to_masks)
41+
extractor = extractor.transform(merge_instance_segments)
42+
extractor = extractor.transform(id_from_image)
43+
extractor = Dataset.from_extractors(extractor) # apply lazy transforms
44+
converter = env.make_converter('voc_segmentation',
45+
apply_colormap=True, label_map='source')
46+
with TemporaryDirectory() as temp_dir:
47+
converter(extractor, save_dir=temp_dir)
48+
make_zip_archive(temp_dir, file_object)
49+
50+
def load(file_object, annotations):
51+
from pyunpack import Archive
52+
from tempfile import TemporaryDirectory
53+
from datumaro.plugins.voc_format.importer import VocImporter
54+
from datumaro.components.project import Environment
55+
from cvat.apps.dataset_manager.bindings import import_dm_annotations
56+
57+
archive_file = file_object if isinstance(file_object, str) else getattr(file_object, "name")
58+
with TemporaryDirectory() as tmp_dir:
59+
Archive(archive_file).extractall(tmp_dir)
60+
61+
dm_project = VocImporter()(tmp_dir)
62+
dm_dataset = dm_project.make_dataset()
63+
masks_to_polygons = Environment().transforms.get('masks_to_polygons')
64+
dm_dataset = dm_dataset.transform(masks_to_polygons)
65+
import_dm_annotations(dm_dataset, annotations)

cvat/apps/dataset_manager/bindings.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,22 @@ def import_dm_annotations(dm_dataset, cvat_task_anno):
211211
for item in dm_dataset:
212212
frame_number = match_frame(item, cvat_task_anno)
213213

214+
# do not store one-item groups
215+
group_map = { 0: 0 }
216+
group_size = { 0: 0 }
217+
for ann in item.annotations:
218+
if ann.type in shapes:
219+
group = group_map.get(ann.group)
220+
if group is None:
221+
group = len(group_map)
222+
group_map[ann.group] = group
223+
group_size[ann.group] = 1
224+
else:
225+
group_size[ann.group] += 1
226+
group_map = {g: s for g, s in group_size.items()
227+
if 1 < s and group_map[g]}
228+
group_map = {g: i for i, g in enumerate([0] + sorted(group_map))}
229+
214230
for ann in item.annotations:
215231
if ann.type in shapes:
216232
cvat_task_anno.add_shape(cvat_task_anno.LabeledShape(
@@ -219,5 +235,6 @@ def import_dm_annotations(dm_dataset, cvat_task_anno):
219235
label=label_cat.items[ann.label].name,
220236
points=ann.points,
221237
occluded=False,
238+
group=group_map.get(ann.group, 0),
222239
attributes=[],
223240
))

cvat/apps/engine/tests/test_rest_api.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2658,9 +2658,9 @@ def _get_initial_annotation(annotation_format):
26582658
elif annotation_format == "COCO JSON 1.0":
26592659
annotations["shapes"] = polygon_shapes_wo_attrs
26602660

2661-
elif annotation_format == "MASK ZIP 1.0":
2662-
annotations["shapes"] = rectangle_shapes_with_attrs + rectangle_shapes_wo_attrs + polygon_shapes_wo_attrs
2663-
annotations["tracks"] = rectangle_tracks_with_attrs + rectangle_tracks_wo_attrs
2661+
elif annotation_format == "MASK ZIP 1.1":
2662+
annotations["shapes"] = rectangle_shapes_wo_attrs + polygon_shapes_wo_attrs
2663+
annotations["tracks"] = rectangle_tracks_wo_attrs
26642664

26652665
elif annotation_format == "MOT CSV 1.0":
26662666
annotations["tracks"] = rectangle_tracks_wo_attrs
@@ -2730,6 +2730,8 @@ def _get_initial_annotation(annotation_format):
27302730
}
27312731

27322732
for loader in annotation_format["loaders"]:
2733+
if loader["display_name"] == "MASK ZIP 1.1":
2734+
continue # can't really predict the result and check
27332735
response = self._upload_api_v1_tasks_id_annotations(task["id"], annotator, uploaded_data, "format={}".format(loader["display_name"]))
27342736
self.assertEqual(response.status_code, HTTP_202_ACCEPTED)
27352737

datumaro/datumaro/components/project.py

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -321,17 +321,17 @@ def from_extractors(cls, *sources):
321321
subsets = defaultdict(lambda: Subset(dataset))
322322
for source in sources:
323323
for item in source:
324-
path = None # NOTE: merge everything into our own dataset
325-
326324
existing_item = subsets[item.subset].items.get(item.id)
327325
if existing_item is not None:
328-
item = self._merge_items(existing_item, item, path=path)
329-
else:
330-
item = item.wrap(path=path, annotations=item.annotations)
326+
path = existing_item.path
327+
if item.path != path:
328+
path = None
329+
item = cls._merge_items(existing_item, item, path=path)
331330

332331
subsets[item.subset].items[item.id] = item
333332

334-
self._subsets = dict(subsets)
333+
dataset._subsets = dict(subsets)
334+
return dataset
335335

336336
def __init__(self, categories=None):
337337
super().__init__()
@@ -419,7 +419,7 @@ def _merge_items(cls, existing_item, current_item, path=None):
419419
image._path = current_item.image.path
420420

421421
if all([existing_item.image._size, current_item.image._size]):
422-
assert existing_item.image._size == current_item.image._size, "Image info differs for item '%s'" % item.id
422+
assert existing_item.image._size == current_item.image._size, "Image info differs for item '%s'" % existing_item.id
423423
elif existing_item.image._size:
424424
image._size = existing_item.image._size
425425
else:

datumaro/datumaro/plugins/coco_format/converter.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -361,6 +361,7 @@ def save_annotations(self, item):
361361

362362
@classmethod
363363
def find_solitary_points(cls, annotations):
364+
annotations = sorted(annotations, key=lambda a: a.group)
364365
solitary_points = []
365366

366367
for g_id, group in groupby(annotations, lambda a: a.group):

datumaro/datumaro/plugins/transforms.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,27 @@ def convert_polygon(polygon, img_h, img_w):
181181
return RleMask(rle=rle, label=polygon.label, z_order=polygon.z_order,
182182
id=polygon.id, attributes=polygon.attributes, group=polygon.group)
183183

184+
class BoxesToMasks(Transform, CliPlugin):
185+
def transform_item(self, item):
186+
annotations = []
187+
for ann in item.annotations:
188+
if ann.type == AnnotationType.bbox:
189+
if not item.has_image:
190+
raise Exception("Image info is required for this transform")
191+
h, w = item.image.size
192+
annotations.append(self.convert_bbox(ann, h, w))
193+
else:
194+
annotations.append(ann)
195+
196+
return self.wrap_item(item, annotations=annotations)
197+
198+
@staticmethod
199+
def convert_bbox(bbox, img_h, img_w):
200+
rle = mask_utils.frPyObjects([bbox.as_polygon()], img_h, img_w)[0]
201+
202+
return RleMask(rle=rle, label=bbox.label, z_order=bbox.z_order,
203+
id=bbox.id, attributes=bbox.attributes, group=bbox.group)
204+
184205
class MasksToPolygons(Transform, CliPlugin):
185206
def transform_item(self, item):
186207
annotations = []

0 commit comments

Comments
 (0)