Release v1.1.0

What's Changed - Brief Version

New features

• Add with_subset_dirs decorator (Add ImagenetWithSubsetDirsImporter)
  (#816)
• Add CommonSemanticSegmentationWithSubsetDirsImporter
  (#826)
• Add DatumaroBinary format
  (#828, #829, #830, #831, #880, #883)
• Add Searcher CLI documentation
  (#838)
• Add version to dataset exported as datumaro format
  (#842)
• Add Ava action data format support
  (#847)
• Add Shift Analyzer (both covariate and label shifts)
  (#855)
• Add YOLO Loose format
  (#856)
• Add Ultralytics YOLO format
  (#859)

Enhancements

• Refactor Datumaro format code and test code
  (#824)
• Add publish to PyPI Github action
  (#867)
• Add --no-media-encryption option
  (#875)

Bug fixes

• Fix image filenames and anomaly mask appearance in MVTec exporter
  (#835)
• Fix CIFAR10 and 100 detect function
  (#836)
• Fix celeba and align_celeba detect function
  (#837)
• Choose the top priority detect format for all directory depths
  (#839)
• Fix MVTec format detect function
  (#843)
• Fix wrong __len__() of Subset when the item is removed
  (#854)
• Fix mask visualization bug
  (#860)
• Fix detect unit tests to test false negatives as well
  (#868)

What's Changed - Full Version

• Add daily/weekly test triggers by @chuneuny-emily in #811
• Raise ImportError on importing malformed COCO directory by @vinnamkim in #812
• Upload data explorer model in public storage by @sooahleex in #813
• Merge back releases/v1.0.0 to develop for taping out v1.0.0.rc by @vinnamkim in #818
• Add with_subset_dirs decorator by @vinnamkim in #816
• Skip some video unit tests on MacOS by @vinnamkim in #825
• Update copyright year in PR template by @vinnamkim in #823
• Refactor Datumaro format code and test code by @vinnamkim in #824
• Add CommonSemanticSegmentationWithSubsetDirsImporter by @vinnamkim in #826
• Develop DatumaroBinaryFormat to export/import the dataset header & DatasetItem by @vinnamkim in #828
• Update weekly_check.yml by @yunchu in #833
• Remove Codacy badge in readme by @chuneuny-emily in #834
• Implement DatumaroBinaryFormat to export/import the image dataset completely by @vinnamkim in #829
• Fix bugs in mvtec exporter by @djdameln in #835
• [Doc] Add documentation for searcher cli by @sooahleex in #838
• Support PointCloud dataset by DatumaroBinary format by @vinnamkim in #830
• [TEST] move test_utils.py to tests package by @yunchu in #841
• Fix CIFAR10 and 100 detect function by @vinnamkim in #836
• Merge back/releases/v1.0.0 by @vinnamkim in #846
• [HOTFIX] Update ipas_default.config by @yunchu in #848
• [HOTFIX] Update ipas_default.config by @yunchu in #849
• [develop] Update .gitattributes by @yunchu in #851
• Choose the top priority detect format for all directory depths by @vinnamkim in #839
• mark datumaro library version when exporting as datumaro by @bonhunko in #842
• Add AVA action data format support by @wonjuleee in #847
• Fix MVTec format detect function by @vinnamkim in #843
• Fix wrong len() of Subset when the item is removed by @vinnamkim in #854
• Fix celeba and align_celeba detect function by @vinnamkim in #837
• Fix mask visualization bug by @vinnamkim in #860
• Add YOLO Loose format by @vinnamkim in #856
• Add ShiftAnalyzer to compute covariate and label shift between two datasets by @wonjuleee in #855
• Add Ultralytics YOLO format by @vinnamkim in #859
• Add full encryption/decryption functionalities for image datasets to DatumaroBinary format by @vinnamkim in #831
• [Doc] AVA action and MVTec data formats by @wonjuleee in #865
• Update PR template by @chuneuny-emily in #869
• Fix detect unit tests to test false negatives as well by @vinnamkim in #868
• Update doc for regex rules in transform by @sooahleex in #874
• Add publish to PyPI Github action by @vinnamkim in #867
• Add Video formats (video_frames and video_keyframes) by @wonjuleee in #872
• Add prefix to secret key by @vinnamkim in #878
• Add missing requirements by @cih9088 in #877
• Add --no-media-encryption option by @vinnamkim in #875
• Update 3rd-party.txt by @chuneuny-emily in #884
• Add datumaro binary format documentation by @vinnamkim in #880
• Add multiprocessing to DatumaroBinaryExporter by @vinnamkim in #883
• Version up to Releases/1.1.0 by @vinnamkim in #886

New Contributors

• @djdameln made their first contribution in #835
• @cih9088 made their first contribution in #877

Full Changelog: v1.0.0...v1.1.0

Release 1.1.0rc1

What's Changed - Brief Version

Added

• Add with_subset_dirs decorator (Add ImagenetWithSubsetDirsImporter)
  (#816)
• Add CommonSemanticSegmentationWithSubsetDirsImporter
  (#826)
• Add DatumaroBinary format
  (#828, #829, #830, #831)
• Add Searcher CLI documentation
  (#838)
• Add version to dataset exported as datumaro format
  (#842)
• Add Ava action data format support
  (#847)
• Add Shift Analyzer (both covariate and label shifts)
  (#855)
• Add YOLO Loose format
  (#856)
• Add Ultralytics YOLO format
  (#859)

Changed

• Refactor Datumaro format code and test code
  (#824)

Fixed

• Fix image filenames and anomaly mask appearance in MVTec exporter
  (#835)
• Fix CIFAR10 and 100 detect function
  (#836)
• Fix celeba and align_celeba detect function
  (#837)
• Choose the top priority detect format for all directory depths
  (#839)
• Fix MVTec format detect function
  (#843)
• Fix wrong __len__() of Subset when the item is removed
  (#854)
• Fix mask visualization bug
  (#860)

What's Changed - Full Version

• Add daily/weekly test triggers by @chuneuny-emily in #811
• Raise ImportError on importing malformed COCO directory by @vinnamkim in #812
• Upload data explorer model in public storage by @sooahleex in #813
• Merge back releases/v1.0.0 to develop for taping out v1.0.0.rc by @vinnamkim in #818
• Add with_subset_dirs decorator by @vinnamkim in #816
• Skip some video unit tests on MacOS by @vinnamkim in #825
• Update copyright year in PR template by @vinnamkim in #823
• Refactor Datumaro format code and test code by @vinnamkim in #824
• Add CommonSemanticSegmentationWithSubsetDirsImporter by @vinnamkim in #826
• Develop DatumaroBinaryFormat to export/import the dataset header & DatasetItem by @vinnamkim in #828
• Update weekly_check.yml by @yunchu in #833
• Remove Codacy badge in readme by @chuneuny-emily in #834
• Implement DatumaroBinaryFormat to export/import the image dataset completely by @vinnamkim in #829
• Fix bugs in mvtec exporter by @djdameln in #835
• [Doc] Add documentation for searcher cli by @sooahleex in #838
• Support PointCloud dataset by DatumaroBinary format by @vinnamkim in #830
• [TEST] move test_utils.py to tests package by @yunchu in #841
• Fix CIFAR10 and 100 detect function by @vinnamkim in #836
• Merge back/releases/v1.0.0 by @vinnamkim in #846
• [HOTFIX] Update ipas_default.config by @yunchu in #848
• [HOTFIX] Update ipas_default.config by @yunchu in #849
• [develop] Update .gitattributes by @yunchu in #851
• Choose the top priority detect format for all directory depths by @vinnamkim in #839
• mark datumaro library version when exporting as datumaro by @bonhunko in #842
• Add AVA action data format support by @wonjuleee in #847
• Fix MVTec format detect function by @vinnamkim in #843
• Fix wrong len() of Subset when the item is removed by @vinnamkim in #854
• Fix celeba and align_celeba detect function by @vinnamkim in #837
• Fix mask visualization bug by @vinnamkim in #860
• Add YOLO Loose format by @vinnamkim in #856
• Add ShiftAnalyzer to compute covariate and label shift between two datasets by @wonjuleee in #855
• Add Ultralytics YOLO format by @vinnamkim in #859
• Add full encryption/decryption functionalities for image datasets to DatumaroBinary format by @vinnamkim in #831

New Contributors

• @djdameln made their first contribution in #835

Full Changelog: v1.0.0...v1.1.0rc1

Release v1.0.0

Added

• Add Data Explorer (#773)
• Add Ellipse annotation type (#807)
• Add MVTec anomaly data support (#810)

Changed

• Refactor existing tests (#803)
• Raise ImportError on importing malformed COCO directory (#812)
• Remove the duplicated and cyclical category context in documentation (#822)

Fixed

• Fix for importing CVAT image 1.1 data format exported to project level (#795)
• Fix a problem on setting log-level via CLI (#800)
• Fix code format with the latest black==23.1.0 (#802)
• Fix Explain command cannot find the model (#721) (#804)
• Fix a problem found on model remove CLI command (#805)

Release v0.5.0

Added

• Add Tile transformation (#790)
• Add Video keyframe extraction (#791)
• Add TileTransform documentation and Jupyter notebook example (#794)
• Add MergeTile transformation (#796)

Changed

• Improved mask_to_rle performance (#770)

Deprecated

• N/A

Removed

• N/A

Fixed

• Fix auto-documentation for the data_format plugins (#793)

Security

• Add security.md file for the SDL (#798)

Release v0.4.0.1

Added

• Support for exclusive of labels with LabelGroup (#742)
• Jupyter samples
  • Introducing how to merge datasets (#738)
  • Introducing how to visualize dataset (#747)
  • Introducing how to filter dataset (#748)
  • Introducing how to transform dataset (#759)
• Visualization Python API
  • Bbox feature (#744)
  • Label, Points, Polygon, PolyLine, and Caption visualization features (#746)
  • Mask, SuperResolution, Depth visualization features (#747)
• Documentation for Python API (#753)
  • dataset handler, visualizer, filter descriptions (#761)
• Support for exporting as CVAT video format (#757)
• Jupyter notebook example rendering to documentation (#758)
• An interface to manipulate 'infos' to store the dataset meta-info (#767)
• 'bbox' annotation when importing a COCO dataset (#772)

Changed

• Wrap title text according to its plot width (#769)
• Get list of subsets and support only Image media type in visualizer (#768)

Deprecated

• N/A

Removed

• N/A

Fixed

• Correcting static type checking (#743)
• Fixing a VOC dataset export error when a label contains 'space' (#771)

Security

• N/A

Release v0.3.1

Added

• Support for custom media types, new PointCloud media type, DatasetItem.media and .media_as(type) members (#539)
• [API] A way to request dataset and extractor media type with media_type (#539)
• BraTS format (import-only) (.npy and .nii.gz), new MultiframeImage media type (#628)
• Common Semantic Segmentation dataset format (import-only) (#685)
• An option to disable data/ prefix inclusion in YOLO export (#689)
• New command describe-downloads to print information about downloadable datasets (#678)
• Detection for Cityscapes format (#680)
• Maximum recursion --depth parameter for detect-dataset CLI command (#680)
• An option to save a single subset in the download command (#697)
• Common Super Resolution dataset format (import-only) (#700)
• Kinetics 400/600/700 dataset format (import-only) (#706)
• NYU Depth Dataset V2 format (import-only) (#712)

Changed

• env.detect_dataset() now returns a list of detected formats at all recursion levels instead of just the lowest one (#680)
• Open Images: allowed to store annotations file in root path as well (#680)
• Improved parsing error messages in COCO, VOC and YOLO formats (#684, #686, #687)
• YOLO format now supports almost any subset names, except backup, names and classes (instead of just train and valid). The reserved names now raise an error on exporting. (#688)

Deprecated

• --save-images is replaced with --save-media in CLI and converter API (#539)
• [API] image, point_cloud and related_images of DatasetItem are replaced with media and media_as(type) members and c-tor parameters (#539)

Removed

• N/A

Fixed

• Detection for LFW format (#680)
• Adding depth value of image when dataset is exported in VOC format (#726)
• Adding to handle the numerical labels in task chains properly (#726)
• Fixing the issue that annotations inside another annotation (polygon) are duplicated during import for VOC format (#726)

Security

• N/A

Release v0.3: Video Support

Added

• Ability to import a video as frames with the video_frames format and to split a video into frames with the datum util split_video command (#555)
• --subset parameter in the image_dir format (#555)
• MediaManager API to control loaded media resources at runtime (#555)
• Command to detect the format of a dataset (#576)
• More comfortable access to library API via import datumaro (#630)
• CLI command-like free functions (export, transform, ...) (#630)
• Reading specific annotation files for train dataset in Cityscapes (#632)
• Random sampling transforms (random_sampler, label_random_sampler) to create smaller datasets from bigger ones (#636, #640)
• API to report dataset import and export progress; API to report dataset import and export errors and take action (skip, fail)
  (supported in COCO, VOC and YOLO formats) (#650)
• Support for downloading the ImageNetV2 and COCO datasets (#653, #659)
• A way for formats to signal that they don't support detection (#665)
• Removal transforms to remove items/annoations/attributes from dataset (remove_items, remove_annotations, remove_attributes) (#670)

Changed

• Allowed direct file paths in datum import. Such sources are imported like when the rpath parameter is specified, however, only the selected path is copied into the project (#555)
• Improved stats performance, added new filtering parameters, image stats (unique, repeated) moved to the dataset section,
  removed mean and std from the dataset section (#621)
• Allowed Image creation from just size info (#634)
• Added image search in VOC XML-based subformats (#634)
• Added image path equality checks in simple merge, when applicable (#634)
• Supported saving box attributes when downloading the TFDS version of VOC (#668)
• Switched to a pyproject.toml-based build (#671)

Deprecated

• TBD

Removed

• Official support of Python 3.6 (due to it's EOL) (#617)
• Backward compatibility annotation symbols in components.extractor (#630)

Fixed

• Prohibited calling add, import and export commands without a project (#555)
• Calling make_dataset on empty project tree now produces the error properly (#555)
• Saving (overwriting) a dataset in a project when rpath is used (#613)
• Output image extension preserving in the Resize transform (#606)
• Memory overuse in the Resize transform (#607)
• Invalid image pixels produced by the Resize transform (#618)
• Numeric warnings that sometimes occurred in stats command (e.g. #607) (#621)
• Added missing item attribute merging in simple merge (#634)
• Inability to disambiguate VOC from LabelMe in some cases (#658)

Security

• TBD

Release v0.2.3: Public dataset downloading

Added

• Command to download public datasets (#582)
• Extension autodetection in ByteImage (#595)
• MPII Human Pose Dataset (import-only) (.mat and .json) (#584)
• MARS format (import-only) (#585)

Changed

• smooth_line from datumaro.util.annotation_util - the function is renamed to approximate_line and has updated interface (#592)
• The pycocotools dependency lower bound is raised to 2.0.4 (#449)

Deprecated

• Python 3.6 support

Fixed

• Fails in multimerge when lines are not approximated and when there are no label categories (#592)
• Cannot convert LabelMe dataset, that has no subsets (#600)

Release v0.2.2

Added

• Video reading API (#521)
• Python API documentation site (#526)
• Mapillary Vistas dataset format (Import-only) (#537)
• Datumaro can now be installed on Windows on Python 3.9 (#547)
• SYNTHIA dataset format (Import-only) (#532)
• Support of score attribute in KITTI detection (#571)
• Support for Accuracy Checker dataset meta files in formats (#553, #569, #575)
• VoTT dataset format (Import-only) (#573)
• Image resizing transform (#581)

Changed

• The following formats can now be detected unambiguously: ade20k2017, ade20k2020, camvid, coco, cvat, datumaro, icdar_text_localization, icdar_text_segmentation, icdar_word_recognition, imagenet_txt, kitti_raw, label_me, lfw, mot_seq, open_images, vgg_face2, voc, widerface, yolo (#531, #536, #550, #557, #558)
• Allowed export options in the datum merge command (#545)

Deprecated

• Using Image, ByteImage from datumaro.util.image - these classes are moved to datumaro.components.media (#538)

Removed

• Equality comparison support between datumaro.components.media.Image and numpy.ndarray (#568)

Fixed

• Bug #560: import issue with MOT dataset when using seqinfo.ini file (#564)
• Empty lines in VOC subset lists are now ignored (#587)

Release v0.2.1

A bugfix release. Relaxes some requirements on formats.

Added

• Import for CelebA dataset format (#484)

Changed

• File people.txt became optional in LFW (#509)
• File image_ids_and_rotation.csv became optional Open Images (#509)
• Allowed underscores (_) in subset names in COCO (#509)
• Allowed annotation files with arbitrary names in COCO (#509)
• The icdar_text_localization format is no longer detected in every directory (#531)
• Updated pycocotools version to 2.0.2 (#534)

Fixed

• Unhandled exception when a file is specified as the source for a COCO or MOTS dataset (#530)