I’m trying to convert an rtmdet model to CoreML.

[Jungkyu-Lee]

I’m trying to convert an rtmdet model to CoreML.
I’ve successfully created a converter with a fixed 1 batch size, but having trouble with dynamic batch sizes.
Is there a way to create a converter that supports dynamic batch sizes?

Environment

06/26 17:36:04 - mmengine - INFO -

06/26 17:36:04 - mmengine - INFO - **********Environmental information**********
06/26 17:36:05 - mmengine - INFO - sys.platform: darwin
06/26 17:36:05 - mmengine - INFO - Python: 3.8.19 (default, Mar 20 2024, 15:27:52) [Clang 14.0.6 ]
06/26 17:36:05 - mmengine - INFO - CUDA available: False
06/26 17:36:05 - mmengine - INFO - MUSA available: False
06/26 17:36:05 - mmengine - INFO - numpy_random_seed: 2147483648
06/26 17:36:05 - mmengine - INFO - GCC: Apple clang version 15.0.0 (clang-1500.3.9.4)
06/26 17:36:05 - mmengine - INFO - PyTorch: 2.0.0
06/26 17:36:05 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 4.2
  - C++ Version: 201703
  - clang 13.1.6
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - Build settings: BLAS_INFO=accelerate, BUILD_TYPE=Release, CXX_COMPILER=/Applications/Xcode_13.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_PYTORCH_METAL_EXPORT -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DUSE_COREML_DELEGATE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=braced-scalar-init -Werror=range-loop-construct -Werror=bool-operation -Winconsistent-missing-override -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wvla-extension -Wno-range-loop-analysis -Wno-pass-failed -Wsuggest-override -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-constexpr-not-const -Wno-missing-braces -Wunused-lambda-capture -Wunused-local-typedef -Qunused-arguments -fcolor-diagnostics -fdiagnostics-color=always -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -DUSE_MPS -fno-objc-arc -Wno-unguarded-availability-new -Wno-unused-private-field -Wno-missing-braces -Wno-constexpr-not-const, LAPACK_INFO=accelerate, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.0.0, USE_CUDA=0, USE_CUDNN=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=OFF, USE_ROCM=OFF,

06/26 17:36:05 - mmengine - INFO - TorchVision: 0.15.0
06/26 17:36:05 - mmengine - INFO - OpenCV: 4.10.0
06/26 17:36:05 - mmengine - INFO - MMEngine: 0.10.4
06/26 17:36:05 - mmengine - INFO - MMCV: 2.0.0
06/26 17:36:05 - mmengine - INFO - MMCV Compiler: clang 14.0.3
06/26 17:36:05 - mmengine - INFO - MMCV CUDA Compiler: not available
06/26 17:36:05 - mmengine - INFO - MMDeploy: 1.3.0+e31f5a6
06/26 17:36:05 - mmengine - INFO -

06/26 17:36:05 - mmengine - INFO - **********Backend information**********
06/26 17:36:05 - mmengine - INFO - tensorrt:	None
06/26 17:36:05 - mmengine - INFO - ONNXRuntime:	None
06/26 17:36:05 - mmengine - INFO - pplnn:	None
06/26 17:36:05 - mmengine - INFO - ncnn:	None
06/26 17:36:05 - mmengine - INFO - snpe:	None
06/26 17:36:05 - mmengine - INFO - openvino:	None
06/26 17:36:06 - mmengine - INFO - torchscript:	2.0.0
06/26 17:36:06 - mmengine - INFO - torchscript custom ops:	Available
06/26 17:36:06 - mmengine - INFO - rknn-toolkit:	None
06/26 17:36:06 - mmengine - INFO - rknn-toolkit2:	None
06/26 17:36:06 - mmengine - INFO - ascend:	None
06/26 17:36:06 - mmengine - INFO - coreml:	6.3.0
06/26 17:36:06 - mmengine - INFO - tvm:	None
06/26 17:36:06 - mmengine - INFO - vacc:	None
06/26 17:36:06 - mmengine - INFO -

06/26 17:36:06 - mmengine - INFO - **********Codebase information**********
06/26 17:36:06 - mmengine - INFO - mmdet:	3.3.0
06/26 17:36:06 - mmengine - INFO - mmseg:	None
06/26 17:36:06 - mmengine - INFO - mmpretrain:	None
06/26 17:36:06 - mmengine - INFO - mmocr:	None
06/26 17:36:06 - mmengine - INFO - mmagic:	None
06/26 17:36:06 - mmengine - INFO - mmdet3d:	None
06/26 17:36:06 - mmengine - INFO - mmpose:	None
06/26 17:36:06 - mmengine - INFO - mmrotate:	None
06/26 17:36:06 - mmengine - INFO - mmaction:	None
06/26 17:36:06 - mmengine - INFO - mmrazor:	None
06/26 17:36:06 - mmengine - INFO - mmyolo:	0.6.0

then i execute

python tools/deploy.py \
    configs/mmdet/detection/detection_coreml_dynamic.py \
    ../mmdetection/configs/rtmdet/rtmdet_m_8xb32-300e_coco.py \
https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_m_8xb32-300e_coco/rtmdet_m_8xb32-300e_coco_20220719_112220-229f527c.pth \
    ../mmdetection/demo/demo.jpg \
    --work-dir work_dir/rtmdet-m \
    --device cpu \
    --dump-info

detection_coreml_dynamic.py file like this:

_base_ = [ './base_torchscript.py', '../../_base_/backends/coreml.py']

ir_config = dict(input_shape=(320, 320))
backend_config = dict(model_inputs=[
    dict(
        input_shapes=dict(
            input=dict(
                min_shape=[1, 3, 320, 320],
                max_shape=[32, 3, 320, 320],
                default_shape=[1, 3, 320, 320])))
])

but i got following error messeges.

06/26 17:38:56 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
06/26 17:38:56 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
06/26 17:38:58 - mmengine - INFO - Start pipeline mmdeploy.apis.pytorch2torchscript.torch2torchscript in subprocess
06/26 17:39:00 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
06/26 17:39:00 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
Loads checkpoint by http backend from path: https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet_m_8xb32-300e_coco/rtmdet_m_8xb32-300e_coco_20220719_112220-229f527c.pth
The model and loaded state dict do not match exactly

unexpected key in source state_dict: data_preprocessor.mean, data_preprocessor.std

06/26 17:39:00 - mmengine - INFO - Export PyTorch model to torchscript.
/Users/sweaterr/Projects/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py:257: UserWarning: Detect multiple valid rewriters formmdeploy.mmcv.ops.nms._multiclass_nms, use the first rewriter.
  warnings.warn(
/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1678455016227/work/aten/src/ATen/native/TensorShape.cpp:3484.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdet/models/dense_heads/anchor_head.py:115: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead
  warnings.warn('DeprecationWarning: anchor_generator is deprecated, '
/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdet/models/dense_heads/anchor_head.py:108: UserWarning: DeprecationWarning: `num_anchors` is deprecated, for consistency or also use `num_base_priors` instead
  warnings.warn('DeprecationWarning: `num_anchors` is deprecated, '
/Users/sweaterr/Projects/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/single_stage.py:80: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  img_shape = [int(val) for val in img_shape]
/Users/sweaterr/Projects/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/single_stage.py:80: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  img_shape = [int(val) for val in img_shape]
/Users/sweaterr/Projects/mmdeploy/mmdeploy/mmcv/ops/nms.py:573: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert batch_size == 1, 'batched nms is not supported for now.'
/Users/sweaterr/Projects/mmdeploy/mmdeploy/pytorch/functions/topk.py:91: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  k = torch.tensor(k, device=input.device, dtype=torch.long)
06/26 17:39:01 - mmengine - INFO - Save PyTorch model: work_dir/rtmdet-m/end2end.pt.
06/26 17:39:02 - mmengine - INFO - Finish pipeline mmdeploy.apis.pytorch2torchscript.torch2torchscript
06/26 17:39:02 - mmengine - INFO - Start pipeline mmdeploy.apis.utils.utils.to_backend in main process
inputs 1
inputs [TensorType[name=input, shape=<coremltools.converters.mil.input_types.Shape object at 0x2bf1f0760>, dtype=None]]
Support for converting Torch Script Models is experimental. If possible you should use a traced model for conversion.
Tuple detected at graph output. This will be flattened in the converted model.
Converting PyTorch Frontend ==> MIL Ops:  53%|██████████████████████████████████████████████████████▍                                                | 608/1150 [00:00<00:00, 4953.52 ops/s]Saving value type of int64 into a builtin type of int32, might lose precision!
Saving value type of int64 into a builtin type of int32, might lose precision!
Saving value type of int64 into a builtin type of int32, might lose precision!
Saving value type of int64 into a builtin type of int32, might lose precision!
Saving value type of int64 into a builtin type of int32, might lose precision!
Saving value type of int64 into a builtin type of int32, might lose precision!
Converting PyTorch Frontend ==> MIL Ops:  97%|██████████████████████████████████████████████████████████████████████████████████████████████████▉   | 1116/1150 [00:00<00:00, 3888.61 ops/s]
Traceback (most recent call last):
  File "tools/deploy.py", line 335, in <module>
    main()
  File "tools/deploy.py", line 259, in main
    backend_files = to_backend(
  File "/Users/sweaterr/Projects/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 356, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "/Users/sweaterr/Projects/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "/Users/sweaterr/Projects/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "/Users/sweaterr/Projects/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/Users/sweaterr/Projects/mmdeploy/mmdeploy/apis/utils/utils.py", line 98, in to_backend
    return backend_mgr.to_backend(
  File "/Users/sweaterr/Projects/mmdeploy/mmdeploy/backend/coreml/backend_manager.py", line 108, in to_backend
    from_torchscript(
  File "/Users/sweaterr/Projects/mmdeploy/mmdeploy/backend/coreml/torchscript2coreml.py", line 109, in from_torchscript
    mlmodel = ct.convert(
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 492, in convert
    mlmodel = mil_convert(
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 212, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 285, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 108, in __call__
    return load(*args, **kwargs)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 63, in load
    return _perform_torch_convert(converter, debug)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 102, in _perform_torch_convert
    prog = converter.convert()
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 284, in convert
    convert_nodes(self.context, self.graph)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 88, in convert_nodes
    add_op(context, node)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 4818, in where
    broadcast_inputs = _broadcast_tensors([cond, inputs[1], inputs[2]])
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 5251, in _broadcast_tensors
    results.append(_broadcast(name, tensor, broadcast_shape))
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 4027, in _broadcast
    res = mb.tile(x=tensor, reps=reps, name=name)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 182, in add_op
    return cls._add_op(op_cls_to_add, **kwargs)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/mil/builder.py", line 166, in _add_op
    new_op = op_cls(**kwargs)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/mil/operation.py", line 187, in __init__
    self._validate_and_set_inputs(input_kv)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/mil/operation.py", line 496, in _validate_and_set_inputs
    self.input_spec.validate_inputs(self.name, self.op_type, input_kvs)
  File "/Users/sweaterr/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/coremltools/converters/mil/mil/input_type.py", line 162, in validate_inputs
    raise ValueError(msg.format(name, var.name, input_type.type_str,
ValueError: Op "2170_after_broadcast" (op_type: tile) Input reps="2170_after_broadcast_reps_0" expects tensor or scalar of dtype from type domain ['int32'] but got tensor[0,fp32]