English | 简体中文

Is Your HD Map Constructor Reliable under Sensor Corruptions?

Xiaoshuai Hao¹    Mengchuan Wei¹    Yifan Yang¹    Haimei Zhao
Hui Zhang¹    Yi Zhou¹    Qiang Wang¹    Weiming Li
Lingdong Kong^3,‡    Jing Zhang<sup>2,‡

1</sup>Samsung R&D Institute China-Beijing    ²The University of Sydney    ³National University of Singapore

About

MapBench is the first comprehensive benchmark designed to evaluate the out-of-domain robustness of HD map construction methods against various sensor corruptions.

Our benchmark encompasses a total of 16 corruption types for HD map construction, which can be categorized into exterior, interior, and sensor failure scenarios. Besides, we define 13 multi-sensor corruptions by combining the camera and LiDAR sensor failure types.

Updates

• [2024.06] - Launch of the MapBench benchmark. In this version, we include a total of 31 HD map construction models, evaluated on 29 different camera and LiDAR corruption types across 3 severity levels.

Outline

• Benchmark Definition
• Installation
• Data Preparation
• Getting Started
• Model Zoo
• Benchmark
• TODO List
• Citation
• License
• Acknowledgements

📊 Benchmark Definition

MapBench consists of a total of 29 different sensor corruption scenarios, including 8 types of camera corruptions, 8 types of LiDAR corruptions, and 13 types of camera-LiDAR corruption combinations.

LiDAR Sensor Corruptions

Wet Ground	Snow	Beam Missing	Incomplete Echo
Fog	Motion Blur	Crosstalk	Cross-Sensor

Type	Description	Parameter	Easy	Moderate	Hard
Wet Ground	significantly attenuated laser echoes	(water height, noise floor)	(0.2, 0.2)	(1.0, 0.3)	(1.2, 0.7)
Snow	back-scattering and attenuation of LiDAR points	(snowfall rate, terminal velocity)	(0.5, 2.0)	(1.0, 1.6)	(2.5, 1.6)
Beam Missing	loss of certain light impulses	beam number to drop	8	16	24
Incomplete Echo	incomplete LiDAR readings	drop ratio	0.75	0.85	0.95
Fog	back-scattering and attenuation of LiDAR points	beta	0.008	0.05	0.2
Motion Blur	blur caused by vehicle movement	trans std	0.2	0.3	0.4
Crosstalk	light impulses interference	percentage	0.03	0.07	0.12
Cross-Sensor	cross sensor data	beam number to drop	8	16	20

Camera Sensor Corruptions

Brightness	Low-Light	Fog
Snow	Motion Blur	Color Quant

Type	Description	Parameter	Easy	Moderate	Hard
Brightness	varying daylight intensity	adjustment in HSV space	0.2	0.4	0.5
Low-Light	varying daylight intensity	scale factor	0.5	0.4	0.3
Fog	a visually obstructive form of precipitation	(thickness, smoothness)	(2.0, 2.0)	(2.5, 1.5)	(3.0, 1.4)
Snow	a visually obstructive form of precipitation	(mean, std, scale, threshold, blur radius, blur std, blending ratio)	(0.1, 0.3, 3.0, 0.5, 10.0, 4.0, 0.8)	(0.2, 0.3, 2, 0.5, 12, 4, 0.7)	(0.55, 0.3, 4, 0.9, 12, 8, 0.7)
Motion Blur	moving camera quickly	(radius, sigma)	(15, 5)	(15, 12)	(20, 15)
Color Quant	reducing the number of colors	bit number	5	4	3

Camera Sensor Failures

Frame Lost	Camera Crash

Type	Description	Parameter	Easy	Moderate	Hard
Frame Lost	dropping temporal frames	probability of frame dropping	2/6	4/6	5/6
Camera Crash	dropping view images	number of dropped cameras	2	4	5

⚙️ Installation

For details related to installation, kindly refer to INSTALL.md.

♨️ Data Preparation

Our datasets are hosted by OpenDataLab.

OpenDataLab is a pioneering open data platform for the large AI model era, making datasets accessible. By using OpenDataLab, researchers can obtain free formatted datasets in various fields.

Kindly refer to DATA_PREPARE.md for the details to prepare the training and evaluation datasets.

🚀 Getting Started

To learn more usage about this codebase, kindly refer to GET_STARTED.md.

🔆 Model Zoo

 HD Map Construction

• HDMapNet, ICRA 2022. ^[Code]
• VectorMapNet, ICML 2023. ^[Code]
• MapTR, ICLR 2023. ^[Code]
• MapTRv2, arXiv 2023. ^[Code]
• PivotNet, ICCV 2023. ^[Code]
• BeMapNet, CVPR 2023. ^[Code]
• StreamMapNet, WACV 2024. ^[Code]
• HIMap, CVPR 2024. ^[Code]

⛳ Benchmark

The mean average precision (mAP) is consistently used as the main indicator for evaluating model performance in our HD Map construction benchmark.

The following two metrics are adopted to compare among models' robustness under sensor corruptions:

• mCE (the lower the better): The average corruption error (in percentage) of a candidate model compared to the baseline model, which is calculated among all corruption types across three severity levels.
• mRR (the higher the better): The average resilience rate (in percentage) of a candidate model compared to its "clean" performance, which is calculated among all corruption types across three severity levels.

🚗  Camera-Only Benchmarking Results

Model	mCE	mRR	Clean	Camera	Frame	Quant	Motion	Bright	Dark	Fog	Snow
HDMapNet	18.78	43.3	23.0	4.6	5.1	18.9	20.8	16.7	9.3	10.6	5.2
VectorMapNet	148.5	40.6	40.9	13.9	12.3	26.6	29.7	25.2	7.8	18.3	2.9
PivotNet	96.3	45.2	57.4	17.1	16.7	36.4	34.1	43.5	16.5	37.4	4.6
BeMapNet	78.5	50.3	59.8	18.8	18.5	38.1	35.3	50.7	23.2	46.5	9.6
MapTR	100.0	49.3	50.3	15.0	14.2	35.4	23.5	44.3	22.7	38.5	3.8
MapTRv2	72.6	51.4	61.5	18.8	18.2	45.3	31.0	54.9	32.3	50.7	1.1
StreamMapNet	64.8	54.4	63.4	13.4	15.5	48.1	44.3	57.0	36.1	52.4	9.1
HIMap	56.9	56.6	65.5	19.4	19.0	52.0	42.5	60.9	40.6	57.1	5.1

🚙  LiDAR-Only Benchmarking Results

Model	mCE	mRR	Clean	Fog	Wet	Snow	Motion	Beam	Crosstalk	Echo	Sensor
VectorMapNet	94.9	63.4	34.0	15.7	20.3	15.9	28.8	19.2	19.7	31.3	9.5
MapTR	100.0	55.1	55.6	19.9	19.1	9.6	27.1	16.5	16.3	32.3	6.4
MapTRv2	74.6	57.2	61.5	28.5	29.5	10.3	36.9	27.9	15.4	44.7	14.0
HIMap	73.1	59.2	64.3	26.6	24.6	16.1	37.4	24.4	26.7	43.1	10.8

🚕  Camera-LiDAR Fusion Benchmarking Results

Model	Modality	Camera	Lidar	APped	APdiv	APbou	mAP
MapTR	C & L	✓	✓	55.9	62.3	69.3	62.5
MapTR	C	✓	—	46.3	51.5	53.1	50.3
MapTR	C	Camera Crash	—	18.0	14.5	12.4	15.0
MapTR	C	Frame Lost	—	13.9	15.1	13.4	14.2
MapTR	C & L	✗	✓	15.0	18.2	34.4	22.5
MapTR	C & L	Camera Crash	✓	32.5	36.5	48.4	39.1
MapTR	C & L	Frame Lost	✓	29.1	33.7	46.1	36.3
MapTR	L	—	✓	26.6	31.7	41.8	33.4
MapTR	L	—	Incomplete Echo	26.3	29.9	40.6	32.3
MapTR	L	—	Crosstalk	13.6	15.0	20.3	16.3
MapTR	L	—	Cross-Sensor	3.5	6.6	8.9	6.4
MapTR	C & L	✓	✗	20.7	27.4	13.1	20.4
MapTR	C & L	✓	Incomplete Echo	47.9	55.2	62.2	55.1
MapTR	C & L	✓	Crosstalk	36.7	42.5	45.3	41.5
MapTR	C & L	✓	Cross-Sensor	33.9	42.9	42.0	39.6
MapTR	C & L	Camera Crash	Incomplete Echo	32.4	35.6	47.8	38.6
MapTR	C & L	Camera Crash	Crosstalk	19.7	21.6	26.9	22.7
MapTR	C & L	Camera Crash	Cross-Sensor	18.4	20.8	23.2	20.8
MapTR	C & L	Frame Lost	Incomplete Echo	28.9	32.8	45.5	35.8
MapTR	C & L	Frame Lost	Crosstalk	16.9	19.9	25.5	20.8
MapTR	C & L	Frame Lost	Cross-Sensor	15.8	19.4	22.2	19.1

Model	Modality	Camera	Lidar	APped	APdiv	APbou	mAP
HIMap	C & L	✓	✓	71.0	72.4	79.4	74.3
HIMap	C	✓	—	62.2	66.5	67.9	65.5
HIMap	C	Camera Crash	—	27.3	19.4	11.6	19.4
HIMap	C	Frame Lost	—	21.7	19.1	16.1	19.0
HIMap	C & L	✗	✓	40.9	46.4	74.7	50.7
HIMap	C & L	Camera Crash	✓	36.3	27.7	20.9	28.3
HIMap	C & L	Frame Lost	✓	29.9	25.0	23.8	26.2
HIMap	L	—	✓	54.8	64.7	73.5	64.3
HIMap	L	—	Incomplete Echo	35.4	41.1	52.7	43.1
HIMap	L	—	Crosstalk	20.9	23.8	35.3	26.7
HIMap	L	—	Cross-Sensor	7.8	10.2	14.4	10.8
HIMap	C & L	✓	✗	30.7	38.7	29.0	32.8
HIMap	C & L	✓	Incomplete Echo	59.1	63.7	69.9	64.2
HIMap	C & L	✓	Crosstalk	54.1	57.5	63.4	58.3
HIMap	C & L	✓	Cross-Sensor	44.2	50.7	50.8	48.5
HIMap	C & L	Camera Crash	Incomplete Echo	36.2	26.9	20.5	27.9
HIMap	C & L	Camera Crash	Crosstalk	29.2	19.3	12.9	20.5
HIMap	C & L	Camera Crash	Cross-Sensor	23.1	13.8	5.9	14.3
HIMap	C & L	Frame Lost	Incomplete Echo	29.9	24.4	23.5	25.9
HIMap	C & L	Frame Lost	Crosstalk	23.6	18.9	18.0	20.2
HIMap	C & L	Frame Lost	Cross-Sensor	17.7	14.3	11.2	14.4

📝 TODO List

• Initial release. 🚀
• Add scripts for creating common corruptions.
• Add evaluation scripts on corruption sets.
• ...

Citation

If you find this work helpful, please kindly consider citing our paper:

@article{hao2024mapbench,
    author = {Xiaoshuai Hao and Mengchuan Wei and Yifan Yang and Haimei Zhao and Hui Zhang and Yi Zhou and Qiang Wang and Weiming Li and Lingdong Kong and Jing Zhang},
    title = {Is Your HD Map Constructor Reliable under Sensor Corruptions?},
    journal={arXiv preprint arXiv:2406.12214},
    year = {2024},
}

License

This work is under the Apache License Version 2.0, while some specific operations in this codebase might be with other licenses. Please refer to LICENSE.md for a more careful check, if you are using our code for commercial matters.

Acknowledgements