Skip to content

feat: update ground segmentation evaluator to be compatible with lidarseg dataset #156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

nanoshimarobot
Copy link
Contributor

@nanoshimarobot nanoshimarobot commented May 28, 2025

Types of PR

  • Upgrade of existing features

Description

This PR removes the annotated_rosbag mode from ground_segmentation_evaluator and updates the annotated_pcd mode.

  • Remove annotated_rosbag
    Maintaining this mode requires patching autoware.universe, which has become burdensome and error-prone. So we’re simply discontinuing this mode.

  • Update annotated_pcd
    Until now, segmentation labels were assumed to be written directly into t4_dataset/data/LIDAR_CONCAT/*.pcd.bin.
    Since this PR introduced a standard format and location for lidar segmentation labels, annotated_pcd has been re-implemented to follow that spec.

  • TIER IV internal link

How to review this PR

Test data and scenario

test dataset : TIERIV_INTERNAL_LINK
scenario:

ScenarioFormatVersion: 3.0.0
ScenarioName: ground_segmentation-sample
ScenarioDescription: ground_segmentation-sample
SensorModel: aip_x2_gen2
VehicleModel: j6_gen2
Evaluation:
  UseCaseName: ground_segmentation
  UseCaseFormatVersion: 0.3.0
  Datasets:
    - sample_dataset:
        VehicleId: default
  Conditions:
    Method: annotated_pcd 
    ground_label: 20
    obstacle_label: 7
    accuracy_min: 0.7
    accuracy_max: 1.0
    PassRate: 99.0

Test command

dlr2 simulation run ~/driving_log_replayer_v2 --launch_args rate:=0.1

Test result

{"Condition": {"ground_label": 20, "obstacle_label": 7, "accuracy_min": 0.7, "accuracy_max": 1.0, "PassRate": 99.0}}
{"Result": {"Success": false, "Summary": "NoData"}, "Stamp": {"System": 1748414570.2117581}, "Frame": {}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 1 / 1 -> 100.00"}, "Stamp": {"System": 1748414581.7609808, "ROS": 1736225810.2732875}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 96739, "FP": 0, "TN": 0, "FN": 6, "Accuracy": 0.9999379812910217, "Precision": 0.999999999999999, "Recall": 0.9999379812910217, "Specificity": 0.0, "F1-score": 0.9999689896339005}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 2 / 2 -> 100.00"}, "Stamp": {"System": 1748414585.0651531, "ROS": 1736225811.6182482}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 96577, "FP": 0, "TN": 0, "FN": 10, "Accuracy": 0.9998964663981684, "Precision": 0.9999999999999989, "Recall": 0.9998964663981684, "Specificity": 0.0, "F1-score": 0.9999482304691433}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 3 / 3 -> 100.00"}, "Stamp": {"System": 1748414588.832024, "ROS": 1736225815.2883682}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 74383, "FP": 0, "TN": 0, "FN": 21799, "Accuracy": 0.7733567611403372, "Precision": 0.9999999999999987, "Recall": 0.7733567611403372, "Specificity": 0.0, "F1-score": 0.8721953506968656}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 4 / 4 -> 100.00"}, "Stamp": {"System": 1748414591.9241204, "ROS": 1736225818.6884308}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 69543, "FP": 0, "TN": 0, "FN": 26267, "Accuracy": 0.7258428139025146, "Precision": 0.9999999999999986, "Recall": 0.7258428139025146, "Specificity": 0.0, "F1-score": 0.8411459120302676}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 5 / 5 -> 100.00"}, "Stamp": {"System": 1748414594.6325753, "ROS": 1736225822.2882407}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 84012, "FP": 0, "TN": 0, "FN": 12888, "Accuracy": 0.8669969040247669, "Precision": 0.9999999999999988, "Recall": 0.8669969040247669, "Specificity": 0.0, "F1-score": 0.928760944497878}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 6 / 6 -> 100.00"}, "Stamp": {"System": 1748414596.7132423, "ROS": 1736225824.498351}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 83723, "FP": 0, "TN": 0, "FN": 12866, "Accuracy": 0.8667964260940678, "Precision": 0.9999999999999988, "Recall": 0.8667964260940678, "Specificity": 0.0, "F1-score": 0.9286459026078692}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 7 / 7 -> 100.00"}, "Stamp": {"System": 1748414599.0220072, "ROS": 1736225826.5733135}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 80775, "FP": 0, "TN": 0, "FN": 14445, "Accuracy": 0.8482986767485813, "Precision": 0.9999999999999988, "Recall": 0.8482986767485813, "Specificity": 0.0, "F1-score": 0.9179238046038771}}}}

Note: If no evaluation results are produced, change the point cloud topic to the following:

dlr2 simulation run ~/driving_log_replayer_v2 --launch_args rate:=0.1 evaluation_target_topic:=<topic_name>

or

ros2 launch driving_log_replayer_v2 driving_log_replayer_v2.launch.py scenario_path:=$HOME/driving_log_replayer_v2/ground_segmentation/scenario.yaml play_rate:=0.1 evaluation_target_topic:=<topic_name>

Others

@Copilot Copilot AI review requested due to automatic review settings May 28, 2025 07:17
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR discontinues the annotated_rosbag mode and reimplements annotated_pcd to use the new lidarseg.json spec for segmentation labels.

  • Drop annotated_rosbag support and related synchronization code
  • Load separate label files via lidarseg.json in annotated_pcd mode
  • Update CLOUD_DIM from 6 to 5 for the new point format

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
driving_log_replayer_v2/scripts/ground_segmentation_evaluator_node.py Remove rosbag mode, adjust annotated_pcd to parse labels from lidarseg.json, and set CLOUD_DIM=5.
driving_log_replayer_v2/driving_log_replayer_v2/ground_segmentation.py Remove Method field from Condition model to match dropped rosbag mode.
Comments suppressed due to low confidence (3)

driving_log_replayer_v2/scripts/ground_segmentation_evaluator_node.py:22

  • The import ros2_numpy is no longer used after removing annotated_rosbag_eval_cb; consider removing this unused import.
import ros2_numpy

driving_log_replayer_v2/scripts/ground_segmentation_evaluator_node.py:53

  • Unconditionally loading annotated PCD mode without validating eval_condition.Method may allow unsupported methods; add a check and raise an error if it's not "annotated_pcd".
sample_data_path = Path(self._t4_dataset_paths[0], "annotation", "sample_data.json")

driving_log_replayer_v2/driving_log_replayer_v2/ground_segmentation.py:27

  • The Method field was removed from the Condition model but the evaluator still relies on mode configuration; re-add or update the model to match expected scenario parameters.
    Method: Literal["annotated_pcd", "annotated_rosbag"]

@badai-nguyen
Copy link
Contributor

badai-nguyen commented May 28, 2025

@nanoshimarobot
Shall we remove all documantation and others related to annotatated_rosbag such as

@@ -6,14 +6,6 @@ Evaluate the performance of the Obstacle Segmentation sub-component in Autoware,

The Ground Truth data required for evaluation can be provided using the following two methods, and each can be used by changing the `Evaluation.Conditions.Method` of the scenario.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The Ground Truth data required for evaluation can be provided using the following two methods, and each can be used by changing the `Evaluation.Conditions.Method` of the scenario.
The Ground Truth data required for evaluation can be provided using the following method

Synchronize and subscribe to topics before and after ground removal, and evaluate the accuracy by comparing the number of points with ground and non-ground labels.

In this evaluation framework, the semantic labels are assumed to be recorded in an `INT32` field named `entity_id`.

### annotated_pcd

This method involves adding a field to the point cloud data provided as a dataset (`dataset/data/LIDAR_CONCAT/\*.pcd.bin`) to represent semantic labels.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This's also outdate and need to be updated

@badai-nguyen
Copy link
Contributor

badai-nguyen commented May 29, 2025

@nanoshimarobot could you also put the command you used and the evaluator testing result in description for reference?

@nanoshimarobot
Copy link
Contributor Author

@nanoshimarobot could you also put the command you used and the evaluator testing result in description for reference?

I updated the PR description to include the test command and a sample output.

@badai-nguyen
Copy link
Contributor

I also confirmed 'ground_segmentation_evaluator' works with latest lidar semantic segmentation dataset format

{"Condition": {"ground_label": 20, "obstacle_label": 7, "accuracy_min": 0.7, "accuracy_max": 1.0, "PassRate": 99.0}}
{"Result": {"Success": false, "Summary": "NoData"}, "Stamp": {"System": 1748506589.992203}, "Frame": {}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 1 / 1 -> 100.00"}, "Stamp": {"System": 1748506605.1093938, "ROS": 1736225810.2802138}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 96744, "FP": 0, "TN": 0, "FN": 1, "Accuracy": 0.9999896635485027, "Precision": 0.999999999999999, "Recall": 0.9999896635485027, "Specificity": 0.0, "F1-score": 0.9999948316975401}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 2 / 2 -> 100.00"}, "Stamp": {"System": 1748506607.7246006, "ROS": 1736225811.70285}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 96587, "FP": 0, "TN": 0, "FN": 0, "Accuracy": 0.999999999999999, "Precision": 0.999999999999999, "Recall": 0.999999999999999, "Specificity": 0.0, "F1-score": 0.9999999999499991}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 3 / 3 -> 100.00"}, "Stamp": {"System": 1748506610.6418622, "ROS": 1736225813.2828286}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 68419, "FP": 0, "TN": 0, "FN": 27880, "Accuracy": 0.7104850517658535, "Precision": 0.9999999999999986, "Recall": 0.7104850517658535, "Specificity": 0.0, "F1-score": 0.8307410240046619}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 4 / 4 -> 100.00"}, "Stamp": {"System": 1748506613.043979, "ROS": 1736225814.480516}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 71001, "FP": 0, "TN": 0, "FN": 25074, "Accuracy": 0.7390163934426222, "Precision": 0.9999999999999986, "Recall": 0.7390163934426222, "Specificity": 0.0, "F1-score": 0.8499245851698279}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 5 / 5 -> 100.00"}, "Stamp": {"System": 1748506615.5768092, "ROS": 1736225815.6578078}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 74154, "FP": 0, "TN": 0, "FN": 22028, "Accuracy": 0.7709758582686981, "Precision": 0.9999999999999987, "Recall": 0.7709758582686981, "Specificity": 0.0, "F1-score": 0.8706791282619379}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 6 / 6 -> 100.00"}, "Stamp": {"System": 1748506618.979621, "ROS": 1736225817.283224}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 72536, "FP": 0, "TN": 0, "FN": 23162, "Accuracy": 0.7579677736211825, "Precision": 0.9999999999999986, "Recall": 0.7579677736211825, "Specificity": 0.0, "F1-score": 0.862322717118701}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 7 / 7 -> 100.00"}, "Stamp": {"System": 1748506621.4693022, "ROS": 1736225818.6404264}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 68149, "FP": 0, "TN": 0, "FN": 27661, "Accuracy": 0.711293184427512, "Precision": 0.9999999999999986, "Recall": 0.711293184427512, "Specificity": 0.0, "F1-score": 0.8312931891023684}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 8 / 8 -> 100.00"}, "Stamp": {"System": 1748506624.3023686, "ROS": 1736225820.2779942}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 75367, "FP": 0, "TN": 0, "FN": 20916, "Accuracy": 0.7827653895287842, "Precision": 0.9999999999999987, "Recall": 0.7827653895287842, "Specificity": 0.0, "F1-score": 0.8781473929015133}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 9 / 9 -> 100.00"}, "Stamp": {"System": 1748506626.4814403, "ROS": 1736225821.3029618}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 75938, "FP": 0, "TN": 0, "FN": 20359, "Accuracy": 0.7885811603684426, "Precision": 0.9999999999999987, "Recall": 0.7885811603684426, "Specificity": 0.0, "F1-score": 0.8817952215955432}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 10 / 10 -> 100.00"}, "Stamp": {"System": 1748506628.569387, "ROS": 1736225822.3955405}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 83391, "FP": 0, "TN": 0, "FN": 13509, "Accuracy": 0.8605882352941168, "Precision": 0.9999999999999988, "Recall": 0.8605882352941168, "Specificity": 0.0, "F1-score": 0.9250711349486989}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 11 / 11 -> 100.00"}, "Stamp": {"System": 1748506630.630671, "ROS": 1736225823.438711}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 84998, "FP": 0, "TN": 0, "FN": 11512, "Accuracy": 0.880717024142575, "Precision": 0.9999999999999988, "Recall": 0.880717024142575, "Specificity": 0.0, "F1-score": 0.9365757982621201}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 12 / 12 -> 100.00"}, "Stamp": {"System": 1748506632.7102108, "ROS": 1736225824.455516}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 83125, "FP": 0, "TN": 0, "FN": 13464, "Accuracy": 0.8606052449036631, "Precision": 0.9999999999999988, "Recall": 0.8606052449036631, "Specificity": 0.0, "F1-score": 0.9250809619231922}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 13 / 13 -> 100.00"}, "Stamp": {"System": 1748506634.834357, "ROS": 1736225825.505582}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 83657, "FP": 0, "TN": 0, "FN": 13406, "Accuracy": 0.8618835189516079, "Precision": 0.9999999999999988, "Recall": 0.8618835189516079, "Specificity": 0.0, "F1-score": 0.9258189463867504}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 14 / 14 -> 100.00"}, "Stamp": {"System": 1748506636.852415, "ROS": 1736225826.56906}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 80532, "FP": 0, "TN": 0, "FN": 14688, "Accuracy": 0.8457466918714547, "Precision": 0.9999999999999988, "Recall": 0.8457466918714547, "Specificity": 0.0, "F1-score": 0.9164276935185579}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 15 / 15 -> 100.00"}, "Stamp": {"System": 1748506638.7968888, "ROS": 1736225827.578199}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 76139, "FP": 0, "TN": 0, "FN": 19893, "Accuracy": 0.7928502999000325, "Precision": 0.9999999999999987, "Recall": 0.7928502999000325, "Specificity": 0.0, "F1-score": 0.8844578935564421}}}}
{"Result": {"Success": true, "Summary": "Passed: Ground Segmentation (Success): 16 / 16 -> 100.00"}, "Stamp": {"System": 1748506640.7694693, "ROS": 1736225828.5481339}, "Frame": {"GroundSegmentation": {"Result": {"Total": "Success", "Frame": "Success"}, "Info": {"TP": 73900, "FP": 0, "TN": 0, "FN": 22359, "Accuracy": 0.7677204209476508, "Precision": 0.9999999999999987, "Recall": 0.7677204209476508, "Specificity": 0.0, "F1-score": 0.8685993687764901}}}}

@badai-nguyen badai-nguyen requested a review from MasatoSaeki May 29, 2025 08:28
@badai-nguyen
Copy link
Contributor

@nanoshimarobot plz fix ci(spell-check) error.

@nanoshimarobot
Copy link
Contributor Author

@nanoshimarobot plz fix ci(spell-check) error.

@badai-nguyen
All the spell-check errors were about lidarseg. Since the dataset directory is named lidarseg, do we really need to change it?

Warning: Unknown word (lidarseg)
docs/use_case/ground_segmentation.en.md:11:165 Unknown word (lidarseg)
docs/use_case/ground_segmentation.ja.md:11:79 Unknown word (lidarseg)
driving_log_replayer_v2/scripts/ground_segmentation_evaluator_node.py:58:78 Unknown word (lidarseg)
sample/ground_segmentation/scenario.ja.yaml:13:113 Unknown word (lidarseg)
sample/ground_segmentation/scenario.yaml:13:1[51](https://github.com/tier4/driving_log_replayer_v2/actions/runs/15319725058/job/43100585476?pr=156#step:3:57) Unknown word (lidarseg)
Warning: Unknown word (lidarseg)
Warning: Unknown word (lidarseg)
Warning: Unknown word (lidarseg)
Warning: Unknown word (lidarseg)
Files checked: 238, Issues found: 5 in 5 files.
Error: 5 spelling issues found in 5 of the 238 files checked.

@badai-nguyen badai-nguyen changed the title feat: update ground segmentation evaluator feat: update ground segmentation evaluator to be compatible with lidarseg dataset May 30, 2025
@MasatoSaeki
Copy link
Contributor

I will review it until next week, sorry for delay 🙇

@badai-nguyen
Copy link
Contributor

@MasatoSaeki Could you review this PR?

@MasatoSaeki
Copy link
Contributor

Sorry, just a minute!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants