feat(autoware_ptv3): implemented an inference node for ptv3 using tensorrt

[knzo25]

Description

This PR implements an inference node for Point Transformer V3 (PTv3).

On a 10cm grid in the range of -76m - 76m, the processing times are

• ~1ms preprocessing
• 25-30ms inference (although this is for a blackwell card)

Related links

Branch containing all the PRs needed for inference

• https://github.com/knzo25/autoware.universe/tree/feat/ptv3

Related PRs:

• Generate lidar segmentation pseudo labels from SAM: feat(PCSegSAM2): auto pointcloud segmentation using SAM2 and GroundingDino tier4/AWML#44
• Train and export PTv3: feat(ptv3): add a lidar segmentation model with onnx support tier4/AWML#45

Parent Issue:

• LiDAR semantic segmentation for Autoware #10481

Required PRs before this one can be merged:

• feat(autoware_tensorrt_plugins): implemented a tensorrt plugin for the argsort operation #10601
• feat(autoware_tensorrt_plugins): implemented a tensorrt plugin for the segment_csr operation #10602
• feat(autoware_tensorrt_plugins): implemented tensorrt plugins for the native spconv algorithms #10603
• feat(autoware_tensorrt_plugins): implemented a tensorrt plugin for the unique operation #10604

How was this PR tested?

Notes for reviewers

None.

Interface changes

None.

Effects on system behavior

None.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[knzo25]

@amadeuszsz @scepter914
I was testing this with a model trained for nuscenes, and have the following general comments:

• Inference time is dominated by sparse convolution operations, unique/sort operations, and multi head attention.
• I believe the network is bigger than really needed, so the inference time could be reduced (spconv / mha) if less features or a shallower network is used
• Unique/argsort can only be accelerated using less points, 32-bit hashes, fusing unique and argsort, or looking for a better algorithm
• I am not really sure, but looking at the kernels executed by tensorrt, I did not see fa_mha so flash attention may not be used as of now. This could be looked further, implement some fa kernels, or using part of the TensorRT made for LLMs.
• While expected, inference in fp16 work visually worse than fp32. Probably the network needs to be trained int fp16.
• In my basic tests, I think for segmentation there is not much gain in using the whole 120m range or 5cm voxels (the original network was designed for 5cm, which may be the reason that training and inferring for 10cm makes it seem that the network is overdimensioned).

Can you please upload the models for t4dataset that I left? (If you can retrain for 76m - 10cm that would be awesome too 🙏 )

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 429 lines in your changes missing coverage. Please review.

Project coverage is 15.78%. Comparing base (0b9bde4) to head (d361dea).

Files with missing lines	Patch %	Lines
perception/autoware_ptv3/lib/ptv3_trt.cpp	0.00%	180 Missing ⚠️
.../autoware_ptv3/lib/preprocess/preprocess_kernel.cu	0.00%	86 Missing ⚠️
perception/autoware_ptv3/src/ptv3_node.cpp	0.00%	76 Missing ⚠️
...utoware_ptv3/include/autoware/ptv3/ptv3_config.hpp	0.00%	48 Missing ⚠️
...utoware_ptv3/lib/postprocess/postprocess_kernel.cu	0.00%	33 Missing ⚠️
...tion/autoware_ptv3/include/autoware/ptv3/utils.hpp	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10600      +/-   ##
==========================================
- Coverage   15.89%   15.78%   -0.11%     
==========================================
  Files        1347     1356       +9     
  Lines      100088   100746     +658     
  Branches    32887    32981      +94     
==========================================
  Hits        15907    15907              
- Misses      71982    72640     +658     
  Partials    12199    12199              

Flag	Coverage Δ		*Carryforward flag
daily	17.32% <ø> (ø)		Carriedforward from 0b9bde4
daily-cuda	15.95% <ø> (ø)		Carriedforward from 0b9bde4
differential-cuda	0.00% <0.00%> (?)
total-cuda	15.72% <ø> (ø)		Carriedforward from 0b9bde4

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[knzo25]

@amadeuszsz
Thank you for your help with the plugins' PRs.
This branch should be able to be tested directly now

[amadeuszsz]

Thank you for this great PR! Initial review addressed, but could you please share ONNX file as I was not able to deploy your .pth model? I would like to check the runtime and try to crash it, check the latency etc.

[knzo25]

@amadeuszsz
To share the onnx I would need the pth, but essentially I ran out of time. Hopefully, I will manage to apply your review comments, but that may be as far as I can go

[amadeuszsz]

@amadeuszsz To share the onnx I would need the pth, but essentially I ran out of time. Hopefully, I will manage to apply your review comments, but that may be as far as I can go

Sure, I understand. After checking the runtime I can go with fixes by myself, of course if you allow me to work on your PR 🙇🏻‍♂️

[knzo25]

@amadeuszsz
I think I addressed (or attempted do) all the comments. Sadly, CI/CD does not pass, though I do not know why.
Please, try to add any fixes if you have time 🙏

[amadeuszsz]

I think I addressed (or attempted do) all the comments. Sadly, CI/CD does not pass, though I do not know why.
Please, try to add any fixes if you have time 🙏

@knzo25
Thanks for addressing all the comments! I will approve this PR as soon as I can get the ONNX and confirm the runtime.