Skip to content

add mitigation for empty bounding boxes #6923

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 7, 2022
Merged

Conversation

pmeier
Copy link
Collaborator

@pmeier pmeier commented Nov 7, 2022

As part of our performance optimizations we introduced torch.aminmax for bounding boxes in a few places. This works for degenerate inputs on CPU, but fails on CUDA:

>>> t = torch.rand(0, 4)
>>> torch.aminmax(t, dim=-1)
torch.return_types.aminmax(
min=tensor([]),
max=tensor([]))
>>> torch.aminmax(t.cuda(), dim=-1)
RuntimeError: iter.numel() > 0 && iter.ntensors() - iter.noutputs() == 1 && iter.noutputs() >= 1 INTERNAL ASSERT FAILED at "../aten/src/ATen/native/cuda/Reduce.cuh":1134, please report a bug to PyTorch.

This already has a report in pytorch/pytorch#85439. The consensus seems to be to also let the CPU variant raise an error. Thus, to be safe, we should simply exit early for degenerate inputs like we often do for images as well.

This error never surfaced in CI since we currently don't have CUDA CI for prototypes. I made sure it works locally.

cc @vfdev-5 @datumbox @bjuncek

Copy link
Contributor

@datumbox datumbox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@pmeier pmeier merged commit 81a2e10 into pytorch:main Nov 7, 2022
@pmeier pmeier deleted the empty-boxes-cuda branch November 7, 2022 19:13
facebook-github-bot pushed a commit that referenced this pull request Nov 14, 2022
Reviewed By: NicolasHug

Differential Revision: D41265191

fbshipit-source-id: f0fdaf2cd6155cdb309edfaae25c486f2352a2d5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants