Skip to content

Improve NuScenes example with more geo data & blueprint #8130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Nov 14, 2024
Merged

Improve NuScenes example with more geo data & blueprint #8130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
53ebabc
add descriptions to scenes in minisplit
Wumpf Nov 13, 2024
ee518b8
log geo location directly on the car's entity
Wumpf Nov 13, 2024
e00aa02
add latlon to more things in nuscenes
Wumpf Nov 13, 2024
4cf74f7
Add annotations to 2D views in nuscenes
Wumpf Nov 13, 2024
d1fadbf
add trajectory of car on nuscenes map
Wumpf Nov 13, 2024
e225bc4
enable annotation context labels in nuscenes
Wumpf Nov 13, 2024
dcf3369
set default for image plane distance
Wumpf Nov 13, 2024
f1a56d2
fix axis length of vehicle
Wumpf Nov 13, 2024
6d3294e
collapse time panel, sort cameras
Wumpf Nov 14, 2024
9a230f6
update thumbnail
Wumpf Nov 14, 2024
a773aae
fix thumbnail dimensions
Wumpf Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 27 additions & 21 deletions examples/python/nuscenes_dataset/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
<!--[metadata]
title = "nuScenes"
tags = ["Lidar", "3D", "2D", "Object detection", "Pinhole camera", "Blueprint"]
thumbnail = "https://static.rerun.io/nuscenes/9c50bf5cadb879ef818ac3d35fe75696a9586cb4/480w.png"
thumbnail_dimensions = [480, 480]
thumbnail = "https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/480w.png"
thumbnail_dimensions = [480, 301]
channel = "release"
build_args = ["--seconds=5"]
-->

Visualize the [nuScenes dataset](https://www.nuscenes.org/) including lidar, radar, images, and bounding boxes data.

<picture data-inline-viewer="examples/nuscenes_dataset">
<img src="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/nuscenes/64a50a9d67cbb69ae872551989ee807b195f6b5d/1200w.png">
<picture>
<img src="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/nuscenes_dataset/3724a84d6e95f15a71db2ccc443fb67bfae58843/1200w.png">
</picture>

## Used Rerun types
Expand Down Expand Up @@ -83,6 +83,18 @@ rr.log(
)
```

#### GPS data

GPS data is calculated from the scene's reference coordinates and the transformations (starting map point + odometry).
The GPS coordinates are logged as [`GeoPoints`](https://www.rerun.io/docs/reference/types/archetypes/geopoints?speculative-link).

```python
rr.log(
"world/ego_vehicle",
rr.GeoPoints([[lat, long]]),
)
```

### LiDAR data
LiDAR data is logged as [`Points3D`](https://www.rerun.io/docs/reference/types/archetypes/points3d) archetype.
```python
Expand All @@ -101,18 +113,6 @@ Radar data is logged similar to LiDAR data, as [`Points3D`](https://www.rerun.io
rr.log(f"world/ego_vehicle/{sensor_name}", rr.Points3D(points, colors=point_colors))
```

### GPS data

GPS data is calculated from the scene's reference coordinates and the transformations (starting map point + odometry).
The GPS coordinates are logged as [`GeoPoints`](https://www.rerun.io/docs/reference/types/archetypes/geopoints?speculative-link).

```python
rr.log(
"world/ego_vehicle/gps",
rr.GeoPoints([[lat, long]]),
)
```

### Annotations

Annotations are logged as [`Boxes3D`](https://www.rerun.io/docs/reference/types/archetypes/boxes3d), containing details such as object positions, sizes, and rotation.
Expand All @@ -129,6 +129,8 @@ rr.log(
)
```

GPS coordinates are added to the annotations similarly to the vehicle.

### Setting up the default blueprint

The default blueprint for this example is created by the following code:
Expand All @@ -138,6 +140,10 @@ sensor_space_views = [
rrb.Spatial2DView(
name=sensor_name,
origin=f"world/ego_vehicle/{sensor_name}",
# Set the image plane distance to 5m for all camera visualizations.
defaults=[rr.components.ImagePlaneDistance(5.0)],
# TODO(#6670): Can't specify rr.components.FillMode.MajorWireframe right now, need to use batch type instead.
overrides={"world/anns": [rr.components.FillModeBatch("solid")]},
)
for sensor_name in nuscene_sensor_names(nusc, args.scene_name)
]
Expand All @@ -147,7 +153,7 @@ blueprint = rrb.Vertical(
rrb.Vertical(
rrb.TextDocumentView(origin="description", name="Description"),
rrb.MapView(
origin="world/ego_vehicle/gps",
origin="world",
name="MapView",
zoom=rrb.archetypes.MapZoom(18.0),
background=rrb.archetypes.MapBackground(rrb.components.MapProvider.OpenStreetMap),
Expand Down
87 changes: 59 additions & 28 deletions examples/python/nuscenes_dataset/nuscenes_dataset/__main__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ def ensure_scene_available(root_dir: pathlib.Path, dataset_version: str, scene_n
raise ValueError(f"{scene_name=} not found in dataset")


def nuscene_sensor_names(nusc: nuscenes.NuScenes, scene_name: str) -> set[str]:
def nuscene_sensor_names(nusc: nuscenes.NuScenes, scene_name: str) -> list[str]:
"""Return all sensor names in the scene."""

sensor_names = set()
Expand All @@ -75,7 +75,16 @@ def nuscene_sensor_names(nusc: nuscenes.NuScenes, scene_name: str) -> set[str]:
sensor_names.add(sensor_name)
current_camera_token = sample_data["next"]

return sensor_names
# For a known set of cameras, order the sensors in a circle.
ordering = {
"CAM_FRONT_LEFT": 0,
"CAM_FRONT": 1,
"CAM_FRONT_RIGHT": 2,
"CAM_BACK_RIGHT": 3,
"CAM_BACK": 4,
"CAM_BACK_LEFT": 5,
}
return sorted(list(sensor_names), key=lambda sensor_name: ordering.get(sensor_name, float("inf")))


def log_nuscenes(nusc: nuscenes.NuScenes, scene_name: str, max_time_sec: float) -> None:
Expand Down Expand Up @@ -110,7 +119,7 @@ def log_nuscenes(nusc: nuscenes.NuScenes, scene_name: str, max_time_sec: float)
log_lidar_and_ego_pose(location, first_lidar_token, nusc, max_timestamp_us)
log_cameras(first_camera_tokens, nusc, max_timestamp_us)
log_radars(first_radar_tokens, nusc, max_timestamp_us)
log_annotations(first_sample_token, nusc, max_timestamp_us)
log_annotations(location, first_sample_token, nusc, max_timestamp_us)


def log_lidar_and_ego_pose(
Expand All @@ -119,6 +128,8 @@ def log_lidar_and_ego_pose(
"""Log lidar data and vehicle pose."""
current_lidar_token = first_lidar_token

ego_trajectory_lat_lon = []

while current_lidar_token != "":
sample_data = nusc.get("sample_data", current_lidar_token)
sensor_name = sample_data["channel"]
Expand All @@ -131,23 +142,29 @@ def log_lidar_and_ego_pose(

ego_pose = nusc.get("ego_pose", sample_data["ego_pose_token"])
rotation_xyzw = np.roll(ego_pose["rotation"], shift=-1) # go from wxyz to xyzw
position_lat_lon = derive_latlon(location, ego_pose)
ego_trajectory_lat_lon.append(position_lat_lon)

rr.log(
"world/ego_vehicle",
rr.Transform3D(
translation=ego_pose["translation"],
rotation=rr.Quaternion(xyzw=rotation_xyzw),
axis_length=10.0, # The length of the visualized axis.
from_parent=False,
),
rr.GeoPoints(lat_lon=position_lat_lon, radii=rr.Radius.ui_points(8.0), colors=0xFF0000FF),
)
current_lidar_token = sample_data["next"]

# log GPS data
(lat, long) = derive_latlon(location, ego_pose)
# TODO(#6889): We don't want the radius for the trajectory line to be the same as the radius of the points.
# However, rr.GeoPoints uses the same `rr.components.Radius` for this, so these two archetypes would influence each other
# if logged on the same entity. In the future, they will have different tags, which will allow them to live side by side.
rr.log(
"world/ego_vehicle/gps",
rr.GeoPoints(lat_lon=[[lat, long]]),
"world/ego_vehicle/trajectory",
rr.GeoLineStrings(lat_lon=ego_trajectory_lat_lon, radii=rr.Radius.ui_points(1.0), colors=0xFF0000FF),
)

current_lidar_token = sample_data["next"]

data_file_path = nusc.dataroot / sample_data["filename"]
pointcloud = nuscenes.LidarPointCloud.from_file(str(data_file_path))
points = pointcloud.points[:3].T # shape after transposing: (num_points, 3)
Expand Down Expand Up @@ -193,7 +210,7 @@ def log_radars(first_radar_tokens: list[str], nusc: nuscenes.NuScenes, max_times
current_camera_token = sample_data["next"]


def log_annotations(first_sample_token: str, nusc: nuscenes.NuScenes, max_timestamp_us: float) -> None:
def log_annotations(location: str, first_sample_token: str, nusc: nuscenes.NuScenes, max_timestamp_us: float) -> None:
"""Log 3D bounding boxes."""
label2id: dict[str, int] = {}
current_sample_token = first_sample_token
Expand All @@ -207,6 +224,7 @@ def log_annotations(first_sample_token: str, nusc: nuscenes.NuScenes, max_timest
centers = []
quaternions = []
class_ids = []
lat_lon = []
for ann_token in ann_tokens:
ann = nusc.get("sample_annotation", ann_token)

Expand All @@ -218,6 +236,7 @@ def log_annotations(first_sample_token: str, nusc: nuscenes.NuScenes, max_timest
if ann["category_name"] not in label2id:
label2id[ann["category_name"]] = len(label2id)
class_ids.append(label2id[ann["category_name"]])
lat_lon.append(derive_latlon(location, ann))

rr.log(
"world/anns",
Expand All @@ -226,14 +245,13 @@ def log_annotations(first_sample_token: str, nusc: nuscenes.NuScenes, max_timest
centers=centers,
quaternions=quaternions,
class_ids=class_ids,
fill_mode=rr.components.FillMode.Solid,
),
rr.GeoPoints(lat_lon=lat_lon),
)
current_sample_token = sample_data["next"]

# skipping for now since labels take too much space in 3D view (see https://github.com/rerun-io/rerun/issues/4451)
# annotation_context = [(i, label) for label, i in label2id.items()]
# rr.log("world/anns", rr.AnnotationContext(annotation_context), static=True)
annotation_context = [(i, label) for label, i in label2id.items()]
rr.log("world/anns", rr.AnnotationContext(annotation_context), static=True)


def log_sensor_calibration(sample_data: dict[str, Any], nusc: nuscenes.NuScenes) -> None:
Expand Down Expand Up @@ -296,26 +314,39 @@ def main() -> None:
rrb.Spatial2DView(
name=sensor_name,
origin=f"world/ego_vehicle/{sensor_name}",
contents=["$origin/**", "world/anns"],
# TODO(#6670): Can't specify rr.components.FillMode.MajorWireframe right now, need to use batch type instead.
overrides={"world/anns": [rr.components.FillModeBatch("majorwireframe")]},
)
for sensor_name in nuscene_sensor_names(nusc, args.scene_name)
]
blueprint = rrb.Vertical(
rrb.Horizontal(
rrb.Spatial3DView(name="3D", origin="world"),
rrb.Vertical(
rrb.TextDocumentView(origin="description", name="Description"),
rrb.MapView(
origin="world/ego_vehicle/gps",
name="MapView",
zoom=rrb.archetypes.MapZoom(18.0),
background=rrb.archetypes.MapBackground(rrb.components.MapProvider.OpenStreetMap),
blueprint = rrb.Blueprint(
rrb.Vertical(
rrb.Horizontal(
rrb.Spatial3DView(
name="3D",
origin="world",
# Set the image plane distance to 5m for all camera visualizations.
defaults=[rr.components.ImagePlaneDistance(5.0)],
# TODO(#6670): Can't specify rr.components.FillMode.MajorWireframe right now, need to use batch type instead.
overrides={"world/anns": [rr.components.FillModeBatch("solid")]},
),
rrb.Vertical(
rrb.TextDocumentView(origin="description", name="Description"),
rrb.MapView(
origin="world",
name="MapView",
zoom=rrb.archetypes.MapZoom(18.0),
background=rrb.archetypes.MapBackground(rrb.components.MapProvider.OpenStreetMap),
),
row_shares=[1, 1],
),
row_shares=[1, 1],
column_shares=[3, 1],
),
column_shares=[3, 1],
rrb.Grid(*sensor_space_views),
row_shares=[4, 2],
),
rrb.Grid(*sensor_space_views),
row_shares=[4, 2],
rrb.TimePanel(state="collapsed"),
)

rr.script_setup(args, "rerun_example_nuscenes", default_blueprint=blueprint)
Expand Down
20 changes: 10 additions & 10 deletions examples/python/nuscenes_dataset/nuscenes_dataset/download_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,16 @@
import tqdm

MINISPLIT_SCENES = [
"scene-0061",
"scene-0103",
"scene-0553",
"scene-0655",
"scene-0757",
"scene-0796",
"scene-0916",
"scene-1077",
"scene-1094",
"scene-1100",
"scene-0061", # Driving around a corner.
"scene-0103", # Driving straight on a city road.
"scene-0553", # Standing in front of a traffic light.
"scene-0655", # Drive straight only.
"scene-0757", # Goes straight rather slowly.
"scene-0796", # Drive straight only.
"scene-0916", # Driving in a circle on a parking lot.
"scene-1077", # Straight drive on an artery road at night.
"scene-1094", # Slow drive at night.
"scene-1100", # Standing on front of traffic light at night.
]
MINISPLIT_URL = "https://www.nuscenes.org/data/v1.0-mini.tgz"

Expand Down
Loading