Skip to content

DataFusion queries against new catalog #9367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

timsaucer
Copy link
Contributor

Related

Closes https://github.com/rerun-io/dataplatform/issues/407

What

This PR includes methods for turning both request/response and streaming services from our GRPC definitions into DataFusion table providers. It expands the previous work and supersedes #9320

This code creates two different traits for conversion of either single response or streams of data into table providers.

Copy link

github-actions bot commented Mar 24, 2025

Web viewer built successfully. If applicable, you should also test it:

  • I have tested the web viewer
Result Commit Link Manifest
25a8bc4 https://rerun.io/viewer/pr/9367 +nightly +main

Note: This comment is updated whenever you push a commit.

@timsaucer timsaucer force-pushed the tsaucer/new-catalog-datafusion branch 2 times, most recently from d1cf2d8 to 88640b0 Compare March 26, 2025 14:54
@timsaucer timsaucer added include in changelog dataplatform Rerun Data Platform integration ⛃ re_datastore affects the datastore itself labels Mar 26, 2025
@timsaucer timsaucer force-pushed the tsaucer/new-catalog-datafusion branch from 4716df3 to 42ce89b Compare March 27, 2025 16:31
jleibs added a commit that referenced this pull request Apr 2, 2025
### Related

Depends on:
- #9416 which should be merged
first

### What

I hijacked @timsaucer 's previous work from:
- #9367

This now goes through a generic TableEntry listing with both schema and
table provided by the server.

---------

Co-authored-by: Tim Saucer <[email protected]>
Co-authored-by: Antoine Beyeler <[email protected]>
jprochazk pushed a commit that referenced this pull request Apr 2, 2025
### Related

Depends on:
- #9416 which should be merged
first

### What

I hijacked @timsaucer 's previous work from:
- #9367

This now goes through a generic TableEntry listing with both schema and
table provided by the server.

---------

Co-authored-by: Tim Saucer <[email protected]>
Co-authored-by: Antoine Beyeler <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dataplatform Rerun Data Platform integration include in changelog ⛃ re_datastore affects the datastore itself
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant