Skip to content

Implement I/O for parquet files #307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
niksirbi opened this issue Sep 12, 2024 · 1 comment · May be fixed by #562
Open

Implement I/O for parquet files #307

niksirbi opened this issue Sep 12, 2024 · 1 comment · May be fixed by #562
Labels
enhancement New optional feature

Comments

@niksirbi
Copy link
Member

Is your feature request related to a problem? Please describe.
This is mainly to facilitate data exchange with @roaldarbol's animovement R package, which represents data in a tidy dataframe. Read previous discussion on zulip.

Describe the solution you'd like
This would necessitate implementing two new I/O functions:

  • load_poses.from_tidy_df()
  • save_poses.to_tidy_df()

The tidy dataframe could be a pandas version of the table used as the primary data structure by animovement.
After that, we can rely on existing pandas to_parquet and read_parquet methods.

Describe alternatives you've considered
We could also consider wrapping the above functions into load_poses.from_animovement_file and to_animovement_file, which will do both things:

  • read/write parquet from/to a pandas Dataframe (using the aforementioned native pandas functions)
  • convert between a tidy dataframe and a movement xarray dataset (using the two new functions proposed above).

This is similar to how we handle DeepLabCut dataframes and files.

Additional context
Having the ability to convert movement datasets into 2D "tidy" format unlocks all sorts of new possibilities of saving them to formats optimised for "tables" Having the dataset in this form (where every variable is a columns) also makes it easier to use certain plotting libraries, like seaborn.

@niksirbi niksirbi added the enhancement New optional feature label Sep 12, 2024
@niksirbi niksirbi changed the title Implement I/O for parquet file Implement I/O for parquet files Sep 12, 2024
@ShigrafS
Copy link

@niksirbi I'm working on this.

@ShigrafS ShigrafS linked a pull request Apr 27, 2025 that will close this issue
7 tasks
@niksirbi niksirbi moved this from 🤔 Triage to 🚧 In Progress in movement progress tracker May 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New optional feature
Projects
Status: 🚧 In Progress
Development

Successfully merging a pull request may close this issue.

2 participants