Skip to content

Support polars DataFrames, LazyFrames #1373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Mar 15, 2024
Merged

Support polars DataFrames, LazyFrames #1373

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
d9873bd
add initial polars modules: classes, checks, tests
cosmicBboy Sep 23, 2023
d024b23
add builtin checks (#1408)
AndriiG13 Nov 10, 2023
f20ee75
Add polars engine dtypes (#1465)
FilipAisot Feb 19, 2024
71347be
Polars update ci (#1433)
cosmicBboy Feb 21, 2024
cf6b26e
Implement polars backend methods and DataFrameModel (#1431)
cosmicBboy Mar 8, 2024
5ee4655
Add Polars Builtin Check Tests (#1518)
AndriiG13 Mar 8, 2024
ab1333e
add polars LazyFrame generic type, element-wise checks, add docs (#1521)
cosmicBboy Mar 11, 2024
6b95059
use updated error handler logic
cosmicBboy Mar 11, 2024
7359eac
delete unused polars Series stub
cosmicBboy Mar 11, 2024
5396096
update banner link
cosmicBboy Mar 11, 2024
5d1f9fc
add pl.DataFrame to docs
cosmicBboy Mar 11, 2024
57b280d
add granular validation controls for polars
cosmicBboy Mar 12, 2024
6d765d5
update api reference with polars api
cosmicBboy Mar 12, 2024
f4b459b
add support for ellipsis in DataFrameModel.Config check with no args
cosmicBboy Mar 12, 2024
d25586c
Refector the polars type engine to use LazyFrames, add support for Ne…
cosmicBboy Mar 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ on:
branches:
- main
- dev
- polars-dev
- bugfix
- "release/*"

Expand Down
8 changes: 7 additions & 1 deletion .pylintrc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,10 @@ disable=
unnecessary-dunder-call,
use-dict-literal,
invalid-name,
import-outside-toplevel
import-outside-toplevel,
missing-class-docstring,
missing-function-docstring,
fixme,
too-many-locals,
redefined-outer-name,
logging-fstring-interpolation
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ requirements:
pip install -r requirements-dev.txt

docs-clean:
rm -rf docs/**/generated docs/**/methods docs/_build docs/source/_contents
rm -rf docs/source/reference/generated docs/**/generated docs/**/methods docs/_build docs/source/_contents

docs: docs-clean
python -m sphinx -E "docs/source" "docs/_build" && make -C docs doctest
python -m sphinx -W -E "docs/source" "docs/_build" && make -C docs doctest

quick-docs:
python -m sphinx -E "docs/source" "docs/_build" -W && \
Expand Down
7 changes: 5 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,9 @@ This is useful in production-critical or reproducible research settings. With

1. Define a schema once and use it to validate
[different dataframe types](https://pandera.readthedocs.io/en/stable/supported_libraries.html)
including [pandas](http://pandas.pydata.org), [dask](https://dask.org),
[modin](https://modin.readthedocs.io/), and [pyspark](https://spark.apache.org/docs/3.2.0/api/python/user_guide/pandas_on_spark/index.html).
including [pandas](http://pandas.pydata.org), [polars](https://docs.pola.rs/),
[dask](https://dask.org), [modin](https://modin.readthedocs.io/),
and [pyspark](https://spark.apache.org/docs/3.2.0/api/python/user_guide/pandas_on_spark/index.html).
1. [Check](https://pandera.readthedocs.io/en/stable/checks.html) the types and
properties of columns in a `DataFrame` or values in a `Series`.
1. Perform more complex statistical validation like
Expand Down Expand Up @@ -100,6 +101,7 @@ pip install pandera[modin] # validate modin dataframes
pip install pandera[modin-ray] # validate modin dataframes with ray
pip install pandera[modin-dask] # validate modin dataframes with dask
pip install pandera[geopandas] # validate geopandas geodataframes
pip install pandera[polars] # validate polars dataframes
```

</details>
Expand All @@ -120,6 +122,7 @@ conda install -c conda-forge pandera-modin # validate modin dataframes
conda install -c conda-forge pandera-modin-ray # validate modin dataframes with ray
conda install -c conda-forge pandera-modin-dask # validate modin dataframes with dask
conda install -c conda-forge pandera-geopandas # validate geopandas geodataframes
conda install -c conda-forge pandera-polars # validate polars dataframes
```

</details>
Expand Down
5 changes: 4 additions & 1 deletion ci/requirements-py3.10-pandas1.5.3-pydantic1.10.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ grpcio==1.58.0
# via ray
h11==0.14.0
# via uvicorn
hypothesis==6.98.9
hypothesis==6.98.10
identify==2.5.29
# via pre-commit
idna==3.4
Expand Down Expand Up @@ -296,6 +296,7 @@ pandas==1.5.3
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==1.5.2.221213
Expand All @@ -320,6 +321,7 @@ platformdirs==3.10.0
# virtualenv
pluggy==1.3.0
# via pytest
polars==0.20.10
pre-commit==3.4.0
prometheus-client==0.17.1
# via jupyter-server
Expand All @@ -340,6 +342,7 @@ pydantic==1.10.11
# via
# fastapi
# modin
# polars
# ray
pygments==2.16.1
# via
Expand Down
5 changes: 4 additions & 1 deletion ci/requirements-py3.10-pandas1.5.3-pydantic2.3.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ grpcio==1.58.0
# via ray
h11==0.14.0
# via uvicorn
hypothesis==6.98.9
hypothesis==6.98.10
identify==2.5.29
# via pre-commit
idna==3.4
Expand Down Expand Up @@ -298,6 +298,7 @@ pandas==1.5.3
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==1.5.2.221213
Expand All @@ -322,6 +323,7 @@ platformdirs==3.10.0
# virtualenv
pluggy==1.3.0
# via pytest
polars==0.20.10
pre-commit==3.4.0
prometheus-client==0.17.1
# via jupyter-server
Expand All @@ -342,6 +344,7 @@ pydantic==2.3.0
# via
# fastapi
# modin
# polars
# ray
pydantic-core==2.6.3
# via pydantic
Expand Down
5 changes: 4 additions & 1 deletion ci/requirements-py3.10-pandas2.0.3-pydantic1.10.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ grpcio==1.58.0
# via ray
h11==0.14.0
# via uvicorn
hypothesis==6.98.9
hypothesis==6.98.10
identify==2.5.29
# via pre-commit
idna==3.4
Expand Down Expand Up @@ -296,6 +296,7 @@ pandas==2.0.3
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==1.5.2.221213
Expand All @@ -320,6 +321,7 @@ platformdirs==3.10.0
# virtualenv
pluggy==1.3.0
# via pytest
polars==0.20.10
pre-commit==3.4.0
prometheus-client==0.17.1
# via jupyter-server
Expand All @@ -340,6 +342,7 @@ pydantic==1.10.11
# via
# fastapi
# modin
# polars
# ray
pygments==2.16.1
# via
Expand Down
5 changes: 4 additions & 1 deletion ci/requirements-py3.10-pandas2.0.3-pydantic2.3.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ grpcio==1.58.0
# via ray
h11==0.14.0
# via uvicorn
hypothesis==6.98.9
hypothesis==6.98.10
identify==2.5.29
# via pre-commit
idna==3.4
Expand Down Expand Up @@ -298,6 +298,7 @@ pandas==2.0.3
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==1.5.2.221213
Expand All @@ -322,6 +323,7 @@ platformdirs==3.10.0
# virtualenv
pluggy==1.3.0
# via pytest
polars==0.20.10
pre-commit==3.4.0
prometheus-client==0.17.1
# via jupyter-server
Expand All @@ -342,6 +344,7 @@ pydantic==2.3.0
# via
# fastapi
# modin
# polars
# ray
pydantic-core==2.6.3
# via pydantic
Expand Down
3 changes: 3 additions & 0 deletions ci/requirements-py3.10-pandas2.2.0-pydantic1.10.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,7 @@ pandas==2.2.0
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==2.2.0.240218
Expand All @@ -314,6 +315,7 @@ platformdirs==4.2.0
# virtualenv
pluggy==1.4.0
# via pytest
polars==0.20.10
pre-commit==3.6.2
prometheus-client==0.20.0
# via jupyter-server
Expand All @@ -333,6 +335,7 @@ pycparser==2.21
pydantic==1.10.11
# via
# fastapi
# polars
# ray
pygments==2.17.2
# via
Expand Down
3 changes: 3 additions & 0 deletions ci/requirements-py3.10-pandas2.2.0-pydantic2.3.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,7 @@ pandas==2.2.0
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==2.2.0.240218
Expand All @@ -316,6 +317,7 @@ platformdirs==4.2.0
# virtualenv
pluggy==1.4.0
# via pytest
polars==0.20.10
pre-commit==3.6.2
prometheus-client==0.20.0
# via jupyter-server
Expand All @@ -335,6 +337,7 @@ pycparser==2.21
pydantic==2.3.0
# via
# fastapi
# polars
# ray
pydantic-core==2.6.3
# via pydantic
Expand Down
5 changes: 4 additions & 1 deletion ci/requirements-py3.11-pandas1.5.3-pydantic1.10.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ grpcio==1.58.0
# via ray
h11==0.14.0
# via uvicorn
hypothesis==6.98.9
hypothesis==6.98.10
identify==2.5.29
# via pre-commit
idna==3.4
Expand Down Expand Up @@ -291,6 +291,7 @@ pandas==1.5.3
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==1.5.2.221213
Expand All @@ -315,6 +316,7 @@ platformdirs==3.10.0
# virtualenv
pluggy==1.3.0
# via pytest
polars==0.20.10
pre-commit==3.4.0
prometheus-client==0.17.1
# via jupyter-server
Expand All @@ -335,6 +337,7 @@ pydantic==1.10.11
# via
# fastapi
# modin
# polars
# ray
pygments==2.16.1
# via
Expand Down
5 changes: 4 additions & 1 deletion ci/requirements-py3.11-pandas1.5.3-pydantic2.3.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ grpcio==1.58.0
# via ray
h11==0.14.0
# via uvicorn
hypothesis==6.98.9
hypothesis==6.98.10
identify==2.5.29
# via pre-commit
idna==3.4
Expand Down Expand Up @@ -293,6 +293,7 @@ pandas==1.5.3
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==1.5.2.221213
Expand All @@ -317,6 +318,7 @@ platformdirs==3.10.0
# virtualenv
pluggy==1.3.0
# via pytest
polars==0.20.10
pre-commit==3.4.0
prometheus-client==0.17.1
# via jupyter-server
Expand All @@ -337,6 +339,7 @@ pydantic==2.3.0
# via
# fastapi
# modin
# polars
# ray
pydantic-core==2.6.3
# via pydantic
Expand Down
5 changes: 4 additions & 1 deletion ci/requirements-py3.11-pandas2.0.3-pydantic1.10.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ grpcio==1.58.0
# via ray
h11==0.14.0
# via uvicorn
hypothesis==6.98.9
hypothesis==6.98.10
identify==2.5.29
# via pre-commit
idna==3.4
Expand Down Expand Up @@ -291,6 +291,7 @@ pandas==2.0.3
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==1.5.2.221213
Expand All @@ -315,6 +316,7 @@ platformdirs==3.10.0
# virtualenv
pluggy==1.3.0
# via pytest
polars==0.20.10
pre-commit==3.4.0
prometheus-client==0.17.1
# via jupyter-server
Expand All @@ -335,6 +337,7 @@ pydantic==1.10.11
# via
# fastapi
# modin
# polars
# ray
pygments==2.16.1
# via
Expand Down
5 changes: 4 additions & 1 deletion ci/requirements-py3.11-pandas2.0.3-pydantic2.3.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ grpcio==1.58.0
# via ray
h11==0.14.0
# via uvicorn
hypothesis==6.98.9
hypothesis==6.98.10
identify==2.5.29
# via pre-commit
idna==3.4
Expand Down Expand Up @@ -293,6 +293,7 @@ pandas==2.0.3
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==1.5.2.221213
Expand All @@ -317,6 +318,7 @@ platformdirs==3.10.0
# virtualenv
pluggy==1.3.0
# via pytest
polars==0.20.10
pre-commit==3.4.0
prometheus-client==0.17.1
# via jupyter-server
Expand All @@ -337,6 +339,7 @@ pydantic==2.3.0
# via
# fastapi
# modin
# polars
# ray
pydantic-core==2.6.3
# via pydantic
Expand Down
3 changes: 3 additions & 0 deletions ci/requirements-py3.11-pandas2.2.0-pydantic1.10.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ pandas==2.2.0
# modin
# partd
# petl
# polars
# pyspark
# ray
pandas-stubs==2.2.0.240218
Expand All @@ -309,6 +310,7 @@ platformdirs==4.2.0
# virtualenv
pluggy==1.4.0
# via pytest
polars==0.20.10
pre-commit==3.6.2
prometheus-client==0.20.0
# via jupyter-server
Expand All @@ -328,6 +330,7 @@ pycparser==2.21
pydantic==1.10.11
# via
# fastapi
# polars
# ray
pygments==2.17.2
# via
Expand Down
Loading