Validation error_df does not exceed 10 rows depending on location of erroneous data

[elik90]

I found this issue while playing with the solution provided in this SO post: Pandera validate get all valid rows

I'm using 0.9.0 at the time of this post and have added some data to the example and found 2 interesting things

import pandera as pa
import pandas as pd
import numpy as np

schema: pa.DataFrameSchema = pa.DataFrameSchema(columns={
    'Col3': pa.Column(float, checks=pa.Check(lambda series: series.between(1, 2)), nullable=True),
    })

df: pd.DataFrame = pd.DataFrame({
    "Col3": [1.5, 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 10, 1.4, 20],
    })

try:
  schema(df, lazy=True)
except pa.errors.SchemaErrors as exc:
  print(f"shape of exc.data is : {exc.data.shape}")
  filtered_df = df[~df.index.isin(exc.failure_cases["index"])]
  error_df = df[df.index.isin(exc.failure_cases["index"])]

print(f"shape of original df: {df.shape}")

print(f"===== filtered_df =====")
print(f"shape of filtered_df: {filtered_df.shape}")
print(f"filtered_df:\n{filtered_df}")

print(f"===== error_df =====")
print(f"shape of error_df: {error_df.shape}")
print(f"error_df:\n{error_df}")

This outputs:

shape of exc.data is : (17, 1)
shape of original df: (17, 1)
===== filtered_df =====
shape of filtered_df: (7, 1)
filtered_df:
    Col3
0    1.5
11   1.0
12   1.1
13   1.2
14  10.0
15   1.4
16  20.0
===== error_df =====
shape of error_df: (10, 1)
error_df:
    Col3
1    0.0
2    0.1
3    0.2
4    0.3
5    0.4
6    0.5
7    0.6
8    0.7
9    0.8
10   0.9

1. I'm surprised to find that exc.data is the same as the original dataframe. I would've expected it to be the erroneous dataframe. This is more of a curiosity and not an issue since we can easily generate an error_df using the exc.failure_cases["index"]

2. error_df was capped to 10 rows even though there are other rows that didn't satisfy the check criteria of being between 1 and 2. Note the values 10 and 20 in the original dataframe. It appears that the location of these data points is the culprit for inclusion in the final filtered_df. Is there a way to retrieve the filtered_df and error_df so that they return the following rows?

filtered_df:
    Col3
0    1.5
11   1.0
12   1.1
13   1.2
15   1.4

===== error_df =====
error_df:
    Col3
1    0.0
2    0.1
3    0.2
4    0.3
5    0.4
6    0.5
7    0.6
8    0.7
9    0.8
10   0.9
14  10.0
16  20.0

[cosmicBboy]

Hi, this issue is addressed by #784, it should be out in the next release!