Pandas nullable String dtype is not recognized as a Pandera String

[gwerbin-tive]

Describe the bug

A column containing the 'string' dtype (i.e. pandas.StringDtype) is not valid for the Pandera semantic tring data type, pandera.dtypes.String.

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandera.
• (optional) I have confirmed this bug exists on the master branch of pandera.

I attempted to search for this issue, but might have missed it. Sorry if I did!

Code Sample, a copy-pastable example

import pandas as pd
import pandera as pa

schema = pa.DataFrameSchema(
    ['color': pa.Column(pa.dtypes.String)]
)

data = pd.Series(['red', 'green', 'blue'], dtype='string').to_frame('color')

schema.validate(data)

Expected behavior

I expected this schema to pass validation successfully.

Desktop (please complete the following information):

• MacOS 12.4 (ARM64)
• Pandera v0.13.4

[cosmicBboy]

hi @gwerbin-tive, you'll need to use pandera.STRING here, since you want to use the pandera-native string type. See here for all the dtype aliases defined by pandera (pandera.String is the numpy string type).

In general the recommended way of doing this is to use the pd.StringDtype() directly or use the string alias "string". If you want to use the pandera datatype use pandera.STRING, which is just an alias of this.

Need to work on better datatype docs!

[gwerbin-tive]

Thank you for clarifying @cosmicBboy.

I was hoping to support either one of pandas.StringDtype or "object"-containing-str as valid inputs. Does the "string" alias support that?

Also, I am somewhat surprised that pandera.dtypes.String does not (is not intended to?) support pd.StringDtype. I assume there's a good technical reason for that.

[cosmicBboy]

so pandas.dtypes.String is an alias for the str numpy type (i.e. object). pandas.engines.pandas_engine.STRING is the pandas-native string type.

This is simply by definition in the pandera API.

Similarly, pandas.dtypes.Int maps to the numpy integer type, while pandas.engines.pandas_engine.INT64 maps to the pandas-native nullable integer type.

[cosmicBboy]

I was hoping to support either one of pandas.StringDtype or "object"-containing-str as valid inputs. Does the "string" alias support that?

In pandera one needs to be precise about the types. The pandera schema follow whatever conventions are set by the underlying framework (pandas in this case). So the "string" alias literally is just an alias for pd.StringDtype(), and "str" or str is just an alias for the numpy string type.

You'll have to create your own custom data type if you want to define a string type that can either be a pandas-native StringDtype OR a numpy str (object) dtype.

[gwerbin-tive]

Thanks for the clarification, but I was surprised that the Pandera "semantic" types are intended to be 1:1 mappings with underlying physical data types.

Based on my reading of the documentation and of their designation as "semantic", I thought would they have the exact opposite property: to be abstract semantic types, without correspondence to any physical storage type. So the Pandera string type would represent all valid string data, the integer type would represent all valid integer data, etc.

[cosmicBboy]

Converting this issue to a discussion. @gwerbin-tive would you mind marking the appropriate response as the answer?