PyPMML is making incorrect predictions for XGBoost pipeline

[shivangi-ph]

Hi,

I keep getting incorrect predictions when using the PMML-generated pipeline for inference. The pipeline works correctly in Python when using pipeline.predict_proba(), but after converting it to PMML using sklearn2pmml or using command line to convert pickle to pmml and loading it for prediction using pypmml or jpmml-evaluator-python in python, the predictions do not match the expected Python output.

I have narrowed the issue down to the first step in my four-step pipeline, which is responsible for feature engineering. Here is my first step:

def num_exp(x):
    if (x >=0.0 and x!=999.0) or x ==-999999999.0 or x is None: 
        return x
    else: 
        return 999999999.0
    
def special_exp(x):
    if (x < 0.0 or x == 999.0 or x is None):
        return x
    else: 
        return 999999999.0

def create_dataframe_mapper(dataFrame,special_value_cols,categorical_cols_raw):
    transformers = []
    for column in dataFrame.columns:
        if column in special_value_cols:

            special_expr = ExpressionTransformer(Expression("special_exp(X[0])",function_defs = [special_exp]))
            numerical_expr = ExpressionTransformer(Expression("num_exp(X[0])",function_defs = [num_exp]))

            transformers.append(([column], [ContinuousDomain(dtype = float,data_min= -999999999, data_max = 9999999999, missing_value_replacement =  -999999999.0  ),numerical_expr],{"alias":f"{column}_num"}))
            transformers.append(([column], [ContinuousDomain(dtype = float,data_min= -999999999, data_max = 9999999999,  missing_value_replacement =  -999999999.0 ),special_expr],{"alias":f"{column}_special_values"}))
        else:

            passthrough = ExpressionTransformer(expr="X[0]")
            transformers.append(([column], passthrough))

        newcol_mapper = DataFrameMapper(transformers, df_out=True)
    return newcol_mapper,transformers

Specifically, the first step modifies one column, x, and generates two new columns: x_num and x_special_values. The logic that x should follow is shown below and an example of how it should work is also shown in the table below:

x_num: If x >= 0 and x != 999, return x. Else, return 999999999. Leave missing values unchanged.
x_special_values: If x < 0 or x == 999, return x. Else, return 999999999. Leave missing values unchanged.

<style> </style>

x	x_num	x_special_values
1.0	1.0	999999999
2.0	2.0	999999999
-1.0	999999999	-1.0
NaN	NaN	NaN
999	999999999	999

For the second part (else condition), where I leave columns unchanged, I wanted to go with dataframe mapper so I just did a workaround to use Expression Transformer to return column values as they are.

I have tried the following approach as well to generate new columns, which doesn’t work either.

special_expr = ExpressionTransformer(Expression("special_exp(X[0])",function_defs = [special_exp]),map_missing_to = -999999999.0)
 numerical_expr = ExpressionTransformer(Expression("num_exp(X[0])",function_defs = [num_exp]),map_missing_to = -999999999.0) 

transformers.append(([column], numerical_expr, {"alias":f"{column}_num"}))
transformers.append(([column], special_expr, {"alias":f"{column}_special_values"}))

Also tried using inline string instead of UDF but that doesn’t work either.

This is example of one of the columns in pmml file if that helps. I am not too familiar with PMML so I am not sure if this is working as it is supposed to.

<DerivedField name="x_num" optype="continuous" dataType="double">
							<Apply function="if">
								<Apply function="and">
									<Apply function="greaterOrEqual">
										<FieldRef field="x"/>
										<Constant dataType="double">0.0</Constant>
									</Apply>
									<Apply function="notEqual">
										<FieldRef field="x"/>
										<Constant dataType="double">999.0</Constant>
									</Apply>
								</Apply>
								<FieldRef field="x"/>
								<Apply function="if">
									<Apply function="equal">
										<FieldRef field="x"/>
										<Constant missing="true"/>
									</Apply>
									<Constant dataType="double">-9.99999999E8</Constant>
									<Constant dataType="double">9.99999999E8</Constant>

								</Apply>
							</Apply>

The rest of my pipeline has two more dataframe mapper steps which use the columns generated in the first step and then use it for predictions as shown below.

initial_mapper, transfomrer1= create_dataframe_mapper(df_train_pipeline, special_value_cols,categorical_cols_raw)
second_mapper, transfomrer2= feature_eng(fteng_cols,all_cols)
final_mapper,transformer4 = create_final_mapper(binning_process, order_cols, all_num_cols2, all_cat_cols2,categorical_variables)
pipeline_mod = PMMLPipeline([

    ("mapper", initial_mapper),
    ("second_mapper", second_mapper),  
    ("final_mapper", final_mapper),
    ("classifier",model)
])

My question is where are things going wrong in PMML. When I don't do the first step as part of my pipeline, my pmml and python predictions match. I can see output of first step being correct in python as well. Is there a known issue with how sklearn2pmml translates the transformation logic? Any guidance or fix would be greatly appreciated.

Environment:
sklearn2pmml version: 0.112.1.post1
pypmml version: 1.5.6
Python version: 3.12.7

[vruusmann]

How am I supposed to debug this pipeline? Would need a sample model (a pickle dump) plus some data.

Is there a known issue with how sklearn2pmml translates the transformation logic?

In PMML missing values (None) and invalid values (NaN) are two different concepts. In Python data science, there is no clear concept of a missing value, with NaN acting as its surrogate.

With that guidance in mind, can you verify that:

1. The transformer works correctly (ie. predictions match between PMML and Python) for normal input values. By "normal" I mean regular in-domain values such as 1, 2, 3 etc.
2. Do the same for special input values such as 999 and -999999999.
3. Do the same for missing value of None.
4. Do the same for invalid value of (numpy.)NaN.

Are they all broken? Or some work while the others don't? That's your lead.

This is example of one of the columns in pmml file if that helps. I am not too familiar with PMML so I am not sure if this is working as it is supposed to.

The following fragment is strange (it's not outright incorrect, but it definitely looks like no-op and/or undefined):

<Apply function="equal">
	<FieldRef field="x"/>
	<Constant missing="true"/>
</Apply>

I'm not sure that it fill fire correctly for (numpy.)NaN.

This piece of PMML is generated based on the x is None sub-expression. You better re-write it as pandas.isnull(x), which should handle uniformly all None, (numpy.)NaN, pandas.NA etc values.

Also, it's not advisable to perform missingness check inside Python expressions. They should be moved to the parent level by specifying the ExpressionTransformer.map_missing_to attribute (there's a recent issue about this).

[shivangi-ph]

Hi,

Thank you for your response. I apologize for not sharing any samples. The thing is I can't really share my data or PMML as it is. I can try to create some dummy data and model so that you are able to reproduce it on your end but I would like to keep this as last resort. Having said that, I did try to verify if it was certain values that were providing wrong predictions and it seems like all the steps are broken and there is no clear pattern.

I also followed your other suggestions and followed your advice on this issue and made the following changes to my code (tried the if else in a different way and took out the condition for checking for missing values) but still getting the wrong predictions. Would you be able to verify if PMML looks fine on this new piece of code. I don't know if it helps but my model is an XGBoost model.

Here is my code and PMML part for it

def num_exp(x):
    if (x < 0.0 or x==999.0): 
        return 999999999
    else: 
        return x
    
def special_exp(x):
    if (x >= 0.0 and x != 999.0):
        return 999999999
    else:
        return x

def create_dataframe_mapper(dataFrame,special_value_cols,categorical_cols_raw):
    transformers = []
    for column in dataFrame.columns:
        if column in special_value_cols:
            special_expr = ExpressionTransformer(Expression("special_exp(X[0])",function_defs = [special_exp]),map_missing_to = -999999999.0)
            numerical_expr = ExpressionTransformer(Expression("num_exp(X[0])",function_defs = [num_exp]),map_missing_to = -999999999.0)

            transformers.append(([column], numerical_expr {"alias":f"{column}_num"}))
            transformers.append(([column], special_expr, {"alias":f"{column}_special_values"}))

        else:
            passthrough = ExpressionTransformer(expr="X[0]")
            transformers.append(([column], passthrough))

        newcol_mapper = DataFrameMapper(transformers, df_out=True)

PMML:

<DerivedField name="x_num" optype="continuous" dataType="double">
							<Apply function="if" mapMissingTo="-9.99999999E8">
								<Apply function="or">
									<Apply function="lessThan">
										<FieldRef field="x"/>
										<Constant dataType="double">0.0</Constant>
									</Apply>
									<Apply function="equal">
										<FieldRef field="x"/>
										<Constant dataType="double">999.0</Constant>
									</Apply>
								</Apply>
								<Constant dataType="double">9.99999999E8</Constant>
								<FieldRef field="x"/>
							</Apply>
						</DerivedField>

[vruusmann]

I don't know if it helps but my model is an XGBoost model.

Try replacing the XGBoost estimator with some Scikit-Learn estimator, and re-run your experiment.

Use the simplest algorithm possible such as LinearRegression or LogisticRegression. If your dataset is sparse, then append Imputer to your pre-processing pipeline to make linear models work.

The thing is that XGBoost estimators are sensitive towards missing values. So, try to get rid of that "variable" in your debugging configuration.

Also, PyPMML is known to be very bad at dealing with sparse datasets, because it does not discriminate between None (missing) and (numpy.)NaN values (invalid) internally. Be sure to debug using the JPMML-Evaluator-Python exclusively!

[vruusmann]

The thing is I can't really share my data or PMML as it is. I can try to create some dummy data and model.

Create the simplest pipeline (feature pre-processing plus a linear model) that fails to reproduce Python results in (J)PMML. And post a few sample data points that trigger the unwanted behaviour.

I really need to have a PKL file.

[vruusmann]

Renamed this issue to reflect its true nature.

It is widely known that XGBoost and missing values are a deadly combination for PyPMML. Feel free to search among JPMML project, the evidence is scattered across many projects.

Up to this point, all users have managed to solve this issue by switching from PyPMML to JPMML-Evaluator-Python. I believe this particular issue can be solved exactly the same way.

I'm only keeping this issue open, because the opening comment suggest that JPMML-Evaluator-Python has been tried, and is also not making correct predictions. If so, I'd like to see a reproducible example about it - something very minimal, utilizing some publicly available toy dataset (eg. Iris-Species or Auto-MPG). There is no need for proprietary bits.

[vruusmann]

I believe the OP managed to solve his/her issue by upgrading from PyPMML to JPMML-Evaluator-Python, and simply didn't bother to check back in.

[shivangi-ph]

Hi,

I apologize for not responding earlier. I was following your suggestions on my own data and using Logistic Regression instead of XGBoost and the predictions still do not line up even with JPMML-Evaluator. I have been working on reproducing my issue on a toy dataset and will share that sometime next week. I appreciate your help and apologize for not checking in.

[vruusmann]

I was following your suggestions on my own data and using Logistic Regression instead of XGBoost and the predictions still do not line up even with JPMML-Evaluator.

Reopening, because that is something I want to see!