How can use a method similar to pandas' groupby to group a dataset and use `numpy.random.uniform` function？

[lanze0000]

Question 1:
How can I use a method similar to pandas' groupby to group a dataset, calculate the maximum value for each group, and generate a pipeline that can create a PMML file using sklearn2pmml? I have a logic reference in the following code, but it does not execute correctly when generating the PMML file. I am currently investigating the cause and looking for alternative solutions. My guess is that Jpmml does not have a similar function, so it cannot be converted. Is my understanding correct?

Reference code for Question 1:

class MaxIncomeTransformer(BaseEstimator, TransformerMixin):  
    def __init__(self, groupby_column, target_column, output_columns=None):  
        self.groupby_column = groupby_column  
        self.target_column = target_column  
        self.output_columns = output_columns  

    def fit(self, X, y=None):  
        return self  

    def transform(self, X):  
        if not isinstance(X, pd.DataFrame):  
            X = pd.DataFrame(X)  

        # Find the index of maximum income for each group  
        idx = X.groupby(self.groupby_column)[self.target_column].idxmax()  
        result = X.loc[idx].reset_index(drop=True)  
        
        # if have output_columns, use it   
        if self.output_columns is not None:  
            result.columns = self.output_columns  
        
        return result[self.output_columns[1]]   
    
#score1 Pipeline  
fraud_final_cols = ['msisdn','score1']  
mapper_final_fraud = DataFrameMapper([  
                                      (['msisdn', 'score1'],   
                                       [MaxIncomeTransformer(groupby_column='msisdn', target_column='score1',output_columns=['msisdn', 'score1'])],  
                                       {'alias':'score1'})  
],input_df=True,df_out=True)  

Question 2:
How can I use a function like random.uniform(0.1, 0.2) within ExpressionTransformer to randomly generate numbers in a specific range?
What I want to achieve is to add some perturbations or random values to the result, so that the result is evenly distributed in a certain interval.
Reference code for Question 2:

mapper_fea2 = DataFrameMapper([  
    (['score_1_wld', 'score_1_zljr', 'score_1_zlhqd'],  
     [ExpressionTransformer("random.uniform(0.1, 0.2) if X[0]==1 or X[1]==1 or X[2]==1 else 0")],  
     {'alias': 'score_1'}),  
    ], input_df=True, df_out=True)

[vruusmann]

Question 1: How can I use a method similar to pandas' groupby to group a dataset, calculate the maximum value for each group

You are trying to modify the number of rows in the (training-) dataset, right?

This is not conceptually supported by Scikit-Learn nor (J)PMML. You should pre-process your dataset into its "stable representation" (meaning the number and ordering of rows will not change, with every row representing one observation) before it enters the "real ML pipeline".

As you can probably see, the most natural place for your MaxIncomeTransformer is somewhere outside of the (PMML)Pipeline, not inside it (especially inside the DataFrameMapper step).

TLDR: Re-think your workflow. You should do low-level dataset work outside of the (PMML)Pipeline.

Question 2: How can I use a function like random.uniform(0.1, 0.2) within ExpressionTransformer...

There are two parts to this question.

First, technically, it is easy to make the numpy.random.uniform() function (or any other stateless function) work with (J)PMML by implementing a so-called "user-defined Java function" (aka Java UDF).

Second, and much more difficult, is to make the Java UDF work with ExpressionTransformer Python component.

I can quickly think of the following extension, where the ExpressionTransformer constructor takes an extra parameter defining Python-to-Java function mappings:

func_mapping = {
  "numpy.random.uniform" : "org.jpmml.python.functions.NumpyRandomUniformFunction"
}

transformer = ExpressionTransformer("random.uniform(0.1, 0.2)", func_mapping = func_mapping)

Very interesting! I'm currently designing Java UDF support for the JPMML-R library, so it would make all the sense to expand the "design scope" a bit, and come up with something that would work both in R and Scikit-Learn/Python world.

[lanze0000]

Thank you very much for your response. Firstly, the sklearn2pmml package has been very helpful to me, and I hope it continues to improve. I look forward to the release of the func_mapping feature, which I believe will make applications more flexible, though it might also bring more challenges. Looking forward to your updates.

[vruusmann]

How can I use a method similar to pandas' groupby to group a dataset, calculate the maximum value for each group, and generate a pipeline that can create a PMML file using sklearn2pmml?

Starting from SkLearn2PMML 0.116.0 there is a sklearn2pmml.preprocessing.RollingAggregateTransformer class available, which supports calculating rolling aggregates (such as min, max, sum, product, avg, etc) over a number of preceding rows (aka "window")

In the current case, you'd simply need to set the "window" size to some arbitrarily large value so that it would encompass the entire dataset.

Next, starting from SkLearn2PMML 0.116.1, the RollingAggregateTransformer class has block_indicators attribute, which emulates "group by" behaviour. For more information, see the description of PMML's Lag element.

For example, the requested behaviour could be implemented like this:

from sklearn2pmml.preprocessing.RollingAggregateTransformer

# Assume the size of each and every group is less than 1000 items
transformer = RollingAggregateTransformer(function = "max", n = 1000, block_indicators = ["groupby_column_name"])