Sync plugin support as of 2024-12-31 (#1478)

Fixes #1452

This PR syns plugin support as of 2024-12-31.

The changes include:

```
This report documents the differences between the tools existing CSV files and those processed from the plugin.
    Notes:
      1. For new data source/exec/expression from plugin, the first column with supported level will be updated to 'TNEW' for future testing.
      2. Rows marked as "is removed" will be preserved in the final output.
      3. The "Notes" column for rows with "S" for "Supported" will be updated to "None" in the final output.


**supportedDataSource.csv (FROM TOOLS TO PLUGIN)**
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    BOOLEAN: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    BYTE: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    SHORT: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    INT: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    LONG: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    FLOAT: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    DOUBLE: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    DATE: CO -> PS
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    TIMESTAMP: CO -> PS
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    STRING: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    DECIMAL: CO -> S
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    NULL: CO -> NA
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    BINARY: CO -> NS
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    CALENDAR: CO -> NA
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    ARRAY: CO -> PS
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    MAP: CO -> NS
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    STRUCT: CO -> PS
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    UDT: CO -> NS
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    DAYTIME: CO -> NA
Row is changed: JSON, read, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO, CO
    YEARMONTH: CO -> NA

**supportedExecs.csv (FROM TOOLS TO PLUGIN)**
Row is removed: CustomShuffleReaderExec, S, None, Input/Output, S, S, S, S, S, S, S, S, PS, S, S, S, S, NS, PS, PS, PS, NS, NS, NS
Row is removed: RunningWindowFunctionExec, S, None, Input/Output, S, S, S, S, S, S, S, S, PS, S, S, S, NS, NS, PS, PS, PS, NS, NS, NS

**supportedExprs.csv (FROM TOOLS TO PLUGIN)**
Row is changed: HiveHash, S, `hive-hash`, None, project, input, S, S, S, S, S, S, S, S, PS, S, NS, S, NS, NS, NS, NS, NS, NS, NS, NS
    ARRAY: NS -> PS
Row is changed: HiveHash, S, `hive-hash`, None, project, input, S, S, S, S, S, S, S, S, PS, S, NS, S, NS, NS, NS, NS, NS, NS, NS, NS
    STRUCT: NS -> PS
Row is changed: JsonToStructs, NS, `from_json`, This is disabled by default because it is currently in beta and undergoes continuous enhancements. Please consult the [compatibility documentation](../compatibility.md#json-supporting-types) to determine whether you can enable this configuration for your use case, project, jsonStr, NA, NA, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
    Supported: NS -> S
Row is changed: JsonToStructs, NS, `from_json`, This is disabled by default because it is currently in beta and undergoes continuous enhancements. Please consult the [compatibility documentation](../compatibility.md#json-supporting-types) to determine whether you can enable this configuration for your use case, project, jsonStr, NA, NA, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
    Notes: This is disabled by default because it is currently in beta and undergoes continuous enhancements. Please consult the [compatibility documentation](../compatibility.md#json-supporting-types) to determine whether you can enable this configuration for your use case -> None
Row is changed: JsonToStructs, NS, `from_json`, This is disabled by default because it is currently in beta and undergoes continuous enhancements. Please consult the [compatibility documentation](../compatibility.md#json-supporting-types) to determine whether you can enable this configuration for your use case, project, result, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, PS, PS, NA, NA, NA
    Supported: NS -> S
Row is changed: JsonToStructs, NS, `from_json`, This is disabled by default because it is currently in beta and undergoes continuous enhancements. Please consult the [compatibility documentation](../compatibility.md#json-supporting-types) to determine whether you can enable this configuration for your use case, project, result, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, PS, PS, NA, NA, NA
    Notes: This is disabled by default because it is currently in beta and undergoes continuous enhancements. Please consult the [compatibility documentation](../compatibility.md#json-supporting-types) to determine whether you can enable this configuration for your use case -> None
Row is added: MonthsBetween, TNEW, `months_between`, None, project, timestamp1, NA, NA, NA, NA, NA, NA, NA, NA, PS, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: MonthsBetween, TNEW, `months_between`, None, project, timestamp2, NA, NA, NA, NA, NA, NA, NA, NA, PS, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: MonthsBetween, TNEW, `months_between`, None, project, round, PS, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: MonthsBetween, TNEW, `months_between`, None, project, result, NA, NA, NA, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: TruncDate, TNEW, `trunc`, None, project, date, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: TruncDate, TNEW, `trunc`, None, project, format, NA, NA, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: TruncDate, TNEW, `trunc`, None, project, result, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: TruncTimestamp, TNEW, `date_trunc`, None, project, format, NA, NA, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: TruncTimestamp, TNEW, `date_trunc`, None, project, date, NA, NA, NA, NA, NA, NA, NA, NA, PS, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is added: TruncTimestamp, TNEW, `date_trunc`, None, project, result, NA, NA, NA, NA, NA, NA, NA, NA, PS, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
Row is changed: XxHash64, S, `xxhash64`, None, project, input, S, S, S, S, S, S, S, S, PS, S, S, S, NS, NS, NS, NS, NS, NS, NS, NS
    ARRAY: NS -> PS
Row is changed: XxHash64, S, `xxhash64`, None, project, input, S, S, S, S, S, S, S, S, PS, S, S, S, NS, NS, NS, NS, NS, NS, NS, NS
    MAP: NS -> PS
Row is changed: XxHash64, S, `xxhash64`, None, project, input, S, S, S, S, S, S, S, S, PS, S, S, S, NS, NS, NS, NS, NS, NS, NS, NS
    STRUCT: NS -> PS
Row is removed: EphemeralSubstring, S, `substr`; `substring`, None, project, str, NA, NA, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NS, NA, NA, NA, NA, NA, NS, NS
Row is removed: EphemeralSubstring, S, `substr`; `substring`, None, project, pos, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
Row is removed: EphemeralSubstring, S, `substr`; `substring`, None, project, len, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
Row is removed: EphemeralSubstring, S, `substr`; `substring`, None, project, result, NA, NA, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NS, NA, NA, NA, NA, NA, NS, NS
Row is removed: DecimalSum, S, `decimalsum`, None, project, input, NA, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
Row is removed: DecimalSum, S, `decimalsum`, None, project, result, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, PS, NA, NA, NA, NA, NA, NA, NA, NS, NS
```

---------

Signed-off-by: cindyyuanjiang <[email protected]>

Author: cindyyuanjiang

core/src/main/resources/operatorsScore-databricks-aws-a10G.csv

@@ -306,3 +306,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-databricks-aws-t4.csv

@@ -306,3 +306,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-databricks-azure-t4.csv

@@ -294,3 +294,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-dataproc-gke-l4.csv

@@ -288,3 +288,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-dataproc-gke-t4.csv

@@ -288,3 +288,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-dataproc-l4.csv

@@ -294,3 +294,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-dataproc-serverless-l4.csv

@@ -288,3 +288,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-dataproc-t4.csv

@@ -294,3 +294,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-emr-a10.csv

@@ -294,3 +294,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-emr-a10G.csv

@@ -294,3 +294,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-emr-t4.csv

@@ -294,3 +294,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/operatorsScore-onprem-a100.csv

@@ -306,3 +306,6 @@ MaxBy,1.5
 MinBy,1.5
 ArrayJoin,1.5
 RunningWindowFunctionExec,1.5
+MonthsBetween,1.5
+TruncDate,1.5
+TruncTimestamp,1.5

core/src/main/resources/supportedDataSource.csv

@@ -6,7 +6,7 @@ Delta,write,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 HiveText,read,S,S,S,S,S,S,S,S,PS,S,S,NS,NS,NS,NS,NS,NS,NS,NS,NS
 HiveText,write,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Iceberg,read,S,S,S,S,S,S,S,S,PS,S,S,NA,S,NA,PS,PS,PS,NS,S,S
-JSON,read,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO,CO
+JSON,read,S,S,S,S,S,S,S,PS,PS,S,S,NA,NS,NA,PS,NS,PS,NS,NA,NA
 ORC,read,S,S,S,S,S,S,S,S,PS,S,S,NA,NS,NA,PS,PS,PS,NS,NA,NA
 ORC,write,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Parquet,read,S,S,S,S,S,S,S,S,PS,S,S,NA,S,NA,PS,PS,PS,NS,S,S

core/src/main/resources/supportedExecs.csv

@@ -21,6 +21,7 @@ SortAggregateExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,PS,NS,PS,PS,PS,NS
 InMemoryTableScanExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,NS,NS,NS,PS,PS,PS,NS,S,S
 DataWritingCommandExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,PS,NS,S,NS,PS,PS,PS,NS,S,S
 ExecutedCommandExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,S,S,PS,PS,PS,S,S,S
+WriteFilesExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,S,S,PS,PS,PS,S,S,S
 AppendDataExecV1,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,NS,S,NS,PS,PS,PS,NS,S,S
 AtomicCreateTableAsSelectExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,NS,S,NS,PS,PS,PS,NS,S,S
 AtomicReplaceTableAsSelectExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,NS,S,NS,PS,PS,PS,NS,S,S
@@ -53,8 +54,7 @@ WindowInPandasExec,NS,This is disabled by default because it only supports row b
 WindowExec,S,None,partitionSpec,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NS,PS,NS,NS,NS
 WindowExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,S,NS,PS,PS,PS,NS,NS,NS
 HiveTableScanExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,NS,NS,NS,NS,NS,NS,NS,NS,NS
-WriteFilesExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,S,S,PS,PS,PS,S,S,S
-CustomShuffleReaderExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,S,NS,PS,PS,PS,NS,NS,NS
 WindowGroupLimitExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,PS,PS,PS,NS,NS,NS
 MapInArrowExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,NS,NS,NS,NS,PS,NS,PS,NS,NS,NS
+CustomShuffleReaderExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,S,NS,PS,PS,PS,NS,NS,NS
 RunningWindowFunctionExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,PS,PS,PS,NS,NS,NS

core/src/main/resources/supportedExprs.csv

@@ -269,7 +269,7 @@ GreaterThanOrEqual,S,`>=`,None,AST,rhs,S,S,S,S,S,NS,NS,S,PS,S,NS,NS,NS,NS,NS,NA,
 GreaterThanOrEqual,S,`>=`,None,AST,result,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Greatest,S,`greatest`,None,project,param,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NA,NS,NS,NA,NA
 Greatest,S,`greatest`,None,project,result,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NA,NS,NS,NA,NA
-HiveHash,S,`hive-hash`,None,project,input,S,S,S,S,S,S,S,S,PS,S,NS,S,NS,NS,NS,NS,NS,NS,NS,NS
+HiveHash,S,`hive-hash`,None,project,input,S,S,S,S,S,S,S,S,PS,S,NS,S,NS,NS,PS,NS,PS,NS,NS,NS
 HiveHash,S,`hive-hash`,None,project,result,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Hour,S,`hour`,None,project,input,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Hour,S,`hour`,None,project,result,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
@@ -299,8 +299,8 @@ IsNotNull,S,`isnotnull`,None,project,input,S,S,S,S,S,S,S,S,PS,S,S,S,S,NS,PS,PS,P
 IsNotNull,S,`isnotnull`,None,project,result,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 IsNull,S,`isnull`,None,project,input,S,S,S,S,S,S,S,S,PS,S,S,S,S,NS,PS,PS,PS,NS,S,NS
 IsNull,S,`isnull`,None,project,result,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-JsonToStructs,NS,`from_json`,This is disabled by default because it is currently in beta and undergoes continuous enhancements. Please consult the [compatibility documentation](../compatibility.md#json-supporting-types) to determine whether you can enable this configuration for your use case,project,jsonStr,NA,NA,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-JsonToStructs,NS,`from_json`,This is disabled by default because it is currently in beta and undergoes continuous enhancements. Please consult the [compatibility documentation](../compatibility.md#json-supporting-types) to determine whether you can enable this configuration for your use case,project,result,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NS,PS,PS,NA,NA,NA
+JsonToStructs,S,`from_json`,None,project,jsonStr,NA,NA,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+JsonToStructs,S,`from_json`,None,project,result,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NS,PS,PS,NA,NA,NA
 JsonTuple,NS,`json_tuple`,This is disabled by default because Experimental feature that could be unstable or have performance issues.,project,json,NA,NA,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 JsonTuple,NS,`json_tuple`,This is disabled by default because Experimental feature that could be unstable or have performance issues.,project,field,NA,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 JsonTuple,NS,`json_tuple`,This is disabled by default because Experimental feature that could be unstable or have performance issues.,project,result,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA
@@ -384,6 +384,10 @@ Minute,S,`minute`,None,project,result,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,N
 MonotonicallyIncreasingID,S,`monotonically_increasing_id`,None,project,result,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Month,S,`month`,None,project,input,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Month,S,`month`,None,project,result,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+MonthsBetween,S,`months_between`,None,project,timestamp1,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+MonthsBetween,S,`months_between`,None,project,timestamp2,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+MonthsBetween,S,`months_between`,None,project,round,PS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+MonthsBetween,S,`months_between`,None,project,result,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Multiply,S,`*`,None,project,lhs,NA,S,S,S,S,S,S,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Multiply,S,`*`,None,project,rhs,NA,S,S,S,S,S,S,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Multiply,S,`*`,None,project,result,NA,S,S,S,S,S,S,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA
@@ -625,6 +629,12 @@ TransformKeys,S,`transform_keys`,None,project,result,NA,NA,NA,NA,NA,NA,NA,NA,NA,
 TransformValues,S,`transform_values`,None,project,argument,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA
 TransformValues,S,`transform_values`,None,project,function,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,PS,PS,PS,NS,NS,NS
 TransformValues,S,`transform_values`,None,project,result,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA
+TruncDate,S,`trunc`,None,project,date,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+TruncDate,S,`trunc`,None,project,format,NA,NA,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+TruncDate,S,`trunc`,None,project,result,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+TruncTimestamp,S,`date_trunc`,None,project,format,NA,NA,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+TruncTimestamp,S,`date_trunc`,None,project,date,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+TruncTimestamp,S,`date_trunc`,None,project,result,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 UnaryMinus,S,`negative`,None,project,input,NA,S,S,S,S,S,S,NA,NA,NA,S,NA,NA,NS,NA,NA,NA,NA,S,S
 UnaryMinus,S,`negative`,None,project,result,NA,S,S,S,S,S,S,NA,NA,NA,S,NA,NA,NS,NA,NA,NA,NA,S,S
 UnaryMinus,S,`negative`,None,AST,input,NA,NS,NS,S,S,S,S,NA,NA,NA,NS,NA,NA,NS,NA,NA,NA,NA,NS,NS
@@ -650,7 +660,7 @@ WindowExpression,S, ,None,window,result,S,S,S,S,S,S,S,S,PS,S,S,S,S,S,PS,PS,PS,S,
 WindowSpecDefinition,S, ,None,project,partition,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NS,PS,NS,NS,NS
 WindowSpecDefinition,S, ,None,project,value,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NS,PS,NS,NS,NS
 WindowSpecDefinition,S, ,None,project,result,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NS,PS,NS,NS,NS
-XxHash64,S,`xxhash64`,None,project,input,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NS,NS,NS,NS,NS
+XxHash64,S,`xxhash64`,None,project,input,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,PS,PS,PS,NS,NS,NS
 XxHash64,S,`xxhash64`,None,project,result,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Year,S,`year`,None,project,input,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 Year,S,`year`,None,project,result,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA

core/src/test/resources/QualificationExpectations/qual_test_simple_expectation.csv

@@ -1,5 +1,5 @@
 App Name,App ID,SQL DF Duration,SQL Dataframe Task Duration,App Duration,GPU Opportunity,Executor CPU Time Percent,SQL Ids with Failures,Unsupported Read File Formats and Types,Unsupported Write Data Format,Complex Types,Nested Complex Types,Potential Problems,Longest SQL Duration,SQL Stage Durations Sum,NONSQL Task Duration Plus Overhead,Unsupported Task Duration,Supported SQL DF Task Duration,App Duration Estimated,Unsupported Execs,Unsupported Expressions,Estimated Job Frequency (monthly),Total Core Seconds
 "Rapids Spark Profiling Tool Unit Tests","local-1622043423018",11600,132257,16319,9868,37.7,"","","JSON","","","",7143,13770,4719,19744,112513,false,"SerializeFromObject;Scan unknown;Execute InsertIntoHadoopFsRelationCommand json;DeserializeToObject;Filter;MapElements","",1,186
-"Spark shell","local-1651187225439",224,180,355637,74,87.88,"","JSON[string:bigint:int]","","","","",498,228,355101,120,60,false,"SerializeFromObject;CollectLimit;DeserializeToObject;Scan json;Filter;MapElements","",1,2834
-"Spark shell","local-1651188809790",347,283,166215,14,81.18,"","JSON[string:bigint:int]","","","","UDF",715,318,165572,271,12,false,"CollectLimit;Scan json;Project","UDF",1,1318
-"Rapids Spark Profiling Tool Unit Tests","local-1623281204390",1156,4666,6240,1,46.27,"","JSON[string:bigint:int]","JSON","","","UDF",1209,1130,5809,4661,5,false,"Execute InsertIntoHadoopFsRelationCommand json;LocalTableScan;Project;Scan json;Execute CreateViewCommand","UDF",1,64
+"Spark shell","local-1651187225439",224,180,355637,142,87.88,"","","","","","",498,228,355101,66,114,false,"SerializeFromObject;CollectLimit;DeserializeToObject;Filter;MapElements","",1,2834
+"Spark shell","local-1651188809790",347,283,166215,128,81.18,"","","","","","UDF",715,318,165572,178,105,false,"CollectLimit;Project","UDF",1,1318
+"Rapids Spark Profiling Tool Unit Tests","local-1623281204390",1156,4666,6240,122,46.27,"","","JSON","","","UDF",1209,1130,5809,4170,496,false,"Execute InsertIntoHadoopFsRelationCommand json;LocalTableScan;Execute CreateViewCommand;Project","UDF",1,64

core/src/test/resources/QualificationExpectations/qual_test_simple_expectation_persql.csv

@@ -2,17 +2,17 @@ App Name,App ID,Root SQL ID,SQL ID,SQL Description,SQL DF Duration,GPU Opportuni
 "Rapids Spark Profiling Tool Unit Tests","local-1622043423018","",1,"count at QualificationInfoUtils.scala:94",7143,6719
 "Rapids Spark Profiling Tool Unit Tests","local-1622043423018","",3,"count at QualificationInfoUtils.scala:94",2052,1660
 "Rapids Spark Profiling Tool Unit Tests","local-1622043423018","",2,"count at QualificationInfoUtils.scala:94",1933,1551
-"Spark shell","local-1651188809790","",1,"show at <console>:26",196,75
-"Spark shell","local-1651187225439","",0,"show at <console>:26",498,168
-"Spark shell","local-1651187225439","",1,"show at <console>:26",262,80
+"Spark shell","local-1651187225439","",0,"show at <console>:26",498,333
+"Spark shell","local-1651188809790","",0,"show at <console>:26",715,242
 "Rapids Spark Profiling Tool Unit Tests","local-1622043423018","",0,"json at QualificationInfoUtils.scala:76",1306,164
-"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",0,"json at QualificationInfoUtils.scala:130",1209,0
-"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",2,"json at QualificationInfoUtils.scala:136",321,0
-"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",6,"json at QualificationInfoUtils.scala:130",110,0
-"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",3,"json at QualificationInfoUtils.scala:130",108,0
+"Spark shell","local-1651188809790","",1,"show at <console>:26",196,135
+"Spark shell","local-1651187225439","",1,"show at <console>:26",262,110
+"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",2,"json at QualificationInfoUtils.scala:136",321,107
+"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",5,"json at QualificationInfoUtils.scala:136",129,43
+"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",8,"json at QualificationInfoUtils.scala:136",127,42
 "Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",4,"createOrReplaceTempView at QualificationInfoUtils.scala:133",22,22
 "Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",7,"createOrReplaceTempView at QualificationInfoUtils.scala:133",4,4
 "Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",1,"createOrReplaceTempView at QualificationInfoUtils.scala:133",2,2
-"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",5,"json at QualificationInfoUtils.scala:136",129,0
-"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",8,"json at QualificationInfoUtils.scala:136",127,0
-"Spark shell","local-1651188809790","",0,"show at <console>:26",715,5
+"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",0,"json at QualificationInfoUtils.scala:130",1209,0
+"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",6,"json at QualificationInfoUtils.scala:130",110,0
+"Rapids Spark Profiling Tool Unit Tests","local-1623281204390","",3,"json at QualificationInfoUtils.scala:130",108,0

core/src/test/resources/QualificationExpectations/read_dsv1_expectation.csv

@@ -1,2 +1,2 @@
 App Name,App ID,SQL DF Duration,SQL Dataframe Task Duration,App Duration,GPU Opportunity,Executor CPU Time Percent,SQL Ids with Failures,Unsupported Read File Formats and Types,Unsupported Write Data Format,Complex Types,Nested Complex Types,Potential Problems,Longest SQL Duration,SQL Stage Durations Sum,NONSQL Task Duration Plus Overhead,Unsupported Task Duration,Supported SQL DF Task Duration,App Duration Estimated,Unsupported Execs,Unsupported Expressions,Estimated Job Frequency (monthly),Total Core Seconds
-"Spark shell","local-1624371544219",4575,20421,175293,1523,72.15,"","JSON[string:double:date:int:bigint];Text[*]","JSON","","","",1859,5372,176916,13622,6799,false,"CollectLimit;Scan text;Execute InsertIntoHadoopFsRelationCommand json;Scan json","",30,2096
+"Spark shell","local-1624371544219",4575,20421,175293,4365,72.15,"","Text[*]","JSON","","","",1859,5372,176916,938,19483,false,"CollectLimit;Scan text;Execute InsertIntoHadoopFsRelationCommand json","",30,2096

core/src/test/resources/QualificationExpectations/read_dsv2_expectation.csv

@@ -1,2 +1,2 @@
 App Name,App ID,SQL DF Duration,SQL Dataframe Task Duration,App Duration,GPU Opportunity,Executor CPU Time Percent,SQL Ids with Failures,Unsupported Read File Formats and Types,Unsupported Write Data Format,Complex Types,Nested Complex Types,Potential Problems,Longest SQL Duration,SQL Stage Durations Sum,NONSQL Task Duration Plus Overhead,Unsupported Task Duration,Supported SQL DF Task Duration,App Duration Estimated,Unsupported Execs,Unsupported Expressions,Estimated Job Frequency (monthly),Total Core Seconds
-"Spark shell","local-1624371906627",4917,21802,83738,2687,71.3,"","Text[*];json[double]","JSON","","","",1984,5438,83336,9889,11913,false,"CollectLimit;Scan text;Execute InsertIntoHadoopFsRelationCommand json;BatchScan json","",30,997
+"Spark shell","local-1624371906627",4917,21802,83738,4762,71.3,"","Text[*]","JSON","","","",1984,5438,83336,689,21113,false,"CollectLimit;Scan text;Execute InsertIntoHadoopFsRelationCommand json","",30,997