Disable Per-SQL summary text output

[amahussein]

Signed-off-by: Ahmed Hussein (amahussein) [email protected]

Fixes #1527

Disable the text format output generated per-sql. The target is to reduce the noise of the stdout and improve the performance of the core-tools

Impact on the output:

• rapids_4_spark_qualification_output_persql.log is not generated anymore by the qualTool
• remove column AppName from rapids_4_spark_qualification_output_persql.csv
• the rapids_4_spark_qualification_output_persql.log can still be generated by the RunningQualificationApp
• The order of the SQls in the CSV file has changed. Sorted Desc based on (GPU opportunity, and DF duration) whithin each app. Perviously, the SQLs were sorted globally which might cause considerable overhead for a large number of eventogs.

Impact on Performance and usability:

• Improve readability of the stdout/log generated by the tools.
• Reduce the size of lines consumed by the python wrapper.
• Sorting Sqls per-app implies less memory requirements since it is only required to maintain the list of SQL for the current iteration.
• Improve the string construction by avoiding filling Buffer<String, Int>
• Improve the performance by skipping generating and writing the log file to the disk.

Details

This pull request to core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualOutputWriter.scala includes changes to improve the performance and structure of the SQL CSV report generation and related functionality. The most important changes include the removal of the application name from the per-SQL report, the optimization of string concatenation and object allocation, and updates to the test expectations to reflect these changes.

Improvements to report generation:

• core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualOutputWriter.scala: Added a new method writePerSqlCSVReport to generate the per-SQL CSV report with optimized string concatenation and reduced object allocations. Removed the application name from the report and updated the header construction accordingly.

Changes to related classes:

• core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala: Removed the call to writePerSqlTextReport and updated the call to writePerSqlCSVReport to reflect the new method signature.
• core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/RunningQualOutputWriter.scala: Updated the header construction to remove the application name size.
• core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/RunningQualificationApp.scala: Simplified the header construction by removing the application name size.

Updates to test expectations

[parthosa]

Thanks @amahussein. Tested this. LGTME.

Q: Why are we removing the App Name from the CSV?

[amahussein]

Thanks @amahussein. Tested this. LGTME.

Q: Why are we removing the App Name from the CSV?

It was not clear why the the app-name was included in the SQL CSV file in first place.
I guess it was their to make the stdout easier to understand.
Since the stdout is removed, I don't see a reason to keep that column around.