Open
Description
Component(s)
No response
Is your feature request related to a problem? Please describe.
Spark has accumulators which are not scrapped:
"accumulatorUpdates" : [ {
"id" : 445,
"name" : "Perform preparations",
"value" : "855"
}]
Accumulators reference:
https://pluginhighway.ca/blog/speed-up-your-spark-job-with-accumulators-an-ultimate-guide
https://spark.apache.org/docs/latest/api/python/_modules/pyspark/accumulators.html
https://sparkbyexamples.com/pyspark/pyspark-accumulator-with-example/
Describe the solution you'd like
Collect accumulators and send as metrics
Describe alternatives you've considered
No response
Additional context
No response