You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
the StageModel references a StageInfo field to get the details of the stage.
The problem with that design that this causes a deep-levelpointer to data that is not needed by the core tools for now.
@DeveloperApi
class StageInfo(
val stageId: Int,
private val attemptId: Int,
val name: String,
val numTasks: Int,
val rddInfos: Seq[RDDInfo],
val parentIds: Seq[Int],
val details: String,
val taskMetrics: TaskMetrics = null,
private[spark] val taskLocalityPreferences: Seq[Seq[TaskLocation]] = Seq.empty,
private[spark] val shuffleDepId: Option[Int] = None,
val resourceProfileId: Int,
private[spark] var isPushBasedShuffleEnabled: Boolean = false,
private[spark] var shuffleMergerCount: Int = 0) {
/** When this stage was submitted from the DAGScheduler to a TaskScheduler. */
var submissionTime: Option[Long] = None
/** Time when the stage completed or when the stage was cancelled. */
var completionTime: Option[Long] = None
/** If the stage failed, the reason why. */
var failureReason: Option[String] = None
/**
* Terminal values of accumulables updated during this stage, including all the user-defined
* accumulators.
*/
val accumulables = HashMap[Long, AccumulableInfo]()
Ideally, we need to have stub class that only copies what we need.
We did that before in #1206 but we had to roll it back for compatibility with various Spark implementations in #1260
The text was updated successfully, but these errors were encountered:
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
FixesNVIDIA#1524
This commit uses a smaller class `StageInfoStub` to store Spark's
StageInfo. This class is common between all the spark implementations
but it has more fields to the constructor across different versions.
Currently we only use a subset of the class fields. The remaining fields
represent an overhead or redundant storage; especially when it comes to
store the accumulables and taskMetrics for each stage.
To evaluate the memory optimization, a new `Checkpoint` mechanism was
added to allow gathering information at separate stages of the
execution.
The `checkpoint` design and implementation can be further improved and
extended to build a performance profile to compare different tradeoffs.
Describe the bug
the
StageModel
references aStageInfo
field to get the details of the stage.The problem with that design that this causes a deep-levelpointer to data that is not needed by the core tools for now.
Ideally, we need to have stub class that only copies what we need.
We did that before in #1206 but we had to roll it back for compatibility with various Spark implementations in #1260
The text was updated successfully, but these errors were encountered: