Skip to content

Accelerated partitioning #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Nov 30, 2018
Merged

Accelerated partitioning #114

merged 13 commits into from
Nov 30, 2018

Conversation

JulienPeloton
Copy link
Member

This is an aborted PR in which I tried to accelerate the partitioning by filtering the dataset prior to repartitioning:

// Load data
df = spark....load()

// Add a column with future repartitioning
df_colid = df.prePartition(...)

// Keep only data of interest
df_colid_subset = df_colid.filter(cond)

// Repartition only data of interest
df_repart = df_colid_subset.repartitionByCol(...)

But it turns out to be super slow, because of the udf in prePartition.

So I decided to delay this development, but still merging other minor changes (docs + getNeighborNodes) which are useful.

TBC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant