Skip to content

Commit ef2fb35

Browse files
authored
Fix resuming arrow format (#6964)
* fix resuming in arrow format * one more * fix arrow resuming
1 parent 087671d commit ef2fb35

File tree

3 files changed

+357
-215
lines changed

3 files changed

+357
-215
lines changed

docs/source/stream.mdx

+1-1
Original file line numberDiff line numberDiff line change
@@ -415,6 +415,6 @@ This can be used with the `StatefulDataLoader` from `torchdata`:
415415

416416
<Tip>
417417

418-
Resuming returns exactly where the checkpoint was saved except in two cases: 1) examples from shuffle buffers are lost when resuming and the buffers are refilled with new data and 2) combinations of `.with_format(arrow)` and batched `.map()` may skip one batch.
418+
Resuming returns exactly where the checkpoint was saved except if `.shuffle()` is used: examples from shuffle buffers are lost when resuming and the buffers are refilled with new data.
419419

420420
</Tip>

0 commit comments

Comments
 (0)