Skip to content

[BEAM-12260] Java - Backport Firestore connector's ramp-up throttling to Datastore connector #14713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 56 commits into from
Jun 21, 2021
Merged

[BEAM-12260] Java - Backport Firestore connector's ramp-up throttling to Datastore connector #14713

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
112d1ce
Initial implementation of the ramp-up throttler.
danthev Mar 19, 2021
36739f9
Merge branch 'master' into datastore-connector-java/clientside-thrott…
danthev Mar 19, 2021
a1a4787
Fix unit tests.
danthev Mar 20, 2021
0cd7738
Implemented resharding ramp-up
danthev Mar 29, 2021
396124b
Fix nullable field to make tests run.
danthev Mar 30, 2021
ee93491
Add throttler shards parameter, some fixes to ramp-up throttler
danthev Apr 2, 2021
e1b4d95
Change rampup budget growth computation
danthev Apr 2, 2021
cd587a0
Restructure throttler loop
danthev Apr 7, 2021
67c18e6
Added additional integration tests for batch sizing
danthev Apr 8, 2021
302f506
Update batch sizing integration tests
danthev Apr 14, 2021
b33901d
Set user agent, update DatastoreV1 tests.
danthev Apr 21, 2021
63cade3
Switch back to hinted worker count, write ramp-up unit test.
danthev Apr 27, 2021
ad4cb20
Make sleeper visible for testing.
danthev Apr 27, 2021
38d562a
Merge remote-tracking branch 'upstream/master' into datastore-connect…
danthev Apr 27, 2021
990aae8
Fix tests, update documentation.
danthev Apr 28, 2021
e8fbb62
Revert test code
danthev Apr 28, 2021
0aaabfa
Merge changes from updated master
danthev Apr 28, 2021
ad2bf58
Reformat, fix imports.
danthev Apr 28, 2021
cda24d5
Merge remote-tracking branch 'upstream/master' into datastore-connect…
danthev May 3, 2021
29d2d85
Update CHANGES.md
danthev May 3, 2021
8830287
Set minimum ops budget.
danthev May 3, 2021
b46ff1c
Drop one metric
danthev May 3, 2021
08b2db9
Merge CHANGES.md
danthev May 20, 2021
1073dff
Merge remote-tracking branch 'upstream/master' into datastore-connect…
danthev Jun 3, 2021
16d5ebe
Initial implementation of the ramp-up throttler.
danthev Mar 19, 2021
4f6be17
Fix unit tests.
danthev Mar 20, 2021
904905c
Implemented resharding ramp-up
danthev Mar 29, 2021
519897a
Fix nullable field to make tests run.
danthev Mar 30, 2021
2bead9e
Add throttler shards parameter, some fixes to ramp-up throttler
danthev Apr 2, 2021
c1c602d
Change rampup budget growth computation
danthev Apr 2, 2021
d592311
Restructure throttler loop
danthev Apr 7, 2021
d4ac5c6
Added additional integration tests for batch sizing
danthev Apr 8, 2021
07e513f
Update batch sizing integration tests
danthev Apr 14, 2021
709b36a
Set user agent, update DatastoreV1 tests.
danthev Apr 21, 2021
4b1e943
Switch back to hinted worker count, write ramp-up unit test.
danthev Apr 27, 2021
c28bcd9
Make sleeper visible for testing.
danthev Apr 27, 2021
984d967
Fix tests, update documentation.
danthev Apr 28, 2021
e1e4a9f
Revert test code
danthev Apr 28, 2021
4f1eaa7
Merge changes from updated master
danthev Apr 28, 2021
45638bf
Reformat, fix imports.
danthev Apr 28, 2021
98f19d7
Update CHANGES.md
danthev Jun 3, 2021
f33d4cf
Set minimum ops budget.
danthev May 3, 2021
acd5363
Drop one metric
danthev May 3, 2021
d18cc13
Merge branch 'datastore-connector-java/clientside-throttling' of http…
danthev Jun 3, 2021
962166a
Merge remote-tracking branch 'upstream/master' into datastore-connect…
danthev Jun 11, 2021
4b3982d
Merge remote-tracking branch 'upstream/master' into datastore-connect…
danthev Jun 14, 2021
5072afd
Update documentation, add explicit throttling budget max value.
danthev Jun 17, 2021
2ec9b26
Remove firstInstant reset from DoFn setup
danthev Jun 17, 2021
1c51b9f
Merge remote-tracking branch 'upstream/master' into datastore-connect…
danthev Jun 17, 2021
41a9e1f
Update comment, verify counter in unit test.
danthev Jun 17, 2021
eb52390
Merge remote-tracking branch 'upstream/master' into datastore-connect…
danthev Jun 17, 2021
31b5124
Fix format
danthev Jun 17, 2021
bf9864d
Add newline
danthev Jun 17, 2021
748c485
Remove spaces
danthev Jun 17, 2021
08d7135
Merge remote-tracking branch 'upstream/master' into datastore-connect…
danthev Jun 18, 2021
78565af
Update comment.
danthev Jun 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@

* `CREATE FUNCTION` DDL statement added to Calcite SQL syntax. `JAR` and `AGGREGATE` are now reserved keywords. ([BEAM-12339](https://issues.apache.org/jira/browse/BEAM-12339)).
* Flink 1.13 is now supported by the Flink runner ([BEAM-12277](https://issues.apache.org/jira/browse/BEAM-12277)).
* DatastoreIO: Write and delete operations now follow automatic gradual ramp-up,
in line with best practices (Java/Python) ([BEAM-12260](https://issues.apache.org/jira/browse/BEAM-12260), [BEAM-12272](https://issues.apache.org/jira/browse/BEAM-12272)).
* Python `TriggerFn` has a new `may_lose_data` method to signal potential data loss. Default behavior assumes safe (necessary for backwards compatibility). See Deprecations for potential impact of overriding this. ([BEAM-9487](https://issues.apache.org/jira/browse/BEAM-9487)).
* X feature added (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).

Expand Down
Loading