Skip to content
This repository was archived by the owner on Feb 10, 2025. It is now read-only.

Databus Throttling Changes #847

Merged
merged 27 commits into from
Nov 6, 2024
Merged

Databus Throttling Changes #847

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
cce3d28
feat: execution name sanitization and unit test for stepfn
rishuyadavbv Oct 28, 2024
49cfe91
master_size api changes
nabajyotiDash-hub Nov 4, 2024
f082d99
Admin API changes (#845)
shan-1094 Nov 4, 2024
05fb5ed
PD-257044-changes-for-databus (#846)
anurag0510 Nov 4, 2024
62dc31e
added datadog metrics changes
anurag0510 Oct 26, 2024
72ed737
[PD-257044] databus changes rebase (#848)
anurag0510 Nov 5, 2024
0dad904
admin api and queue name split
anandujayan Nov 5, 2024
63d05f0
Pd 257044 databus changes rebase (#849)
anurag0510 Nov 5, 2024
be3df07
Revert "added datadog metrics changes"
nabajyotiDash-hub Nov 5, 2024
6b0b672
build fix
nabajyotiDash-hub Nov 5, 2024
6fab83e
endpoint change and databus changes
anandujayan Nov 5, 2024
ff905b0
updated updateRefInDatabus method
anurag0510 Nov 6, 2024
02c3a35
address review commnets
nabajyotiDash-hub Nov 6, 2024
7d71b48
branch admin -prepare release emodb-6.5.205
Nov 6, 2024
6193999
branch admin -prepare for next development iteration
Nov 6, 2024
05cfcbf
edit doc
nabajyotiDash-hub Nov 6, 2024
1861f8b
fix tests
nabajyotiDash-hub Nov 6, 2024
da60cdb
fix tests
nabajyotiDash-hub Nov 6, 2024
a40a48c
fix review comments
nabajyotiDash-hub Nov 6, 2024
ab2c2d1
removed ;
nabajyotiDash-hub Nov 6, 2024
6e1cf00
branch admin -prepare release emodb-6.5.206
Nov 6, 2024
3373932
upgrade pom version
nabajyotiDash-hub Nov 6, 2024
2941007
branch admin -prepare release emodb-6.5.208
Nov 6, 2024
54122d9
branch admin -prepare for next development iteration
Nov 6, 2024
173f50b
get or default change
anandujayan Nov 6, 2024
13847e8
sor get or default changes
anandujayan Nov 6, 2024
fcd125a
Unit test fixes
anandujayan Nov 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion auth/auth-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-store/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-util/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>emodb</artifactId>
<groupId>com.bazaarvoice.emodb</groupId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion blob-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client-jersey2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion cachemgr/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/astyanax/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client-jax-rs-2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client-jersey2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/dropwizard/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/jersey-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/json/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/stash/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/uuid/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/zookeeper/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
4 changes: 4 additions & 0 deletions databus-api/src/main/java/com/bazaarvoice/emodb/databus/api/Databus.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ Subscription getSubscription(String subscription)
*/
long getEventCountUpTo(String subscription, long limit);

default long getMasterCount(){
return 0;
};

/** Counts the number of events with outstanding claims that cause the events to be skipped by {@link #poll}. */
long getClaimCount(String subscription);

Expand Down
2 changes: 1 addition & 1 deletion databus-client-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client-jersey2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
5 changes: 5 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/DatabusFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,11 @@ public long getEventCountUpTo(String subscription, long limit) {
return _ownerAwareDatabus.getEventCountUpTo(ownerId, subscription, limit);
}

@Override
public long getMasterCount() {
return _ownerAwareDatabus.getMasterEventCountUncached(ownerId);
}

@Override
public long getClaimCount(String subscription) {
return _ownerAwareDatabus.getClaimCount(ownerId, subscription);
Expand Down
12 changes: 12 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/DefaultDatabus.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,18 @@ public long getEventCount(String ownerId, String subscription) {
return getEventCountUpTo(ownerId, subscription, Long.MAX_VALUE);
}

@Override
public long getMasterEventCountUncached(String ownerId) {
_log.info("Inside getMasterEventCountUncached with _masterFanoutChannels length {}", _masterFanoutChannels.size());
long size = 0;
for(String channel : _masterFanoutChannels) {
size += _eventStore.getSizeEstimate(channel, Long.MAX_VALUE);
_log.info("From channel size {} {}:", channel, size);
}
return size;
}


@Override
public long getEventCountUpTo(String ownerId, String subscription, long limit) {
checkSubscriptionOwner(ownerId, subscription);
Expand Down
3 changes: 3 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/OwnerAwareDatabus.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ Subscription getSubscription(String ownerId, String subscription)
long getEventCount(String ownerId, String subscription)
throws UnauthorizedSubscriptionException;

public long getMasterEventCountUncached(String ownerId)
throws UnauthorizedSubscriptionException;

long getEventCountUpTo(String ownerId, String subscription, long limit)
throws UnauthorizedSubscriptionException;

Expand Down
2 changes: 1 addition & 1 deletion datacenter/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion event/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion job-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion job/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion kafka/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion megabus/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
4 changes: 2 additions & 2 deletions parent/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<packaging>pom</packaging>

<name>EmoDB Parent</name>
Expand All @@ -20,7 +20,7 @@
<connection>scm:git:[email protected]:bazaarvoice/emodb.git</connection>
<developerConnection>scm:git:[email protected]:bazaarvoice/emodb.git</developerConnection>
<url>scm:git:[email protected]:bazaarvoice/emodb.git</url>
<tag>emodb-6.5.98</tag>
<tag>emodb-6.5.208-SNAPSHOT</tag>
</scm>

<properties>
Expand Down
2 changes: 1 addition & 1 deletion plugins/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
4 changes: 2 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>parent/pom.xml</relativePath>
</parent>

Expand All @@ -18,7 +18,7 @@
<connection>scm:git:[email protected]:bazaarvoice/emodb.git</connection>
<developerConnection>scm:git:[email protected]:bazaarvoice/emodb.git</developerConnection>
<url>scm:git:[email protected]:bazaarvoice/emodb.git</url>
<tag>emodb-6.5.98</tag>
<tag>emodb-6.5.208-SNAPSHOT</tag>
</scm>

<modules>
Expand Down
2 changes: 1 addition & 1 deletion quality/integration/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.209-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
Loading