This repository was archived by the owner on Feb 10, 2025. It is now read-only.
PD-257019 Producer send and logging changes #838
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* added exception handling * changed producer send logic to be synchronous * removed redundant logs
6 tasks
rishuyadavbv
added a commit
that referenced
this pull request
Oct 26, 2024
* fix: fix send logic and exception handling changes * added exception handling * changed producer send logic to be synchronous * removed redundant logs * uncached_size for dedupq (#837) * chore: rename queuetype as per convention * chore: fix naming issue --------- Co-authored-by: nabajyotiDash-hub <[email protected]>
bharadwaj-k
pushed a commit
that referenced
this pull request
Oct 28, 2024
…function firing (#844) * PD-257019 Stabilization emodb queue service changes (#831) * PD-249429: integrated emodb with datastorage-media-service for getting the metadata from s3. * PD-249429: refactored code. * PD-249428: DELETE blob with blob id. * PD-249428: Implemented upload blob from byte array. * PD-249428: Implemented upload blob from byte array. * PD-249428: Fixed failing ITs. * PD-249428: Fixed failing ITs. * PD-249428: Changed the snapshot version. * PD-249428: Refactored code. * PD-249428: changed the version to 172. * PD-249428: commented tests. * branch admin -prepare release emodb-6.5.172 * PD-249428: changed snapshot versions. * PD-249428: changed snapshot versions. * branch admin -prepare release emodb-6.5.173 * branch admin -prepare for next development iteration * changed the BASE_URL * PD-256742: fixed all bugs. * PD-256742: changed snapshot version. * PD-256742: changed snapshot version. * branch admin -prepare release emodb-6.5.176 * branch admin -prepare for next development iteration * for testing * feat: integrate kafka to emodb * Create kafka producer and admin service * Overload the sendAll method to reroute messages to kafka * Make the required changes in Clients like jersey2 * feat: include logic for separation of queue and dedupq * feat: include guava dependency injection for emodb * fix: seperation of producer and admin service and proper injection * fix: fix DI wiring issue * feat: integrate triggering step function execution on creation of new topic * feat: merge changes for stepfunction * branch admin -prepare release emodb-6.5.175 * branch admin -prepare for next development iteration * feat: add parameter store * fix:add aws core dependency * fix:remove credentialProvider * fix: changes for ci deployment * fix: pom changes for blob * chore: update tags to 177 for deployment * branch admin -prepare release emodb-6.5.177 * branch admin -prepare for next development iteration * chore: update msk servers * fix: include msk server url * chore: snapshot update to 181 * branch admin -prepare release emodb-6.5.181 * branch admin -prepare for next development iteration * feat: include working parameter store and stepfn changes * add proper exception handling in parameter store and stepfn * make abstractqueue service code modular and organized * branch admin -prepare release emodb-6.5.182 * branch admin -prepare for next development iteration * feat: fetch kafka configs from ssm and dependency injection for stepfn * fetch aws configs from ssm parameter store * introduce dependency injection for stepfunction service * branch admin -prepare release emodb-6.5.183 * branch admin -prepare for next development iteration * feat: feature flag implementation for queue migrations * include the proper naming for each stepfn execution * include all required contracts for stepfn * include the feature flag logic, fetch params from parameter store and based on the allowed queues migrate * fix: fix param for allowedQueues * branch admin -prepare release emodb-6.5.184 * branch admin -prepare for next development iteration * fix: remove blob changes * chore: add comments and fix start.sh * fix: changed logic for feature flag * add changed logic for feature flag implementation * wrap the payload for stepfn around a executionInput parameter * fix: fixed exception handling and code refactoring * throw back exceptions after logging * include create topic and checktopic in one function * add comments for better description --------- Co-authored-by: vikram-vikram_bveng <[email protected]> Co-authored-by: jenkins <[email protected]> Co-authored-by: nabajyotiDash-hub <[email protected]> * update pom version * branch admin -prepare release emodb-6.5.192 * branch admin -prepare for next development iteration * modify producer properties (#834) * PD-257019 Add universe fetching and bug fixes (#835) * feat: add universe fetching and bug fixes * fix: remove debug logs * branch admin -prepare release emodb-6.5.193 * branch admin -prepare for next development iteration * uncached_size for dedupq (#837) * PD-257019 Producer send and logging changes (#838) * fix: fix send logic and exception handling changes * added exception handling * changed producer send logic to be synchronous * removed redundant logs * uncached_size for dedupq (#837) * chore: rename queuetype as per convention * chore: fix naming issue --------- Co-authored-by: nabajyotiDash-hub <[email protected]> * branch admin -prepare release emodb-6.5.194 * branch admin -prepare for next development iteration * branch admin -prepare release emodb-6.5.195 * branch admin -prepare for next development iteration * branch admin -prepare release emodb-6.5.196 * branch admin -prepare for next development iteration * chore: snapshot updates * branch admin -prepare release emodb-6.5.194 * branch admin -prepare for next development iteration * [PD-258160] added dd metrics for queue name and queue size (#836) * branch admin -prepare release emodb-6.5.195 * alter pom version for bvengineering deployment * branch admin -prepare release emodb-6.5.196 * update pom to 6.5.197-SNAPSHOT * branch admin -prepare release emodb-6.5.197 * branch admin -prepare for next development iteration * branch admin -prepare release emodb-6.5.198 * branch admin -prepare for next development iteration * add cache * add allowedQueues cache * feat: producer future logic and caching changes * chore: clear out unused code * PD-257019 Producer send and logging changes (#838) * fix: fix send logic and exception handling changes * added exception handling * changed producer send logic to be synchronous * removed redundant logs * uncached_size for dedupq (#837) * chore: rename queuetype as per convention * chore: fix naming issue --------- Co-authored-by: nabajyotiDash-hub <[email protected]> * fix: fix the config changes * Revert "Merge branch 'master' into PD-257019-producer-async-fixes" This reverts commit 93f5a76, reversing changes made to 15ce863. * chore: snapshot updates for deployement * chore: clear up testing logs * fix: remove default servers * branch admin -prepare release emodb-6.5.202 * branch admin -prepare for next development iteration * chore: caching logs refactoring * branch admin -prepare release emodb-6.5.203 * branch admin -prepare for next development iteration --------- Co-authored-by: vikram-vikram_bveng <[email protected]> Co-authored-by: jenkins <[email protected]> Co-authored-by: nabajyotiDash-hub <[email protected]> Co-authored-by: nabajyotiDash-hub <[email protected]> Co-authored-by: Anurag Dubey <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Github Issue
1234
What Are We Doing Here?
Here is where you should describe the problem you are solving as it relates to the Github Issue referred above, adding any fine
details on the solution that might otherwise not be recognizable for someone
unfamiliar with the changes. Add some pictures if it helps.
How to Test and Verify
Risk
Level
Low,Medium, orHigh. Give an indication of what you think is the level of change introduced by this PR.Highmeans a massive change to a core functionality.Lowmeans a really minor change that shouldn't have any regression effect.Required Testing
Smoke,Regression, orManual. (All changes except documentation need smoketesting at a minimum).
Risk Summary
Add one or a few complete sentences about the possible risks or concerns for
this change.
Code Review Checklist
Tests are included. If not, make sure you leave us a line or two for the reason.
Pulled down the PR and performed verification of at least being able to
build and run.
Well documented, including updates to any necessary markdown files. When
we inevitably come back to this code it will only take hours to figure out, not
days.
Consistent/Clear/Thoughtful? We are better with this code. We also aren't
a victim of rampaging consistency, and should be using this course of action.
We don't have coding standards out yet for this project, so please make sure to address any feedback regarding STYLE so the codebase remains consistent.
PR has a valid summary, and a good description.