fix(pubsub): Event queue becoming too long #886
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msbrogli
force-pushed
the
fix/long-queues
branch
from
a8ddf3d to
2fb2ba0
Compare
jansegre
reviewed
Dec 5, 2023
|
It ran much better and the node got further with less memory use, it got to about 1.9M blocks, with ~10GB of memory use, it didn't die, but it's stuck trying to download some transactions, this is what the log looks like: |
|
This is also another relevant exception that happened, not sure if it's related to the changes in this PR though: I think it happened because I forced a p2p reconnect with a USR1 signal. But it shouldn't cause any exceptions. |
jansegre
force-pushed
the
fix/long-queues
branch
from
5691806 to
8f7c55b
Compare
glevco
reviewed
Dec 8, 2023
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #886 +/- ##
==========================================
- Coverage 85.42% 85.29% -0.13%
==========================================
Files 282 283 +1
Lines 22269 22343 +74
Branches 3365 3382 +17
==========================================
+ Hits 19023 19058 +35
- Misses 2580 2605 +25
- Partials 666 680 +14 ☔ View full report in Codecov by Sentry. |
jansegre
force-pushed
the
fix/long-queues
branch
2 times, most recently
from
996132a to
8f7c55b
Compare
1 task
msbrogli
force-pushed
the
fix/long-queues
branch
5 times, most recently
from
c119d99 to
068201b
Compare
msbrogli
changed the title
jansegre
previously approved these changes
Dec 8, 2023
msbrogli
changed the title
msbrogli
force-pushed
the
fix/long-queues
branch
2 times, most recently
from
2d7d58f to
519d92c
Compare
2 tasks
jansegre
force-pushed
the
fix/long-queues
branch
from
fc97e7b to
c4bd15a
Compare
jansegre
approved these changes
Dec 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
In the sync-v1 process, there's a bottleneck occurring due to the differing speeds of data processing and data arrival. Firstly, the
sync_v1_agent.send_data_queue.priority_queueis unable to process transactions as quickly as they arrive. This discrepancy results in the queue continuously expanding throughout the synchronization process, keeping references toBaseTransactionobjects.In addition, each set of blocks received by sync-v1 triggers approximately 200 events in the pubsub system. However, similar to the first issue, the event queue processes these transactions at a slower rate than their arrival rate. Consequently, this queue also grows continually during synchronization, maintaining references to
BaseTransactionobjects.Both of these issues contribute to a critical memory management problem. Since these queues hold onto references to vertices, the garbage collector is unable to free up the memory space that these objects occupy.
For context, when operating a full node on my local computer (a MacBook), it initially consumed 3.19 GB of memory while processing 450,000 vertices. After this fix, the memory usage significantly decreased to just 0.62 GB. I also checked the number of blocks in memory every 25,000 blocks and this number was always between 800 and 2,000.
Useful commands:
Acceptance Criteria
Checklist
master, confirm this code is production-ready and can be included in future releases as soon as it gets merged