Skip to content

Improve elements memory usage #1190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jan 26, 2023
Merged

Improve elements memory usage #1190

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
68eb086
Refactor CBlockIndex to make proof and dynafed fields optional.
gwillen Nov 8, 2022
7c55557
Add a flag for header-trimming; clear P2P network feature bits when e…
gwillen Nov 8, 2022
eff6be3
Trim headers when loading block index from disk.
gwillen Nov 8, 2022
4f0da3c
Trim headers when flushing to disk, and reduce flush interval from 1h…
gwillen Nov 8, 2022
f5c7995
When people ask us for headers we do not have (over REST or RPC), do …
gwillen Nov 10, 2022
7e6fe4a
Don't let header syncing get too far ahead of block syncing
gwillen Nov 15, 2022
a2db974
Fix false-positive lint
gwillen Nov 15, 2022
763d298
Fix deserialization issues in certain functional tests
gwillen Nov 16, 2022
394e59d
Politely and safely refuse getheaders requests for headers we have tr…
gwillen Nov 23, 2022
fba37b7
Move enough-headers check up, rejecting excess headers when received.
gwillen Dec 7, 2022
29345e9
pgreco's startup crash fix
gwillen Dec 7, 2022
b9b7f65
Apply suggestions from code review
gwillen Jan 4, 2023
8cad8cd
Further fixes from code review
gwillen Jan 5, 2023
e8475e7
Fix stuck IBD when discarding unwanted headers from peers.
gwillen Jan 25, 2023
e622156
Update documentation
gwillen Jan 25, 2023
15dbe64
Remove stray comment
gwillen Jan 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion src/validation.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ static const unsigned int EXTRA_DESCENDANT_TX_SIZE_LIMIT = 10000;
/** Maximum kilobytes for transactions to store for processing during reorg */
static const unsigned int MAX_DISCONNECTED_TX_POOL_SIZE = 20000;
/** Time to wait between writing blocks/block index to disk. */
static constexpr std::chrono::hours DATABASE_WRITE_INTERVAL{1};
static constexpr std::chrono::minutes DATABASE_WRITE_INTERVAL{5};
/** Time to wait between flushing chainstate to disk. */
static constexpr std::chrono::hours DATABASE_FLUSH_INTERVAL{24};
/** Maximum age of our tip for us to be considered current for fee estimation */
Expand Down Expand Up @@ -2346,13 +2346,40 @@ bool CChainState::FlushStateToDisk(
}
std::vector<const CBlockIndex*> vBlocks;
vBlocks.reserve(setDirtyBlockIndex.size());
std::set<CBlockIndex*> setTrimmableBlockIndex(setDirtyBlockIndex);
for (std::set<CBlockIndex*>::iterator it = setDirtyBlockIndex.begin(); it != setDirtyBlockIndex.end(); ) {
vBlocks.push_back(*it);
setDirtyBlockIndex.erase(it++);
}
if (!pblocktree->WriteBatchSync(vFiles, nLastBlockFile, vBlocks)) {
return AbortNode(state, "Failed to write to block index database");
}

if (fTrimHeaders) {
LogPrintf("Flushing block index, trimming headers, setTrimmableBlockIndex.size(): %d\n", setTrimmableBlockIndex.size());
int trim_height = m_chain.Height() - nMustKeepFullHeaders;
int min_height = std::numeric_limits<int>::max();
CBlockIndex* min_index = nullptr;
for (std::set<CBlockIndex*>::iterator it = setTrimmableBlockIndex.begin(); it != setTrimmableBlockIndex.end(); it++) {
(*it)->assert_untrimmed();
if ((*it)->nHeight < trim_height) {
(*it)->trim();
if ((*it)->nHeight < min_height) {
min_height = (*it)->nHeight;
min_index = *it;
}
}
}

// Handle any remaining untrimmed blocks that were too recent for trimming last time we flushed.
if (min_index) {
min_index = min_index->pprev;
while (min_index && !min_index->trimmed()) {
min_index->trim();
min_index = min_index->pprev;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the logic which trims blocks that were new when we got them, but are now old enough that they should be trimmed?

It looks like it only gets triggered when we flush to disk. How often does this flushing take place?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. Adjusting the flush frequency is part of this PR elsewhere -- I think I lowered it from 1 hour to 5 minutes. It's not ideal for it to be time-based instead of memory-pressure-based, but that would be a more invasive change.

}
}
}
// Finally remove any pruned files
if (fFlushForPrune) {
Expand Down