fix(p2p): stream tx pool hydration to bound startup memory (A-968)#23086
Open
PhilWindle wants to merge 4 commits intomerge-train/spartanfrom
Open
fix(p2p): stream tx pool hydration to bound startup memory (A-968)#23086PhilWindle wants to merge 4 commits intomerge-train/spartanfrom
PhilWindle wants to merge 4 commits intomerge-train/spartanfrom
Conversation
hydrateFromDatabase previously deserialized every persisted tx into a
single { tx, meta }[] array before processing it. Tx objects (with proof
data) can be megabytes each, so a busy mempool (20k+ entries at 10 TPS)
OOMed the node on startup.
Stream the txs DB instead: build metadata and resolve mined status one
tx at a time, dropping the Tx reference before moving to the next entry.
Only the (much smaller) TxMetaData objects accumulate. The previously
separate #loadAllTxsFromDb and #markMinedStatusBatch helpers fold into
the streaming loop.
Both calls are independent I/O — running them in parallel roughly halves per-tx hydration latency for large mempools. getTxEffect rejections are still treated as non-fatal (tx is just considered not-yet-mined), matching the previous behavior.
…drating-txs-at-startup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hydrateFromDatabasewas deserializing every persisted tx into a single{ tx, meta }[]array before doing anything else.Stream the txs DB instead: deserialize, build metadata, and resolve mined status one tx at a time, dropping the Tx reference before moving on. Only the much smaller
TxMetaDataobjects accumulate. The previously separate#loadAllTxsFromDband#markMinedStatusBatchhelpers fold into the streaming loop (they were only used here).Fixes A-968.
Test plan
tx_pool_v2.test.tscases pass (including the hydration / restart suite).tx_pool_v2.compat.test.tscases pass.