giga: link mem_block_db with Autobahn for block RPC lookups (CON-272)#3408
Open
wen-coding wants to merge 10 commits intomainfrom
Open
giga: link mem_block_db with Autobahn for block RPC lookups (CON-272)#3408wen-coding wants to merge 10 commits intomainfrom
wen-coding wants to merge 10 commits intomainfrom
Conversation
Replaces data.State's in-memory hash→height map with a writer/reader pair against sei-db/ledger_db/block.BlockDB (mem_block_db backend for now). runExecute writes the block before each executeBlock call; GigaRouter.BlockByHash reads via BlockDB.GetBlockByHash. The data-layer index, GlobalBlockByHash, and TestGlobalBlockByHash are removed. Crash recovery for BlockDB is not wired yet — the in-memory backend is per-process and reads return "unknown" for blocks finalized but not yet started executing. Will be revisited when a persistent backend lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wen-coding
changed the title
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3408 +/- ##
==========================================
- Coverage 59.10% 59.09% -0.01%
==========================================
Files 2108 2107 -1
Lines 173535 173492 -43
==========================================
- Hits 102564 102528 -36
+ Misses 62094 62086 -8
- Partials 8877 8878 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Replaces the BinaryBlock/BinaryTransaction byte-container types with Block/Transaction interfaces (Hash/Height/Time/Transactions and Hash/Bytes). mem_block_db stores the interface directly, so the tx-by-hash index works naturally without serialization. blocksim and the cross-backend tests get small synthetic implementations of the interfaces. On the giga side, the proto encode/decode helpers are gone — replaced with a globalBlockAdapter that wraps *atypes.GlobalBlock. Per-tx hashes use tmhash.Sum (sha256), matching CometBFT's tx-hash convention. The single translateBlock now serves both BlockByNumber (data.State path, wrapped via the adapter) and BlockByHash (BlockDB path), so the read path emits the same shape regardless of source. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-256) Extends block.Transaction with Result/Height/Index. Result returns (bytes, ok) — consumers tell "executed" from "block written, results not yet attached" without nil-as-sentinel. New BlockDB.SetTransactionResults(blockHash, []Result) attaches per-tx results post-execution. mem_block_db keeps results in a separate map and wraps stored Transactions on read with composedTx that overrides Result(). Blocksim's synthetic genTx returns (nil, false); test fixtures get a parallel testTx + testResult. GigaRouter.executeBlock now returns the FinalizeBlock TxResults too; runExecute calls SetTransactionResults right after execution, wrapping each *abci.ExecTxResult in execResultAdapter (lazy Marshal). New GigaRouter.Tx returns a fully-translated coretypes.ResultTx — env.Tx delegates to it under giga, mirroring the BlockByHash routing pattern. giga_router_test now round-trips every tx in the latest block through GigaRouter.Tx, asserting hash/height/index/bytes faithfulness and that TxResult.Code lands as 0 (testApp returns CodeTypeOK). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds ErrTxResultPending. GigaRouter.Tx now returns it when the parent block has been WriteBlock'd but SetTransactionResults hasn't run yet — the window inside runExecute that spans the entire FinalizeBlock + PushAppHash + Commit + mempool.Update sequence. Before this, that window returned a ResultTx with a zero-value TxResult. Code=0 is the success code, so a polling broadcast_tx_commit client could not distinguish "tx executed and succeeded" from "tx committed but not yet executed" — which would mislead pollers into treating a not-yet-executed tx (that may still fail) as a success. New TestGigaRouter_TxResultPending pins this in isolation: WriteBlock, assert ErrTxResultPending, SetTransactionResults, assert successful ResultTx. Built on a small txStub/blockStub/resultStub that satisfies the block.Block / Transaction / Result interfaces directly, so the test exercises GigaRouter.Tx without standing up the full consensus harness used by TestGigaRouter_FinalizeBlocks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-instance result (CON-256)
Fixes review finding (2): the same tx hash included in two different
GlobalBlocks (different lanes producing the same tx) used to overwrite
each other in mem_block_db's flat tx-by-hash map, and a Prune of the
older block silently delete the index entry that pointed at the still-
retained newer one.
The Transaction interface now carries only the invariant body
(Hash + Bytes). Per-block-occurrence data (height, index, marshaled
exec result) moves to Result, returned alongside Transaction by
GetTransactionByHash:
GetTransactionByHash(ctx, hash) (tx Transaction, results []Result, found bool, err)
`results` lists every recorded execution (one per block that has had
SetTransactionResults called for it); pending entries are filtered out
so callers get exactly the executions, never empty wrappers. The
caller (GigaRouter.Tx here) decides which one is canonical:
1. Lowest-height execution with abci.CodeTypeOK (a tx is expected
to succeed at most once across the chain — lowest height is just
a deterministic tiebreaker).
2. Else highest-height failure (most recent attempt).
3. Else (found && len(results)==0) → ErrTxResultPending.
mem_block_db keeps a per-tx-hash entry with a two-level map of
{blockHash -> {height, index, bytes}}. WriteBlock registers a pending
instance per (txHash, blockHash); SetTransactionResults attaches
bytes; Prune removes only the per-block instance and drops the entry
when the inner map empties. Other blocks containing the same tx hash
stay reachable.
New tests:
- block_db_test.TestTransactionMultipleBlocks: shared tx hash across
two blocks, both results reachable, pruning either leaves the
other intact.
- giga_router_test.TestGigaRouter_TxMultipleBlocks_PrefersSuccess:
A=fail, B=success → Tx returns B regardless of insertion order.
- giga_router_test.TestGigaRouter_TxMultipleBlocks_FallsBackToLatestFailure:
both fail → Tx returns the highest-height failure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s (CON-256)
Addresses review findings 3-5 plus the doc gaps flagged on the
Block/Result interfaces:
(3) Stale comment in giga_router.go referenced data.State's prior
blockHashes hash index — the index was removed in commit 1.
Rewritten to describe the current contract (BlockDB, RPC unknown-
hash semantics) without the dangling reference.
(4) globalBlockAdapter.Transactions() used to allocate a fresh
[]block.Transaction and re-sha256 every payload tx on every call.
mem_block_db calls it on WriteBlock, SetTransactionResults, and
Prune — the Prune call happens under the write lock. Switched to
a newGlobalBlockAdapter constructor that builds the slice once
and caches it; Transactions() now returns the cached slice.
(5) mem_block_db.SetTransactionResults marshaled each result via
Result.Bytes() inside the write lock. For a 1000-tx block that's
~MB of proto Marshal blocking every concurrent reader. Pre-marshal
happens before acquiring the lock; the lock now holds only the map
writes. The error paths (unknown block, count mismatch) waste the
pre-marshal but are exceptional.
Doc updates:
- Block.Transactions: contract pinned ("must be cheap to call
repeatedly; backends may call it more than once").
- Result.Bytes: lifecycle pinned ("backends call it exactly once
per result inside SetTransactionResults and cache the bytes").
New test: TestSetTransactionResultsOverwrites — exercises the
documented "second call replaces" semantics, useful for callers that
re-execute on recovery.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wen-coding
changed the title
…, deterministic reads (CON-272)
Addresses the second-round review findings on mem_block_db:
(1) WriteBlock idempotency. A second WriteBlock for the same block hash
used to silently overwrite the entire instances map for every tx in
the block — including any results SetTransactionResults had already
attached. WriteBlock now no-ops on duplicate blockHash.
(2) Deterministic Tx selection on ties. mem_block_db.GetTransactionByHash
iterated entry.instances in random Go map order, so when GigaRouter.Tx
needed a tie-breaker between two executions at the same height the
chosen winner depended on iteration order — different across calls,
different across nodes. Sort the inner map keys by blockHash so the
returned slice is stable. With the existing GigaRouter.Tx selection
(lowest-height success / highest-height failure), the tie-break is
now "first lexicographic blockHash wins."
(3) Read isolation. composedResult was a value copy of the internal
resultInstance, but the bytes []byte shared the underlying array
with the storage slot. A subsequent SetTransactionResults overwrite
of the same instance would mutate what the caller was reading.
Deep-copy bytes on read so the returned Result is immune.
While here, drop composedResult entirely — it had identical fields to
resultInstance, so the latter now implements block.Result directly.
(6) Tx body collision detection. New ErrTxHashCollision sentinel.
WriteBlock validates that any pre-existing entry for a tx hash has
matching bytes; mismatch refuses the entire write (two-pass
validate-then-mutate, so partial state isn't left behind on
rejection). Catches a bug class — wrong-hash-fn, malformed input —
that would otherwise silently keep the first-writer's bytes.
New tests:
- TestWriteBlockIdempotent: re-WriteBlock preserves attached results.
- TestWriteBlockTxHashCollision: mismatched second-block bytes are
rejected; partial state isn't left behind.
- TestGetTransactionByHashDeterministicOrder: 3 blocks at same height
with same tx hash, repeated reads return the same order.
- TestGetTransactionByHashReadIsolation: an earlier-read Result is
immune to a later SetTransactionResults overwrite.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third-round review cleanup, focused on removing dead defensive code
and documenting the trade-offs we decided to defer rather than fix:
GetTransactionByHash deep-copy was dead defense. The comment claimed
the deep-copy isolated callers from a later SetTransactionResults
overwrite — but SetTransactionResults reassigns inst.bytes (slice
header swap) rather than mutating bytes in place, so Go's value-copy
of the resultInstance struct already isolates the caller's slice
header. Removed the make+copy; updated the resultInstance type doc
and TestGetTransactionByHashReadIsolation comment to describe what
provides isolation now (and to call out that the test still catches
the regression if a future change makes SetTransactionResults mutate
in place).
ErrTxHashCollision doc no longer claims "without cost." A rejected
write means the corrupted tx hash is permanently poisoned — every
future legitimate block reusing the hash will also fail until pruned
out. Doc now flags this as operator-attention territory rather than
something to retry.
ErrTxResultPending doc is honest about the dead-process retry case:
on the happy path retry lands in milliseconds, on the unhappy path
(executeBlock errored, runExecute exited) the result will never
land — operators will see the sentinel forever for any tx in the
orphaned block.
Follow-up TODOs left in code (deferred to subsequent PRs):
- giga_router.go blockDB field: BlockDB DI through GigaRouterConfig;
blockDB.Prune wiring (mem_block_db grows without bound today).
- giga_blockdb.go globalBlockAdapter.Hash(): memoize parallel to
txs cache.
- mem_block_db.go memBlockDB type: -race concurrency test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wen-coding
changed the title
…(CON-272)
The Giga Transaction Query Architecture proposal makes the BlockDB /
Receipt Store split explicit: BlockDB is block-storage-only;
canonical txHash → execution result lookup belongs on a separate
Receipt Store unified for EVM and Cosmos txs. This commit pulls all
the tx-by-hash work out of BlockDB to align.
Removed from sei-db/ledger_db/block:
- block.Result interface
- BlockDB.GetTransactionByHash, BlockDB.SetTransactionResults
- ErrUnknownBlock, ErrResultCountMismatch, ErrTxHashCollision
- mem_block_db's two-level txEntries / resultInstance index, the
sort-by-blockHash read order, the pre-marshal-outside-lock
optimization, the tx-body collision check.
Removed from sei-tendermint/internal/p2p:
- execResultAdapter
- GigaRouter.Tx
- ErrTxResultPending
- executeBlock's []*abci.ExecTxResult return value
- The runExecute SetTransactionResults call
Removed from sei-tendermint/internal/rpc/core/tx.go:
- The env.Tx delegate to GigaRouter.Tx. /tx now reverts to the
legacy CometBFT path under Autobahn (returns "querying disabled"
since EventSinks are empty); a TODO points at the Receipt Store
follow-up.
Removed from giga_router_test.go:
- txStub / blockStub / resultStub fixtures
- TestGigaRouter_TxResultPending
- TestGigaRouter_TxMultipleBlocks_PrefersSuccess
- TestGigaRouter_TxMultipleBlocks_FallsBackToLatestFailure
- The Tx round-trip block inside TestGigaRouter_FinalizeBlocks
Removed from block_db_test:
- TestSetTransactionResults*, TestTransactionMultipleBlocks,
TestGetTransactionByHash*, TestWriteBlockTxHashCollision.
TestWriteBlockIdempotent stays — block-level idempotency still
applies and is worth pinning.
What remains (the original PR scope):
- block.Block / block.Transaction interfaces, BlockDB block-only API
- mem_block_db with blockHash/height indexes + idempotent WriteBlock
- globalBlockAdapter / txAdapter (memoized at construction)
- runExecute calls WriteBlock before executeBlock
- GigaRouter.BlockByHash delegates to BlockDB.GetBlockByHash
- translateBlock unifies BlockByNumber + BlockByHash translation
- data.State.blockHashes map + GlobalBlockByHash removed (the map
hack from #3310)
Net diff: ~950 lines removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires
sei-db/ledger_db/block.BlockDB(today'smem_block_db, in-memory) into Autobahn soenv.BlockByHashresolves through it under giga, replacing the temporarydata.State.blockHashesmap introduced in #3310. Block-only by design — per the Giga Transaction Query proposaltxHash → execution resultlookup belongs on a separate Receipt Store, not on BlockDB.What changed
sei-db/ledger_db/block — block-only typed interfaces
Block—Hash,Height,Time,Transactions. Backends may callTransactions()repeatedly.Transaction—Hash,Bytes. BlockDB does not index by tx hash; per-txHash()exists so a future Receipt Store (or any other consumer) can iterateBlock.Transactions()and register its own(txHash → block, index)mapping at write time.BlockDBsurface:WriteBlock,Flush,GetBlockByHash,GetBlockByHeight,Prune,GetLowestBlockHeight,GetHighestBlockHeight,Close. No tx-by-hash, noResult, noSetTransactionResults.WriteBlockis idempotent on duplicate block hash (silent no-op).mem_block_db backend
blockHash → Blockandheight → Block. StandardPrune. Backed by a singlesync.RWMutex.Autobahn / giga
runExecutewrites the block to BlockDB beforeexecuteBlock, soBlockByHashis RPC-visible from finalization. No tx-result tracking here (deferred to Receipt Store).globalBlockAdapterwraps*atypes.GlobalBlockto satisfyblock.Block. Per-tx hashes viatmhash.Sum. Slice + hashes computed once at construction and memoized.txAdapterwraps a single Autobahn tx + its CometBFT-style hash to satisfyblock.Transaction.GigaRouter.BlockByHashdelegates toBlockDB.GetBlockByHash;BlockByNumberstill pulls fromdata.State.GlobalBlock. Both feed through a singletranslateBlock(block.Block)to produce the samecoretypes.ResultBlockshape.env.BlockByHashdelegates toGigaRouter.BlockByHashwhen running under giga; CometBFT path untouched.env.Txis unchanged (returns "querying disabled" under Autobahn since EventSinks are empty); will be wired through the Receipt Store in a follow-up.Removed
data.State.inner.blockHashesmap anddata.State.GlobalBlockByHash(the temporary hash index from rpc: route /block + /block_results + /block_by_hash + /validators through Autobahn under GigaEnabled (CON-257) #3310).Why this scope
Per the Giga Transaction Query proposal, the canonical
txHash → execution resultroute lives on a unified Receipt Store (already used by EVM today, to be extended for Cosmos txs). BlockDB stays block-storage-only. This PR therefore deliberately does not include:BlockDB.GetTransactionByHash/SetTransactionResults/ResultinterfaceGigaRouter.Tx/env.Txrouting through BlockDBErrTxResultPendingEarlier iterations of this branch had all of the above; they were pulled out once the proposal made the BlockDB / Receipt Store split explicit.
Known limitations / follow-ups
mem_block_dbgrows without bound in this PR.runExecutecallsdata.PruneBeforebut neverblockDB.Prune. WiringPruneis the obvious next step alongside a persistent backend.BlockDBis not injectable:NewGigaRouterhard-codesmemblockdb.NewMemBlockDB(). Acfg.BlockDBOption is the natural fix when the persistent backend lands.mem_block_dbis empty;BlockByHashreturns&ResultBlock{Block: nil}until blocks get re-executed. The data layer's WAL remains the primary durability story.-raceconcurrency tests onmem_block_db. Lock invariants verified only by inspection./tx,/tx_search,/block_search,/broadcast_tx_commitslated for follow-up work per the proposal —/txto be wired through the Receipt Store; the search andbroadcast_tx_commitpaths to be sunset experimentally on Giga testnet.Test plan
go build ./...cleangofmt -s -lcleango vet ./...cleangolangci-lint runon touched packages:0 issuesgo test ./sei-db/ledger_db/block/...— passgo test ./sei-tendermint/internal/p2p/...— pass (incl.TestGigaRouter_FinalizeBlocksend-to-end round-trip withBlockByHash)go test ./sei-tendermint/internal/rpc/core/...— passgo test ./sei-tendermint/internal/autobahn/...— passgo test -tags autobahn_integration ./integration_test/autobahn/...— pass (~130s)🤖 Generated with Claude Code
Note
Medium Risk
Medium risk because it changes core block lookup behavior under Autobahn/Giga and refactors the
BlockDBAPI to new interfaces, which could affect RPC correctness and memory growth (in-memory BlockDB is currently unpruned).Overview
Routes Autobahn/Giga
BlockByHashRPC lookups throughsei-db’sBlockDB(currentlymem_block_db), withrunExecutewriting each finalized block into BlockDB before execution and a newglobalBlockAdaptertranslatingGlobalBlockinto theblock.Blockinterface.Refactors
BlockDBto be block-only by introducingblock.Block/block.Transactioninterfaces (including blockTime()) and removing tx-by-hash APIs;mem_block_dbis updated accordingly andWriteBlockis now explicitly idempotent on duplicate block hash (pinned by new tests).Removes the temporary in-memory
data.Stateblock-hash index and itsGlobalBlockByHashquery path, and adds a TODO inrpc/corenoting/txwill later be routed via a separate Receipt Store.Reviewed by Cursor Bugbot for commit fe4fb27. Bugbot is set up for automated code reviews on this repo. Configure here.