tbc: add ordinal indexer#1053
Open
marcopeereboom wants to merge 42 commits into
Open
Conversation
Add ordinal indexer to TBC that tracks individual satoshi ownership via sat ranges and indexes Bitcoin inscriptions. Follows the existing Indexer/indexer framework pattern, modeled on zkindexer.go. Database: - New LevelDB "ordinals" with prefix-separated keys: 'r' (sat ranges per UTXO), 'i' (inscription by ID), 's' (inscribed sat → outpoint), 'a' (sat → all inscriptions), 'n' (inscriptions by block). - OrdinalKey type, 7 new Database interface methods. - Database version bump 5 → 6. Indexer: - FIFO sat range redistribution engine. - Inscription envelope parser supporting all tags: content type, pointer, parent, delegate, metaprotocol. - Cursed inscription detection (5 pre-jubilee rules). - Flag-driven variable-length value encoding. - Inscribed sat movement tracking via DB range scan. - Zero dependency on txindex during wind/unwind. Wiring: - Config: OrdinalIndex bool, MaxCachedOrdinals int. - Sync order: ui → ti → ki → zki → oi. - Prometheus gauge, SyncInfo field, Synced() check. Tests: - 41 test functions covering FIFO engine, sat computation, envelope parsing, cursed detection, value encoding, pointer tag handling, key construction, cache scanning. - Negative tests for malformed envelopes, boundary conditions, and edge cases. - Updated TestDbUpgradeFull and TestDbUpgradeV5 for v6.
Implement the 6 ordinal RPCs defined in the SOW: - InscriptionByID: lookup inscription by txid + input index - InscriptionContent: retrieve raw content, follows delegation chains - InscriptionsByBlock: enumerate inscriptions created in a block - InscriptionsByAddress: cross-reference utxo index to find held inscriptions - InscriptionsBySat: list all inscriptions on a sat (reinscription support) - SatRangesByOutpoint: diagnostic lookup of sat ranges for a UTXO Each RPC follows the existing dispatch pattern: Cmd constants and request/response structs in api/tbcapi, dispatch cases in getTBCAPICommandHandler, handler methods in rpc.go, and Server business logic methods in tbc.go. Adds decodeInscriptionValue (mirrors encodeInscriptionValue) with round-trip tests including positive and negative cases. Adds OrdinalInscriptionsBySat to the Database interface (iterates the 'a' prefix for all inscription IDs on a given sat).
Wire OrdinalIndex to the TBC_ORDINAL_INDEX environment variable so the ordinal indexer can be enabled at runtime. Follows the existing TBC_HEMI_INDEX / TBC_ZK_INDEX pattern.
Performance fixes (measured before/after on testnet4 heavy zone): 1. Block-level inscribed-sat pre-scan: collect all input sat ranges per block, single DB scan for the merged range. Replaces per-input DB scans. O(inputs_per_block) → O(1). inscSat phase: 5m51s → 965ms (363x). 2. Sorted inscribed-sat slice with binary search: store block inscribed sats as sorted []uint64, binary search to find only sats overlapping each tx input range. Eliminates O(N) iteration over 96K+ inscribed sats per tx. Blocks taking 1.7s → <500ms. Zero SLOW blocks across full testnet4 sync. 3. Parallel fixupCacheHook: 128-way concurrent DB reads for input sat range pre-fetch. Mirrors fixupCacheChannel from utxoindex.go. 4. Min/max inscribed-sat boundary tracking: skip DB scan when block input range falls outside [minInscribedSat, maxSat]. 5. OrdinalInscribedSatBounds: two LevelDB iterator seeks for min/max on restart. Replaces OrdinalInscribedSatsInRange(0, MaxUint64) which would load 70M entries (560MB) on mainnet. Correctness fixes: 6. Fee sat conservation: two-pass windBlock processes non-coinbase txs first to collect fee ranges, then coinbase with subsidy + fees. Without this, fee sats vanish from the index. 7. Zero-value outputs: record empty sat ranges for txOut.Value==0 so spendable UTXOs exist in the index. Root cause: testnet4 block 32203 tx 73acc6ca...ff0e output 1 (value=0, v0_p2wpkh). Adds benchmarks (ordinalindex_bench_test.go) validating each optimization and design doc (ordinalindex_design.go) documenting rationale with measured results.
TestOrdinalIndexFork exercises the ordinal indexer through
full wind/unwind cycles across three competing chains:
Chain geometry:
/-> b1a -> b2a (inscription A)
genesis -> b1 -> b2 (inscription B) -> b3
\-> b1b -> b2b (inscription C)
Exhaustive verification of all 5 ordinal entry types on every
wind/unwind transition:
- n: InscriptionsByBlock (block sequence)
- i: InscriptionByID (inscription data, txid, block hash)
- s: OrdinalOutpointBySat (inscribed sat -> current outpoint)
- a: InscriptionsBySat (sat -> inscription IDs)
- r: SatRangesByOutpoint (UTXO sat ranges, zero-value outputs)
Precise sat number assertions verify FIFO split correctness:
- [5B, 10B) splits to [5B, 8B) + [8B, 10B) + empty
- create-and-spend in same block: [5B, 8B) -> [5B, 6.1B) + [6.1B, 8B)
- coinbase subsidy: [10B, 15B) at height 2, [15B, 20B) at height 3
- ranges verified on all 3 forks, confirmed absent after each unwind
- inscribed sat tracks through tx chain (s entry at b3 tx2:0)
Fixes found by the test:
- ordinalSatInscriptionKey value nil -> []byte{} (a prefix was
silently deleted on every inscription)
- batch-local annihilation: when a sat range is created and spent
within the same cache batch, delete from cache instead of marking
nil. Mirrors utxo indexer IsDelete/annihilation pattern via
batchCreated set. Without this, unwind cannot restore ranges
that were never persisted to DB.
parseEnvelopeTags did not recognize OP_0 (0x00) as the body separator because envelopeTag() only matched OP_1..OP_16 and single-byte data pushes. OP_0 returned -1, causing the body content push to be silently skipped. Every InscriptionContent call returned an empty body. Add explicit OP_0 handling before the envelopeTag check. After OP_0, all subsequent data pushes until OP_ENDIF or a recognized tag are collected as body content. The existing case 5 (OP_5 body encoding) is preserved for backward compatibility.
Expose ordinal cache capacity as a configurable env var. Default remains 1e6. Allows tuning for machines with less memory.
Two root causes of ordinal index data loss, fixed together: 1. OrdinalKey was type string, causing heap-allocated map keys. Under heavy write load (~960K entries), GC pressure on map[string][]byte corrupted goleveldb batch index during Transaction.Write, triggering panic: "leveldb: invalid type". Fix: OrdinalKey is now [45]byte, matching Outpoint and TxKey. OrdinalValue type added with IsDelete()/Bytes() methods, mirroring CacheOutput for the utxo indexer. 2. Cache entries silently lost between flush cycles. The batchCreated annihilation pattern used delete(cache, key) to remove map keys during processing. Combined with fetchSatRangesParallel goroutines racing against cache.Clear() on early return, this caused 10.57% of ordinal range entries (1,488,292 / 14,079,206) to be missing from DB after a full testnet4 index. Fix: remove batchCreated entirely. Spent outpoints are marked with nil (overwrite) instead of deleted from the map, matching the utxo indexer sentinel pattern. Add defer w.Wait() in fixupCacheHook and unwindBlock so goroutines always complete before the cache can be cleared. After: 0 missing ordinal range entries across 14,079,300 UTXOs.
Wire Is* methods (IsRange, IsSat, IsInscription, IsSatInscription, IsBlockInscription) in existing TestKeyConstruction subtests, replacing raw byte prefix checks with the typed methods. Add TestOrdinalKeyIsMethods: table-driven cross-validation that each key constructor activates exactly one Is* method. Includes zero-key negative case. Add TestOrdinalValueMethods: nil-is-delete sentinel, non-nil, and empty-non-nil boundary cases. Fix redundant txid copy in SatRangesByOutpoint — chainhash.Hash is already [32]byte, no intermediate variable needed.
E2E tests (bitcoind + TBC + ordinal indexer + RPC websocket): TestRpcOrdinalSatRanges: mines 10 blocks via bitcoind, syncs TBC with ordinal indexing, queries coinbase sat ranges via websocket RPC. Verifies FIFO sat allocation at height 1 and height 5 — proves the full pipeline: bitcoind → P2P → TBC → ordinal indexer → LevelDB → RPC response. TestRpcOrdinalNotFound: negative tests for all 5 ordinal RPC endpoints (inscription by ID, inscription content, inscriptions by block, inscriptions by sat, sat ranges by outpoint). Verifies correct not-found/empty responses through the dispatch layer. Helper: createTbcServerWithOrdinals creates a TBC server with OrdinalIndex enabled and AutoIndex false for explicit sync control via SyncIndexersToHash. Unit tests: FuzzParseInscriptionEnvelope: fuzz test for the inscription parser. Seeds cover valid envelope, empty witness, truncated data, wrong magic, and garbage. 45K+ executions without panic in initial run.
TestRpcOrdinalInscriptionE2E proves the complete inscription
lifecycle through every component:
gozer wallet → TBC RPC broadcast → bitcoind mempool →
bitcoin block → TBC P2P sync → ordinal indexer → TBC RPC query
The test builds a real taproot inscription using the existing
wallet infrastructure:
1. Mine 101 blocks via bitcoind (mature coinbase)
2. gozer.UtxosByAddress finds a spendable P2PKH UTXO
3. Build commit TX: P2PKH spend → P2TR output with inscription
script commitment (single-leaf tap tree, OP_TRUE + ord envelope)
4. wallet.TransactionSign signs the P2PKH input
5. gozer.BroadcastTx sends commit through TBC to bitcoind
6. Mine to confirm
7. Build reveal TX: script-path spend of committed P2TR with
inscription witness [tapscript, controlBlock]
8. gozer.BroadcastTx sends reveal through TBC to bitcoind
9. Mine to confirm
10. TBC syncs, SyncIndexersToHash processes ordinals
11. Verify via websocket RPC: InscriptionByID, InscriptionContent
(content type + body), SatRangesByOutpoint
No bitcoin-cli shortcuts for transaction handling — all TX
construction and signing goes through the gozer/wallet stack.
Adds 18 tests targeting every uncovered branch in the ordinal inscription parser: applyTag (25% → 100%): tag 2 pointer (valid + oversized), tag 3 parent (valid + invalid length), tag 7 metaprotocol, tag 11 delegate (valid + invalid length), default even/odd — all exercised through the OP_0 body mid-tag path which is the only call site for applyTag. envelopeTag (80% → 100%): single-byte data push (OP_DATA_1) tag encoding path. parseEnvelopeTags (76.4% → 92.7%): tag 5 (OP_5 alternate content encoding), OP_5 followed by recognized tag, body followed by tag, tag value tokenizer exhaustion. parseEnvelopeFromScript (86.7% → 93.3%): tokenizer exhaustion after OP_FALSE, after OP_FALSE OP_IF. Remaining gaps are indexer error paths in ordinalindex.go (DB failures, context cancellation) that require mock injection.
TestRpcOrdinal exercises all 6 ordinal RPC handlers through the websocket dispatch layer without requiring bitcoind. Seeds LevelDB directly with ordinal entries via BlockOrdinalUpdate, starts TBC with OrdinalIndex enabled, connects via websocket, and runs 7 table-driven subtests. Mirrors the TestRpcZK pattern. Positive: SatRangesByOutpoint verified with pre-seeded range. Negative: InscriptionByID, InscriptionContent, InscriptionsByBlock, InscriptionsBySat, InscriptionsByAddress, SatRangesByOutpoint — all not-found/empty paths exercised. RPC handler coverage: 0% → 55-78%.
FuzzDecodeSatRanges: fuzz decode with round-trip verification (decode → re-encode → decode must match). Skips non-multiple-of-16 inputs (panic by design). 20K+ executions clean. FuzzDecodeInscriptionValue: fuzz decode with round-trip through encodeInscriptionValue. Covers all flag combinations (cursed, parent, delegate, metaprotocol). 60K+ executions clean. FuzzDecodeVarUint: fuzz the little-endian variable-length uint decoder. 76K+ executions clean.
decodeInscriptionValue: max uint64 sat round-trip, empty metaprotocol (flag set, zero bytes remaining), unknown flag bits (4-7 set, decoder ignores), exact 41-byte minimum, metaprotocol-only (no parent/delegate). EncodeSatRanges/DecodeSatRanges: zero-count range round-trip, max uint64 Start/Count round-trip. decodeVarUint: max uint64 (8 × 0xff), overflow (>8 bytes silently ignored via Go shift-past-64 semantics).
TestBlockOrdinalUpdateAndQuery: 16 subtests covering all 9 ordinal DB functions with positive and negative paths. Seeds LevelDB directly via BlockOrdinalUpdate, then queries each function. Positive: OrdinalSatRangesByOutpoint, OrdinalInscriptionByID, OrdinalInscriptionsByBlockHash, OrdinalInscriptionsBySat, OrdinalOutpointBySat, OrdinalInscribedSatsInRange, OrdinalInscribedSatBounds. Negative: not-found for outpoint/inscription/sat lookups, empty results for block/sat range queries, BlockHeaderByOrdinalIndex without block headers. BlockOrdinalUpdate unwind: verifies direction=-1 deletes entries. TestBlockOrdinalUpdateEmptyCache: nil cache does not error.
TestRpcOrdinal: 13 table-driven subtests exercising all 6 ordinal RPC handlers through the websocket dispatch layer. Seeds LevelDB with ordinal + UTXO entries. Positive and negative paths for all handlers including InscriptionsByAddress. TestPrometheusOrdinalMetric: starts TBC with OrdinalIndex + PrometheusListenAddress, curls /metrics, verifies ordinal_sync_height gauge is present. TestDbUpgradeV6: validates v5→v6 database upgrade (version bump, data survives, ordinal DB functional after upgrade). hemictl tbcdb: add ordinalrangesbyoutpoint, ordinalinscriptionbyid, ordinalinscriptionsbyblock, ordinalinscriptionsbysat commands for direct database inspection.
Wire ordinal read cache into satRanges and fetchSatRangesParallel.
Check LRU between write cache miss and DB read, populate on DB hit,
clear on sync complete via onSyncComplete hook.
Config: TBC_ORDINAL_READ_CACHE_SIZE (default "1gb").
Prometheus: ordinal_read_cache_{hits,misses,purges,size,items}.
Log line: rcache hits/usage percentages via readCacheInfo.
Fix BlockheaderCacheSize rename in test files.
Set ordinal indexer genesis to block 766854 (first inscription 767430 minus 576). Testnet3/4/localnet start from chain genesis. Mainnet block hash is a placeholder — fill before mainnet use.
Remove all sat range precomputation from windBlock/windTx. The ordinal indexer now only scans witness data for inscription envelopes and stores reveals. No sat numbers, no FIFO redistribution, no s-key tracking, no read cache, no fixup pre-fetching. Sat numbers and transfer tracking are deferred to query time via backward walk to coinbase using raw blocks and tx index. Remove ordinal read cache (config, prom, Server field) since there are no DB reads during indexing. Skip TestOrdinalIndexFork — expects s-key tracking which no longer exists at index time.
Replace precomputed sat range DB lookups with on-demand computation in SatRangesByOutpoint. Walks backward through the spending chain via tx index and raw blocks to derive sat ranges at query time. Coinbase outputs return subsidy-only ranges; fee sat computation is deferred (requires resolving all block txs which is expensive). Reduce hemictl cache allocations for CLI use (block cache 64mb, disable utxo read cache). Remove synthetic 'r'/'s'/'a' seed data from RPC tests. Skip SatRangesByOutpoint, InscriptionsBySat, and InscriptionsByAddress test cases that depend on removed precomputed data.
populateInscription now derives the inscribed sat number via backward walk when the stored value is 0. Looks up the inscription's input outpoint, computes sat ranges, and takes the first sat. This makes InscriptionByID and InscriptionsByBlock return actual sat numbers instead of 0. Removes OrdinalOutpointBySat lookup (current location tracking) from populateInscription — transfer tracking is deferred. Note: sat numbers are subsidy-only approximations until coinbase fee computation is implemented.
Both RPCs depended on precomputed sat→inscription and sat range lookups that were removed. Stub with TODO and return empty results until outpoint→inscription tracking is implemented at index time.
Replace full-output range computation in computeInscribedSat with satTracer.traceSat: follows one sat through the FIFO using amounts only (flat tx index lookups), always linear to coinbase. At coinbase: if sat offset < subsidy, deterministic. If in fee range, fee amounts identify which specific tx contributed the fee sat, then follows that tx's input chain (same linear walk). No fan-out, no resolving all block txs. Handles fees correctly without the exponential blowup of the full-output approach. Retain satRangeContext.compute for SatRangesByOutpoint RPC (subsidy-only at coinbase — full output ranges are rarely queried).
Add TBC_REQUEST_TIMEOUT config option (default 10s, tbcd default 120s) replacing the hardcoded value. Longer timeouts accommodate on-demand sat computation for deep ancestry chains. Propagate context cancellation from computeInscribedSat through populateInscription as an error instead of silently returning sat_number=0. Add ctx.Err() check in the tracer loop to abort promptly when the request context expires.
NewServer rejects RequestTimeout <= 0 to prevent
context.WithTimeout(ctx, 0) from expiring every RPC call.
Test configs constructing &Config{} directly got Go zero value
instead of NewDefaultConfig's default, causing instant timeouts.
Add RequestTimeout: 10 to all test Config{} structs across
ordinalindex_rpc_test, rpc_test, tbcfork_test, tbc_test,
utxoindex_test, and tbcgozer_test.
Remove utxo read cache stats log at tip — the same data is
already in the periodic status line during indexing.
Change blockCache from lru.Cache[chainhash.Hash, []byte] to lru.Cache[chainhash.Hash, *btcutil.Block]. Cache hits return the parsed block directly — zero deserialization overhead. Uses NewBlockFromReader so serializedBlock stays nil, avoiding double memory (raw bytes in struct + raw bytes on disk). BlockInsert parses before caching. BlockByHash returns from cache on hit, parses-then-caches on miss. Cold 56ms, warm 18ms on InscriptionByID.
windBlock writes 'w' prefix entries alongside 'i' and 'n' for each inscription reveal. The 'w' key is 7 bytes (prefix + block_height + seq), different from OrdinalKey[45] — a len check distinguishes them. The value stores the commit outpoint (txid + vout) needed by the future background populator to compute sat numbers via traceSat. unwindBlock deletes 'w' entries alongside 'i' and 'n'. New DB types: OrdinalWorkKey[7], OrdinalWorkValue. New DB method: BlockOrdinalWorkUpdate for the ordinal LevelDB. No backward walks, no sat computation, no 'a' prefix writes. IBD speed unchanged at ~34s for testnet4 (135K blocks, 253K inscriptions).
windBlock branches on watermark state: - Below watermark (IBD): fast path writes 'w' work queue with commit outpoint + inscription ID (72 bytes). 34s testnet4. - Above watermark: backward walk via computeInscribedSat, writes 'i' with real sat, 'a' (sat->inscription), no 'w'. Watermark set once during IBD when block timestamp crosses now - OrdinalWatermarkGap (default 24h). Stored as 'm' prefix OrdinalKey, committed atomically with ordinal data. Only moves down. Sentinel = ordinal genesis height. unwindBlock panics if reorg reaches watermark. Reads 'i' for sat number before deleting, conditionally deletes 'a'. InscriptionsBySat scans 'a' prefix for sat->inscription lookup. InscriptionsByAddress chains UTXOs -> computeSatRanges -> 'a'. Both work for above-watermark inscriptions. Below-watermark inscriptions return sat=0 until populator processes them. OrdinalInscribedSatsInRange changed from 's' to 'a' prefix. hemictl: add ordinalinscriptionsbyaddress command. Populator disabled pending proper tip detection and abort-on-new-block. DB infrastructure present for future use. BlockOrdinalWorkUpdate and OrdinalDataWrite bypass startTransaction -- must fix before enabling populator. Tests: fork test verifies wind/unwind across 3 forks with real sat numbers and 'a' entries. E2E RPC test verifies all 6 ordinal RPCs including InscriptionsBySat and InscriptionsByAddress.
windBlockRanges computes FIFO sat ranges for every output in blocks above the watermark. Writes 'r' (outpoint -> sat ranges) to the ordinal cache. For inputs from previous blocks, reads 'r' from DB. For inputs below watermark, falls back to computeSatRanges. Tracks 's' (inscribed sat -> current outpoint) above watermark: - New inscriptions: 's' set after FIFO resolution via memo map - Sat movement: spent outpoints checked for inscribed sats via OrdinalInscribedSatsInRange, 's' updated to new outpoint - Unwind: deletes 'r' for all outputs, restores 's' to spent outpoint for inscribed sats Populator writes 's' for historical inscriptions: checks canonical zone first (DB lookup), only writes if not already tracked. SatRangesByOutpoint and InscriptionsByAddress check DB 'r' first, fall back to backward walk if not found. Also: pass computeSatRanges into ordinal indexer, use runCtx instead of context.Background in populator, clean up OrdinalInscription API struct.
Introduce the 'o' prefix mapping an inscribed sat's location
(outpoint + byte offset within the output) to its inscription ID.
Key: 'o' + txid(32) + vout(4) + offset(8) = 45 bytes. Value: the
36-byte inscription ID. Prefix-scannable by outpoint via
OrdinalInscriptionsByOutpoint, returning inscriptions in offset
order. This is the direct index for ownership queries
(InscriptionsByAddress): UTXOs map to inscriptions with no walk.
windBlock computes the inscribed sat's landing location from input
and output amounts (placeInOutputs, FIFO, no backward walk) and
writes 'o' in both the above- and below-watermark paths. Ownership
tracking does not depend on the sat number, so it runs at IBD speed.
This commit covers reveals only: the inscribed sat lands in one of
the reveal tx's own outputs. Transfers (spending a tracked outpoint)
and fee→coinbase flow are handled in following commits. A reveal
whose sat lands beyond all outputs (fee) is not yet tracked.
unwindBlock recomputes each reveal's landing location the same way
and deletes the 'o' entry.
The populator no longer writes sat→outpoint; it is now solely the
background sat-number ('a') filler. The dead sat→outpoint and
sat-range writes were removed from wind/unwind/populator along with
windBlockRanges and satToOutpoint. The 's' and 'r' key helpers and
their DB read methods remain temporarily, pending a dedicated
removal once all consumers are cut over.
Fork test asserts 'o' is present at the reveal outpoint across three
competing forks and absent after each unwind.
windBlock gathers all inscribed sats in flight per tx (reveals plus transfers), sorts by offset, and places them FIFO into outputs. A transfer is detected when an input spends an outpoint the 'o' tracker (DB or in-block cache) records as holding an inscribed sat; the sat's FIFO position carries to the new output and its 'o' entry moves. The 'o' value carries the source needed to reverse a transfer: inscID(36) + srcInputIdx(2) + srcOffset(8) = 46 bytes. srcInputIdx is the index, within the spending tx, of the input that consumed the sat; a reveal stores the 0xFFFF sentinel. The source outpoint is not stored — unwind recovers it for free from the block via tx.TxIn[srcInputIdx].PreviousOutPoint. unwindBlock is therefore fully self-contained: it scans this tx's outputs (prefix scan of the 'o' index, keyed), deletes each entry, and for a transfer restores the entry at the spent outpoint with its stored source offset. It reads only the ordinal index and the block in hand — no input amounts, no tx/utxo index. This removes the cross-indexer ordering dependency: ordinals runs in a fixed last slot in both SyncIndexersToHash and syncIndexersToBest. windBlock still reads tx output amounts for forward FIFO placement, which is why ordinals winds last (after the tx index is current); unwind has no such dependency. OrdinalInscriptionsByOutpointWithOffset returns each entry's raw value so the tbc layer can decode the source; the value-length guard accepts the 46-byte form. OrdinalInscriptionsByOutpoint delegates to it. trackedInBlock resolves create-and-spend within one block from the pending cache. The work-queue value is reduced to the inscription ID alone (reveal txid + input index), all the background sat-number populator needs. Fork test exercises the b2->b3 transfer (revealed in b2, spent forward in b3, including a same-block create-and-spend) and verifies the inscription tracks to its new outpoint with correct removal on unwind.
… D1) Widen the ordinal outpoint tracker (o) value from 46 to 53 bytes: inscID(36) + srcKind(1) + srcTxIdx(4) + srcInputIdx(4) + srcOffset(8). Add srcKind discriminator (REVEAL/TRANSFER/FEE/LOST) and block-relative srcTxIdx to support fee-to-coinbase and lost-sat tracking in Phase D2. Add p (predecessor) prefix to store the raw o value that existed at the source outpoint before a transfer moved the sat. On unwind, the predecessor value is restored at the source instead of copying the destination value -- fixing a multi-hop unwind corruption bug where chained transfers would carry the wrong source info backward. Wind: captures t.Value before deleting source o, writes p at destination for transfers, leaves prior p entries untouched. Unwind: reads p before tombstoning, restores correct predecessor at source, deletes both o and p at destination. DB: add OrdinalValueByKey for generic point-get of p entries.
Add feeList accumulation in windBlock: when an inscribed sat FIFO position exceeds the tx output total, the sat is carried to the coinbase phase with its block-wide fee pool offset (blockFeeBase + feeInternal). Coinbase phase places fee sats into coinbase outputs (srcKind=FEE) or the lost sentinel (srcKind=LOST) when the fee pool position exceeds the coinbase output value (miner did not claim full reward). Lost sentinel: all-zero txid, vout 0xFFFFFFFF. Lost sat offsets encode the block height for per-block identification during unwind. Unwind: remove coinbase skip from unwindOutpointTracker. Extract unwindOutpointEntries helper that handles all srcKind values. FEE/LOST entries derive the source tx from block.Transactions()[srcTxIdx]. Lost sentinel entries are scanned and filtered by block height.
Add per-reveal timing for synchronous sat computation via computeInscSat. Logs block height, txid, input index, and duration for reveals exceeding 500ms. Matches the threshold established in ordinalindex_design.go. The watermark/populator decision requires a production sync run to collect data. This instrumentation enables that measurement.
Wire InscriptionsByAddress to the 'o' outpoint tracker: scan 'o'
entries at each UTXO outpoint instead of computing sat ranges via
the dead 'r' prefix or backward walk + 'a' range scan.
Remove dead 'r' (sat ranges) and 's' (sat→outpoint) prefix code:
- Delete OrdinalSatRangesByOutpoint, OrdinalOutpointBySat from
the database interface and LevelDB implementation
- Delete ordinalRangeKey, ordinalSatKey key constructors
- Delete satAtOutputOffset (unused FIFO engine relic)
- Delete IsRange(), IsSat() OrdinalKey methods
- Remove dead 'r' lookup from SatRangesByOutpoint (now backward
walk only)
InscriptionsBySat was already wired to the 'a' prefix in Phase C.
The parsed-block cache (commit 7678759) caused OOM during IBD on memory-constrained hosts. Any code path that touches a cached *btcutil.Block can trigger internal serialization (hashing, witness access) which caches raw bytes inside btcutil objects, doubling memory uncontrollably. The LRU size function cannot account for this. Revert to caching []byte (raw wire format) with exact len(v) costing, which is what origin/main uses and what has always worked.
…try (Phase G) Replace map[OrdinalKey]OrdinalValue with map[Outpoint]*OrdinalCacheEntry so cache lookups during wind/unwind are O(1) instead of O(cache_size). The 45-byte OrdinalKey format encoded the outpoint + offset as a flat byte array, preventing Go map lookup by outpoint alone. trackedInBlock and locatedAtOutpoint compensated by scanning the entire cache linearly for every tx input — pprof confirmed 96% CPU in map iteration. OrdinalCacheEntry groups inscriptions, predecessors, and auxiliary data (i/n/a) under a single outpoint key. The DB layer (BlockOrdinalUpdate) unpacks entries into LevelDB batch operations, constructing o/p keys from Outpoint + offset — same pattern as BlockUtxoUpdate. Key changes: - Eliminate trackedInBlock: replaced by cache[spentOP].Inscriptions O(1) - Fix locatedAtOutpoint: O(1) cache overlay instead of linear scan - OrdinalCache.Len() running counter via PutInscription/PutPredecessor/PutAux - Watermark carried via aux on a sentinel outpoint (vout 0xFFFFFFFE) - predecessorValue takes (op, offset) instead of full OrdinalKey - Aux entries always hosted on reveal tx vout 0 (wind/unwind symmetric) - Delete dead ordinalOutpointKey (DB layer constructs keys inline) - On-disk format unchanged; all existing DB data remains compatible
…nal indexer Add a per-txid LRU cache (TBC_ORDINAL_OUTPUT_CACHE_SIZE, default 256mb) that caches all output values when resolving a tx for the first time. Subsequent inputs spending different vouts from the same parent tx hit the cache instead of deserializing the full block again. Cache miss path uses BlockRawByHash + lazyBlock.FindTx + TxOutputValues for zero-copy per-tx access — no btcutil.NewBlockFromBytes, no full block deserialization, no GC pressure from discarded MsgTx objects. inputOutputValue is now a method on ordinalIndexer instead of a free function, giving it access to the cache. Wire cache stats (hits, misses, purges, size, items) to prometheus using the same pattern as the utxo read cache.
Update all ordinal callers for the new BlockHashByTxId signature that returns (*chainhash.Hash, wire.TxLoc, error). inputOutputValue now uses TxLoc for O(1) tx lookup on cache miss: when TxLoc is available, deserializes only the target tx from its byte offset in the raw block. Falls back to lazyBlock.FindTx for legacy entries without TxLoc. DB version bumped from 6 to 7 for ordinals index. Ordinals upgrade test renamed to TestDbUpgradeV7 and adjusted for v6→v7 transition.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an ordinal indexer to TBC that tracks individual satoshi ownership via FIFO sat range redistribution and indexes Bitcoin inscriptions (including cursed, reinscriptions, parent-child, and delegation).
Depends on #1052 (tx byte location) and #1051 (lazy block reader).
What's in this PR
Ordinal indexer core
Cache and performance (Phases G+)
map[OrdinalKey]OrdinalValuetomap[Outpoint]*OrdinalCacheEntryfor O(1) lookupsTBC_ORDINAL_OUTPUT_CACHE_SIZE, default 256 MB) caching parent tx output valuesBlockRawByHash+lazyBlockfor zero-copy per-tx accessTxLocfrom tx index for O(1) byte offset jumps (no SHA256 scanning)RPCs
InscriptionByID— working, verified against 2500+ inscriptionsInscriptionsBySat— stub (Phase H)InscriptionsByAddress— stub (Phase H)Testing
Files changed
Key files (non-exhaustive):
service/tbc/ordinalindex.go— core indexer, wind/unwind, cacheservice/tbc/ordinalquery.go— RPC query handlersservice/tbc/ordinalparser.go— inscription envelope parserservice/tbc/ordinalcache.go— cache typesservice/tbc/ordinalindex_test.go— testsservice/tbc/tbcfork_test.go— fork/reorg testsdatabase/tbcd/level/ordinal.go— LevelDB ordinal operationsdatabase/tbcd/level/ordinal_test.go— DB testscmd/tbcd/tbcd.go— new config flagsRelated