Skip to content

Add new getcfilters message for utreexo nodes#192

Draft
alainjr10 wants to merge 5 commits into
utreexo:mainfrom
alainjr10:sob-getcfilters
Draft

Add new getcfilters message for utreexo nodes#192
alainjr10 wants to merge 5 commits into
utreexo:mainfrom
alainjr10:sob-getcfilters

Conversation

@alainjr10
Copy link
Copy Markdown

@alainjr10 alainjr10 commented Jun 28, 2024

This is meant to be just a draft PR

Add support for new getcfilters message called UtreexoCFilter
When a new block is added, a new UtreexoCFilter filter is created and persisted on the node.
This filter is created by serializing the contents of the roots at that height.
When a node receives a getcfilters message, the logic doesn't change much, it handles it almost identical to how it deals with basic filters.

@alainjr10 alainjr10 marked this pull request as draft June 28, 2024 15:11
kcalvinalvin
kcalvinalvin reviewed Jul 2, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@kcalvinalvin kcalvinalvin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe creating a new index is a better way of approaching this. With the new index you'd only store the filter header.

Comment thread blockchain/chainio.go
Comment thread blockchain/indexers/cfindex.go
Comment thread wire/msgcfilter.go
Comment thread btcutil/gcs/builder/builder.go
Comment thread blockchain/indexers/utreexoproofindex.go Outdated
@kcalvinalvin
Copy link
Copy Markdown
Contributor

kcalvinalvin commented Jul 18, 2024

Could you make the commits atomic?

Ideally all the changes to each of the packages should be committed together with each commit achieving one single thing.

The change to btcutil/gcs/builder/builder.go could be a single commit. Maybe even the first commit. The change to wire/msgcfilter.go as well.

Then all the changes to the package indexers can be committed. Then the changes to server.go. Then config.go and utreexod.go. Then finally the rpc related files.

@kcalvinalvin
Copy link
Copy Markdown
Contributor

kcalvinalvin commented Jul 19, 2024 

[N] calvin@nixos ~/b/u/utreexod ((296c58f6))> ./utreexod --datadir=. --logdir=. --signet --prune=0 --utreexocfilters
2024-07-19 15:40:47.600 [INF] BTCD: Version 0.3.1-beta
2024-07-19 15:40:47.600 [INF] BTCD: Loading block database from 'signet/blocks_ffldb'
2024-07-19 15:40:47.638 [INF] BTCD: Block database loaded
2024-07-19 15:40:47.640 [INF] INDX: Utreexo C filter index enabled
2024-07-19 15:40:47.641 [INF] INDX: Catching up indexes from height -1 to 0
2024-07-19 15:40:47.641 [INF] BTCD: Gracefully shutting down the database...
2024-07-19 15:40:47.693 [INF] BTCD: Shutdown complete
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6f1d73]
        
goroutine 1 [running]:
github.com/utreexo/utreexod/database/ffldb.rollbackOnPanic(0xc000189450)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/database/ffldb/db.go:2221 +0x56
panic({0x9898e0?, 0xf379b0?})
        /nix/store/zg65r8ys8y5865lcwmmybrq5gn30n1az-go-1.21.6/share/go/src/runtime/panic.go:914 +0x21f
github.com/utreexo/utreexod/blockchain.(*BlockChain).IsUtreexoViewActive(0x0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/utreexoviewpoint.go:849 +0x13
github.com/utreexo/utreexod/blockchain/indexers.(*UtreexoCFIndex).fetchUtreexoRoots(0xc00030a060, {0xbeebf0, 0xc000189450}, 0xc0000bec00?)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/utreexocfindex.go:178 +0x32
github.com/utreexo/utreexod/blockchain/indexers.(*UtreexoCFIndex).ConnectBlock(0xbeebf0?, {0xbeebf0, 0xc000189450}, 0x1a?, {0x1a?, 0x10?, 0xc0000bec00?})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/utreexocfindex.go:155 +0x45
github.com/utreexo/utreexod/blockchain/indexers.dbIndexConnectBlock({0xbeebf0, 0xc000189450}, {0xbed2a0, 0xc00030a060}, 0xc000210480, {0x0, 0x0, 0x0})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:123 +0x11a
github.com/utreexo/utreexod/blockchain/indexers.(*Manager).Init.func6({0xbeebf0?, 0xc000189450?})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:509 +0x3b
github.com/utreexo/utreexod/database/ffldb.(*db).Update(0xc000318000?, 0xc0001d2600)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/database/ffldb/db.go:2278 +0x7e
github.com/utreexo/utreexod/blockchain/indexers.(*Manager).Init(0xc00030a090, 0xbe77a0?, 0xc000318000?)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:508 +0xdf7
github.com/utreexo/utreexod/blockchain.New(0xc0002236e0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/chain.go:2525 +0x8c2
main.newServer({0xc00005dd40, 0x1, 0x1}, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xbec1a0, ...}, ...)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/server.go:3474 +0x1945
main.btcdMain(0x0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/utreexod.go:406 +0xea7
main.main()
        /home/calvin/bitcoin-projects/utreexod1/utreexod/utreexod.go:592 +0x96

The index panics out on start

@alainjr10
Copy link
Copy Markdown
Author

alainjr10 commented Jul 19, 2024 

[N] calvin@nixos ~/b/u/utreexod ((296c58f6))> ./utreexod --datadir=. --logdir=. --signet --prune=0 --utreexocfilters
2024-07-19 15:40:47.600 [INF] BTCD: Version 0.3.1-beta
2024-07-19 15:40:47.600 [INF] BTCD: Loading block database from 'signet/blocks_ffldb'
2024-07-19 15:40:47.638 [INF] BTCD: Block database loaded
2024-07-19 15:40:47.640 [INF] INDX: Utreexo C filter index enabled
2024-07-19 15:40:47.641 [INF] INDX: Catching up indexes from height -1 to 0
2024-07-19 15:40:47.641 [INF] BTCD: Gracefully shutting down the database...
2024-07-19 15:40:47.693 [INF] BTCD: Shutdown complete
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6f1d73]
        
goroutine 1 [running]:
github.com/utreexo/utreexod/database/ffldb.rollbackOnPanic(0xc000189450)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/database/ffldb/db.go:2221 +0x56
panic({0x9898e0?, 0xf379b0?})
        /nix/store/zg65r8ys8y5865lcwmmybrq5gn30n1az-go-1.21.6/share/go/src/runtime/panic.go:914 +0x21f
github.com/utreexo/utreexod/blockchain.(*BlockChain).IsUtreexoViewActive(0x0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/utreexoviewpoint.go:849 +0x13
github.com/utreexo/utreexod/blockchain/indexers.(*UtreexoCFIndex).fetchUtreexoRoots(0xc00030a060, {0xbeebf0, 0xc000189450}, 0xc0000bec00?)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/utreexocfindex.go:178 +0x32
github.com/utreexo/utreexod/blockchain/indexers.(*UtreexoCFIndex).ConnectBlock(0xbeebf0?, {0xbeebf0, 0xc000189450}, 0x1a?, {0x1a?, 0x10?, 0xc0000bec00?})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/utreexocfindex.go:155 +0x45
github.com/utreexo/utreexod/blockchain/indexers.dbIndexConnectBlock({0xbeebf0, 0xc000189450}, {0xbed2a0, 0xc00030a060}, 0xc000210480, {0x0, 0x0, 0x0})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:123 +0x11a
github.com/utreexo/utreexod/blockchain/indexers.(*Manager).Init.func6({0xbeebf0?, 0xc000189450?})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:509 +0x3b
github.com/utreexo/utreexod/database/ffldb.(*db).Update(0xc000318000?, 0xc0001d2600)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/database/ffldb/db.go:2278 +0x7e
github.com/utreexo/utreexod/blockchain/indexers.(*Manager).Init(0xc00030a090, 0xbe77a0?, 0xc000318000?)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:508 +0xdf7
github.com/utreexo/utreexod/blockchain.New(0xc0002236e0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/chain.go:2525 +0x8c2
main.newServer({0xc00005dd40, 0x1, 0x1}, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xbec1a0, ...}, ...)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/server.go:3474 +0x1945
main.btcdMain(0x0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/utreexod.go:406 +0xea7
main.main()
        /home/calvin/bitcoin-projects/utreexod1/utreexod/utreexod.go:592 +0x96

The index panics out on start

Let me look into it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kcalvinalvin kcalvinalvin kcalvinalvin left review comments

@Davidson-Souza Davidson-Souza Davidson-Souza left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alainjr10 @kcalvinalvin @Davidson-Souza