Semantics of Table.Changes

[bbassingthwaite]

Hello, just wanted to ask a clarifying question about the behaviour/stability of Table.Changes. When opening up a new change stream, the current or expected behaviour is that the first set of changes returned from Next will include a snapshot of the entire table? That is, consuming the entire iter is a snapshot of the current entries in the table, and it wouldn't require multiple invocations of Next to receive the entire set of items in the table. We currently rely on this behaviour in order to reconcile our dataplane and just wanted to make sure this isn't a wrong assumption, or that there isn't certain edge cases (size of the table/number of entries). Thanks!

PS: Great Project! ❤️

[joamaki]

Hi! Yes that's right, the first call to Next will return an iterator for all current objects in the entire table:

	// Next returns the sequence of unobserved changes up to the given ReadTxn (snapshot) and
	// a watch channel.

And as it's documented in the API that the semantics are to return full sequence of changes in the given ReadTxn this won't change in the future.

I would maybe suggest looking into RWTable.RegisterInitializer and Table.Initialized. This way you can populate the table asynchronously and signal to the consumers that the table is now initialized fully with the initial set of data. With Changes() you'd call Next(), iterate over the changes and then check table.Initialized() to see if all registered initializers are now done and we've just observed those changes. That's what the StateDB reconciler does to know when the initial data set has been reconciled (https://github.com/cilium/statedb/blob/main/reconciler/reconciler.go#L88).

BTW, as you're one of the first users (that I know of) of StateDB outside the Cilium project, how do you feel about the fact that cilium/statedb has tight coupling to cilium/hive? It's convenient for the Cilium project as cilium/statedb can implement things like script commands directly (https://docs.cilium.io/en/latest/contributing/development/statedb/#script-commands) and reconciler can build on top of the job library in Hive, but it does pull in somewhat large amount of dependencies. I've been somewhat idly thinking about whether to either split the repo or split into multiple modules to minimize the direct dependencies of the core StateDB implementation.

Also in case you've not seen, there's more StateDB documentation in https://docs.cilium.io/en/stable/contributing/development/statedb/ which might be useful. For building reconcilers in general the StateDB reconciler might be a useful building block: https://docs.cilium.io/en/stable/contributing/development/statedb/#reconcilers, https://github.com/cilium/statedb/tree/main/reconciler/example. It does as mentioned above essentially require use of cilium/hive though this can be abstracted out (it's really mostly just the use of Hive job groups to manage background goroutines).

[bbassingthwaite]

I would maybe suggest looking into RWTable.RegisterInitializer and Table.Initialized. This way you can populate the table asynchronously and signal to the consumers that the table is now initialized fully with the initial set of data. With Changes() you'd call Next(), iterate over the changes and then check table.Initialized() to see if all registered initializers are now done and we've just observed those changes. That's what the StateDB reconciler does to know when the initial data set has been reconciled (https://github.com/cilium/statedb/blob/main/reconciler/reconciler.go#L88).

I think I need to give this some more thought. My current use case is we have an Envoy xDS control plane. We store all of the Envoy configs within statedb. When the control plane comes up (before we allow requests), we perform a full sync from our database so we know the table is already consistent.

When Envoy connects to the control plane, it ultimately invokes Changes on the underlying table and we perform a full sync with the first call to Next. This might be the first time Envoy connects or it's reconnecting and it will send us all of the configs it currently has. Either way do a state of the world check and make sure it has all of the right resources (either add, updates or removal of stale resources). It's vital that the first Next returns everything in this use case, otherwise we may erroneously remove resources from Envoy. This is mainly why I wanted to confirm the expected behaviour and I wasn't relying on it working by happenstance.

BTW, as you're one of the first users (that I know of) of StateDB outside the Cilium project, how do you feel about the fact that cilium/statedb has tight coupling to cilium/hive? It's convenient for the Cilium project as cilium/statedb can implement things like script commands directly (https://docs.cilium.io/en/latest/contributing/development/statedb/#script-commands) and reconciler can build on top of the job library in Hive, but it does pull in somewhat large amount of dependencies. I've been somewhat idly thinking about whether to either split the repo or split into multiple modules to minimize the direct dependencies of the core StateDB implementation.

Currently we're not using the reconciler portion nor any of the hive pieces. It's not negatively impacting us though. We may in the future look at adopting the reconciler for some of our eBPF control plane components. (Similarish to what CIlium does) Currently our dataplane nodes store everything to a local pebbleDB. It would be interesting to have some form of persistent with statedb since we prefer to not depend on an external control plane to do the initial boostrap. Have you folks investigated or ever considered doing disk persistence? Only as a mechanism to bootstrap the statedb itself and still keep everything in memory.

Thanks for the reply!

[joamaki]

Currently we're not using the reconciler portion nor any of the hive pieces. It's not negatively impacting us though. We may in the future look at adopting the reconciler for some of our eBPF control plane components. (Similarish to what CIlium does) Currently our dataplane nodes store everything to a local pebbleDB. It would be interesting to have some form of persistent with statedb since we prefer to not depend on an external control plane to do the initial boostrap. Have you folks investigated or ever considered doing disk persistence? Only as a mechanism to bootstrap the statedb itself and still keep everything in memory.

What we are doing in Cilium for persinsting state when we really need to is using a write-ahead log: https://github.com/cilium/cilium/blob/main/pkg/wal/wal.go, https://github.com/cilium/cilium/blob/b4da57fa0ee5ac8076faf2493ec6eeb4273c215d/pkg/datapath/linux/device/reconciler.go#L122. But currently not used to restore tables but more for figuring out what e.g. kernel state we've set up that we might need to clean up after restart. You could use such a mechanism if you make a wrapper around WriteTxn/RWTable to write to disk before committing and then on restart rehydrate the table from the WAL.

The main missing piece for integrating it more tightly into StateDB would be some sort of hooks that run at commit time. The main problem is how to deal with errors when writing to disk. Should error from a hook abort the transaction? And would we need to change Commit() ReadTxn to Commit() (ReadTxn, error)?

EDIT: Actually another way to do this would be to implement a new index (tableIndex) that stores to disk using something like Pebble. Same questions regarding Commit errors applies though.
EDIT2: For fun I tried out how this could look like: #155