E2E Test: reproducing epoch 22 issue

[SurfingNerd]

DMDcoin/Diamond#6

[SurfingNerd]

i still could not reproduce the incident-22 with chaos testing....
while i could verify that after the chaos testing, there is a balance missmatch.
here a result of a chaos test:

0x5da44a7bcb386BF637c7391a866Ca8C23A757AFA  =>  18332.745760318785491994
   delegator  0xB4fa2bE7CEB446a839b952416cC562768521a303  stake:  106.122377071901997368
   delegator  0x5da44a7bcb386BF637c7391a866Ca8C23A757AFA  stake:  10000
   delegator  0x53E893C6eb0407B8a252b75af943b2094A195fE0  stake:  106.052216007167126065
   delegator  0xfA1B2E452712Cba0fb64184f8B37278d9B0432ce  stake:  106.052216007167126065
   delegator  0x6e2137e5e1f09e880CDd19ae5B5f6d6d8Bf5A925  stake:  105.911442564743070035
   delegator  0x903105796B6bdC85dC4Ab2BE962f7cA1a6940363  stake:  105.840160899392349279
   delegator  0x83dD5b980a74C09a4725129f06eDa1bDa7a26814  stake:  105.57079668044547068
   delegator  0x3e80f03a458AB64836c4D0360750fcB718d4e5c9  stake:  105.303491698581153149
   delegator  0x158832C7b8c9363459c14550D1244EE0eB148833  stake:  105.303491698581153149
   delegator  0x11A19dDc08a4cAd2E92a97Da80Fc09DBef9Eb552  stake:  105.238097810485882906
   delegator  0xdd2cFbE5bB8C45F5Dbea9E39F8d95aA1Fdd4180e  stake:  105.17031764751570503
   delegator  0x8587a7371fAD1dB26B378D0802FB9372D656FC0a  stake:  105.17031764751570503
   delegator  0x589fE170909d413cB91B81221F1AEDe41361cec7  stake:  105.103856686046092659
   delegator  0xA9adfeC1595d2E5ca6C22F5ad1b56BA2F960fcE5  stake:  104.781966635759146165
   delegator  0x544f36F830Ebd8F0B14cE3e3715d7d654CE529af  stake:  104.719304356922937747
   delegator  0xb03d2FaD2D5AeEd71B32515771E0c08bC84D5781  stake:  104.719304356922937747
   delegator  0xf47eD9c23ad119204eb622B255ec3B42a8Cf3e76  stake:  104.719304356922937747
   delegator  0xaa51c9D3C2483b13cB43CDCaBF796aDece6D90AE  stake:  104.594457288645907307
   delegator  0x88a041c08865213a079086Aee3FE9E5d47292E21  stake:  104.529826598331631866
   delegator  0x01Ba2B71EAcDF235aa29DA359e675c14a8536292  stake:  104.529826598331631866
   delegator  0xF2eA513B6766993A4D7737B6b6561e2F301F89F9  stake:  104.470399523053967753
   delegator  0x719e302a72aB220558Ace85DcEe82dD4de8f612B  stake:  104.235063846174144733
   delegator  0x83eB91C75FA789B8fD7d9764f7d5172C32cD70e9  stake:  104.235063846174144733
   delegator  0x03e23636B7dc3552A1C5Fe5a330f770d12De790B  stake:  104.177172057215778248
   delegator  0x308E056EA3BD73CF22F12586185303B755a8441b  stake:  104.11982964124354226
   delegator  0x9160d196F4C3891054Bd16151c24E18177D20Ae0  stake:  104.003475069642702614
   delegator  0x9bE0d4c2211Df2a373588f874C649C19C97b5Ce9  stake:  104.003475069642702614
   delegator  0x740ec16743e4E7e920Be2DB621f22fA209783498  stake:  104.003475069642702614
   delegator  0x23dc511568667e14042f096691Be32BaB7912852  stake:  103.836969764605434508
   delegator  0x81e314BD84F1fca75d528382BF82b6C7030991E7  stake:  103.836969764605434508
   delegator  0xe6F17e3213c2f93Dc6eb3AcFc10a3cAfEA3fC765  stake:  103.836969764605434508
   delegator  0x6D81c7e89502A5f35E0C2dbAD950FF835a30A814  stake:  103.78277544759962668
   delegator  0xed1C7412F3d566cEA72fD27dAda3925FD2e737C7  stake:  103.729064211738151683
   delegator  0xABDc93d54BdDE40519B6c0cca9865c16425B76F8  stake:  103.620021290011922924
   delegator  0xfB235562F2e0e15955EcfA92F94E6DC061A054eA  stake:  103.514328301819595228
   delegator  0x477bfb45F8b0556AbC8FB0B3C9A12d04b949f630  stake:  103.302354071582282238
   delegator  0x161aeA8f2f4B81388835780aE84098b5aF2e8f15  stake:  103.250375511449416799
   delegator  0x253075725f166ADCC02B71492cB32dd250A11AF5  stake:  103.044240382630335166
   delegator  0x0997D77B27C162b6e312dCf192Dad6bd1466794a  stake:  103.044240382630335166
   delegator  0xd842b3E889DD577a657f4e14149F633e4Ac14bbd  stake:  102.993435361501302394
   delegator  0x379e2619E89b034cbeC56F174af2d2922A970653  stake:  102.94301647801542784
   delegator  0x10b150F9809ff6BFB8D1535Ef672Ac59EdD36Fc1  stake:  102.94301647801542784
   delegator  0x82d6aE19A581A6906EcC4272D9C5BDc82C9a1767  stake:  102.893381139509787936
   delegator  0xD44b64a7452aF4a2EA2348c6137AC373fE8B1403  stake:  102.844116175954747814
   delegator  0xC32F5155724336680621c37F909D3DBDc4AAac1b  stake:  102.695245978050187275
   delegator  0xF846280cdf065b651d788ed82Ab2dBEA5c1Ec46C  stake:  102.549981476456307904
   delegator  0x64D77b668DD5cf1e006e2C2Bb7ff3654e57DA877  stake:  102.453694851716663819
   delegator  0x5151bba470f2A6920EfB85635f810b7FdA8f01B9  stake:  102.405964084664035593
   delegator  0xc460662Ba4f7D2acf38C71a598103da622fd6A9A  stake:  102.358542070056675662
   delegator  0xB61a53b0d7162eb9038B1A02370323c71585396c  stake:  102.358542070056675662
   delegator  0x76cb0b68f789a47B98698455c0e8FEe6EB5FA802  stake:  102.311747358417251826
   delegator  0x5095C2D60463b6D24a790c8Ae950A24B9C508386  stake:  102.265285538234660646
   delegator  0x29D5D57d4e89582002451D7DdfDb326278df4A3c  stake:  102.168676265822281556
   delegator  0x279F112ca249305F0509bB253e292aE6E3Ef4d61  stake:  102.120701887334661879
   delegator  0x1c272dA711899DA2DAC16dFe9Ca01F7C65443458  stake:  102.029831645831182177
   delegator  0x064bEBb14009877793D281b5782e4876DB611cb7  stake:  101.891311778656790618
   delegator  0x297a0cd0DdD77f3dE37fA059c9b228F4bB4Bc961  stake:  101.801872882641521027
   delegator  0xd069279987672C6bD65b3e915E7B62A69e01342D  stake:  101.757388010989827042
   delegator  0xA87D12D9796A83435153F41A3aB4D7359C954777  stake:  101.757388010989827042
   delegator  0xce65f051b82DDA47a6bBdDAc143E23FB1226cec7  stake:  101.757388010989827042
   delegator  0x493aB15e64575438Aa710B37168632a8A5988384  stake:  101.713802802342967746
   delegator  0x9948e64Ee2FF79D9FB76Bc62052AA79612Caa3B6  stake:  101.627227918276321408
   delegator  0xC2d6E06eFdb3DCc8586f1E807C0e2eA6b1c64b0d  stake:  101.584262780294443288
   delegator  0x4597F770137f8ca5BF67D0A0CffEBc870927D485  stake:  101.541608668980112424
   delegator  0xEe666c0f2f03C4ec789CA9499993D7A6d8d186D0  stake:  101.370017365210761104
   delegator  0xeE90C9ddd6B5110adF93E9c0716e09AABFa1a34B  stake:  101.327988288210567855
   delegator  0x4a043f813Bf357FE37622F1BF8EDC6e4df6A267e  stake:  101.118948101502791625
   delegator  0xA20Cda688ba694F1a1A9368Ad407b01c18AF354d  stake:  101.118948101502791625
   delegator  0xf62D065888172Cb1f1aBe80457ecA87006B5B3f5  stake:  101.118948101502791625
   delegator  0x50a84AFDA6067825eA7795A04Be5b6eD3dB95575  stake:  101.034976833011278239
   delegator  0x1e75e24dB9690CD767CA396243E2bada38c9Efa5  stake:  101.034976833011278239
   delegator  0xe4Cdb9E1A2E85960e7c192790b5c7b9483ff2153  stake:  100.994491057889705381
   delegator  0x4757DE21d94A32488173784C2e933d1187E8958A  stake:  100.994491057889705381
   delegator  0x8242BB3862430f71D55D624601369E3Cb5Cd08f0  stake:  100.832721286852353782
   delegator  0xB79190c10Ce637EbDec5D09973748A98437b7821  stake:  100.832721286852353782
   delegator  0xa8d64d9D6d01f411E481766B1477adD7F9E7fFC7  stake:  100.832721286852353782
   delegator  0x4B7e950a7C73b657e89784FdAbe4c5c449063f10  stake:  100.752365248199647979
   delegator  0x966843Ac82C20053b3743ccfcdd16a048fe81A0B  stake:  100.752365248199647979
   delegator  0x6ffc0a603bB82777DDed2FBb27Da705E037a3822  stake:  100.555008289227416028
   delegator  0xf61e3a7ceDA24A4EaE58a32b59dD2d806E7201DB  stake:  100.555008289227416028
   delegator  0x3C51631b3Ee8D0ff2C9fE2027c61FC978c5cc001  stake:  100.516776872964780166
   delegator  0x70cEF68f4DEE2c0365eff10483A0803C978db803  stake:  100.478754243191167877
   delegator  0xeB2103c92B6A4631A84e307118EE18C4C8F9cE9F  stake:  100.440978210755538874
   delegator  0x6B6505E694b0c01911e31409Fe942258c71862B7  stake:  100.440978210755538874
   delegator  0x3d251746Fd0775DC704FAc9642CE66cfac874a0B  stake:  100.366249639474158293
   delegator  0xc169c5C4f6D9FE887Ca161917B4cFa8bf77a40dc  stake:  100.366249639474158293
   delegator  0x41e6D6456D81a36Aa2aDac0b533cf746b5D1b4f8  stake:  100.255518643637715306
   delegator  0x479c9B227C0A6A345fd879c9aD0310D1011F1bc3  stake:  100.182147317196975278
   calced total stake:  18945.520814803839425692
   difference 612775054485053933698

[SurfingNerd]

first time a chaos test reproduced a incident-22 :
Maybe it only happens if certain action are executed during key gen ?

Number of ACKS written changed from 4 to 0
Missing :
 [
  '0xfBE7A182Ac3b6750C7Fa12cD09D78be6480bC50f',
  '0xFC6d1937E100B3e74F23aaf0bf796D60f896b2a0',
  '0x0E312C317AeDE06f4bB24C8190d108A0e82da3Fd',
  '0xC21074fec063d98A7f321d28f52984e9b49B0EF5'
]
withdraw tx: 0xb50d62bdd8c636f5153e052a61dbe093221e3a253ff290b7e68deb91414235ff
processing block: 106 : 2:09:10 PM txs:  0
processing block: 106 : 2:09:10 PM txs:  0
(un)setting Node operator:
choosing pool 1 0x19a1819fE72078F325f4a99D871d9bbd0258bAC0
node operator  0x19a1819fE72078F325f4a99D871d9bbd0258bAC0  for miner  0xC21074fec063d98A7f321d28f52984e9b49B0EF5  current operator  setup operator 0 true
setting rng node operator share  0x19a1819fE72078F325f4a99D871d9bbd0258bAC0  testPool  348
setting rng node operator share setNodeOperator tx: 0xdf13410a15d83149cafa54e44f74038c8e2d3909726505ec5e03753efc60e884
processing block: 107 : 2:09:11 PM txs:  5
processing block: 107 : 2:09:11 PM txs:  5
Number of Parts written changed from 0 to 4.}
Ok: 
 [
  '0xfBE7A182Ac3b6750C7Fa12cD09D78be6480bC50f',
  '0xFC6d1937E100B3e74F23aaf0bf796D60f896b2a0',
  '0x0E312C317AeDE06f4bB24C8190d108A0e82da3Fd',
  '0xC21074fec063d98A7f321d28f52984e9b49B0EF5'
]
Number of Parts written changed from 0 to 4.}
Ok: 
 [
  '0xfBE7A182Ac3b6750C7Fa12cD09D78be6480bC50f',
  '0xFC6d1937E100B3e74F23aaf0bf796D60f896b2a0',
  '0x0E312C317AeDE06f4bB24C8190d108A0e82da3Fd',
  '0xC21074fec063d98A7f321d28f52984e9b49B0EF5'
]
WARN: User  0x19a1819fE72078F325f4a99D871d9bbd0258bAC0  managed to stake on own Node!!
processing block: 108 : 2:09:12 PM txs:  1
processing block: 108 : 2:09:12 PM txs:  1
Number of ACKS written changed from 0 to 1
Ok: 
 [ '0xfBE7A182Ac3b6750C7Fa12cD09D78be6480bC50f' ]
Missing :
 [
  '0xFC6d1937E100B3e74F23aaf0bf796D60f896b2a0',
  '0x0E312C317AeDE06f4bB24C8190d108A0e82da3Fd',
  '0xC21074fec063d98A7f321d28f52984e9b49B0EF5'
]
Number of ACKS written changed from 0 to 1
Ok: 
 [ '0xfBE7A182Ac3b6750C7Fa12cD09D78be6480bC50f' ]
Missing :
 [
  '0xFC6d1937E100B3e74F23aaf0bf796D60f896b2a0',
  '0x0E312C317AeDE06f4bB24C8190d108A0e82da3Fd',
  '0xC21074fec063d98A7f321d28f52984e9b49B0EF5'
]
Interval delegator stake and node operator set:
choosing pool 4 0x35F7Cd0E30Ae099fc1C28683Fd6788317C1d9223
Staking as delegator on pool:  0x35F7Cd0E30Ae099fc1C28683Fd6788317C1d9223
funding delegator account tx: 0xd70b95b5a1f342a315d50a4fa210a6e701002b4273a25ca2fcd1b40e29f7b1ce
processing block: 109 : 2:09:13 PM txs:  4
processing block: 109 : 2:09:13 PM txs:  4
Number of ACKS written changed from 1 to 4
Ok: 
 [
  '0xfBE7A182Ac3b6750C7Fa12cD09D78be6480bC50f',
  '0xFC6d1937E100B3e74F23aaf0bf796D60f896b2a0',
  '0x0E312C317AeDE06f4bB24C8190d108A0e82da3Fd',
  '0xC21074fec063d98A7f321d28f52984e9b49B0EF5'
]
Number of ACKS written changed from 1 to 4
Ok: 
 [
  '0xfBE7A182Ac3b6750C7Fa12cD09D78be6480bC50f',
  '0xFC6d1937E100B3e74F23aaf0bf796D60f896b2a0',
  '0x0E312C317AeDE06f4bB24C8190d108A0e82da3Fd',
  '0xC21074fec063d98A7f321d28f52984e9b49B0EF5'
]
funded:  0x35F7Cd0E30Ae099fc1C28683Fd6788317C1d9223
processing block: 110 : 2:09:14 PM txs:  1
processing block: 110 : 2:09:14 PM txs:  1
Delegator staked 100 DMD on pool  0x35F7Cd0E30Ae099fc1C28683Fd6788317C1d9223  tx: 0x39085e16c84874403ac58afc97f40e3dd0f16dbbeee593b6bef27756d8391235
processing block: 111 : 2:09:15 PM txs:  0

[SurfingNerd]

here we go 🙂
I could reproduce the problem with Chaos tests now,
as well i have the first structured test.
The incident seems to only appear in connection of setOperatorShare action during a Key Gen Round.
More details in Commit: b5cf98c

Example:
main.log

'

[SurfingNerd]

Reproduced in a structured test as well,
the incident occurs if a Validator sets itself as a Node Operator, becomes active validator, and later on removes this setting.