[v3.5.0] pos sync failed: unexpected bad block at finalized waypoint

[frangrech]

System information

Erigon version: v3.4.0

OS & Version: Ubuntu 24

Commit hash: e2337db9bb924bb5877af475aa53a79f56123013

Erigon Command (with flags/config): --db.size.limit=12TB --prune.mode=archive

Expected behaviour

Node continues syncing normally.

Actual behaviour

Node fails randomly on certain blocks. When node fails, it requires a resync every time. This is one such error:

[WARN] [03-10|13:20:26.693] gas used mismatch                        block=83962765 header=54196343 execution=54093676 txgas="0=46064, 1=63645, 2=59396, 3=360184, 4=83807, 5=21000, 6=107223, 7=59420, 8=282050, 9=46428, 10=153328, 11=201164, 12=96056, 13=136785, 14=376605, 15=325837, 16=348878, 17=543349, 18=309955, 19=330770, 20=107211, 21=349756, 22=107211, 23=928430, 24=120443, 25=107223, 26=46155, 27=32177, 28=88987, 29=132273, 30=54416, 31=168395, 32=347756, 33=45059, 34=40198, 35=167922, 36=35422, 37=267896, 38=54416, 39=93561, 40=21000, 41=692323, 42=39523, 43=21000, 44=29720, 45=54416, 46=117119, 47=32189, 48=163595, 49=344956, 50=68910, 51=46191, 52=36977, 53=349756, 54=267908, 55=298264, 56=125791, 57=122041, 58=225231, 59=236903, 60=531032, 61=236915, 62=236903, 63=1072865, 64=121075, 65=236903, 66=652974, 67=435518, 68=356681, 69=266622, 70=150533, 71=236903, 72=445477, 73=445527, 74=431343, 75=429377, 76=571324, 77=286997, 78=442928, 79=501647, 80=300105, 81=107211, 82=33050, 83=54044, 84=286939, 85=1104732, 86=54056, 87=113686, 88=665485, 89=367093, 90=236903, 91=236903, 92=54056, 93=723521, 94=404442, 95=369796, 96=168407, 97=355070, 98=574583, 99=202908, 100=271202, 101=266632, 102=269828, 103=201454, 104=37260, 105=662047, 106=120431, 107=586380, 108=98367, 109=381889, 110=118862, 111=526012, 112=559700, 113=106188, 114=264657, 115=482253, 116=59000, 117=591808, 118=404660, 119=142977, 120=142989, 121=142989, 122=142989, 123=142989, 124=142977, 125=142977, 126=142977, 127=435456, 128=142965, 129=404464, 130=270708, 131=378624, 132=127455, 133=71519, 134=121075, 135=361742, 136=98801, 137=293017, 138=273106, 139=372671, 140=486765, 141=355693, 142=396839, 143=86043, 144=389874, 145=412328, 146=142965, 147=401316, 148=86043, 149=537209, 150=86212, 151=86043, 152=230271, 153=574753, 154=502582, 155=54044, 156=962791, 157=389480, 158=372381, 159=372381, 160=34308, 161=464135, 162=2546309, 163=274946, 164=372556, 165=367311, 166=275908, 167=387449, 168=367375, 169=414587, 170=275884, 171=372441, 172=372350, 173=367359, 174=275908, 175=285378, 176=59000, 177=86200, 178=387299, 179=367359, 180=86043, 181=54056, 182=541366, 183=136227, 184=62159, 185=62147, 186=47858, 187=47846, 188=251558, 189=110096, 190=28715, 191=37381, 192=22600, 193=42308, 194=59396, 195=134020, 196=134008, 197=117107, 198=34550, 199=45655, 200=21000, 201=55785, 202=168419, 203=60894, 204=21160, 205=21160, 206=21160, 207=21160, 208=335011, 209=21000, 210=277539, 211=155843, 212=31079"
[WARN] [03-10|13:20:26.694] [4/6 Execution] Execution failed         block=83962765 txNum=6818751147 header-hash=0x9b1cdf4822f91f017f47e6fa723c600a9a8385d7c232d9fb967507817a9bc348 err="invalid block, txnIdx=213, gas used by execution: 54093676, in header: 54196343, headerNum=83962765, 9b1cdf4822f91f017f47e6fa723c600a9a8385d7c232d9fb967507817a9bc348" inMem=false
[WARN] [03-10|13:20:26.694] no unwindable block found from changesets, falling back to latest with commitment
[INFO] [03-10|13:20:26.696] [4/6 Execution] Done                     blk=83962764 blks=5 blk/s=0.7 txs=1148 tx/s=159 gas/s=39.83M buf=1.2MB/512.0MB stepsInDB=0.00 step=4364.0 inMem=false alloc=27.9GB sys=42.7GB
[WARN] [03-10|13:20:26.696] Cannot update chain head                 hash=0xa24d7157ddf70b348a1332490863bddb44dceb215e87b5f670a0e3407d0dc73d err="updateForkChoice: invalid block, txnIdx=213, gas used by execution: 54093676, in header: 54196343, headerNum=83962765, 9b1cdf4822f91f017f47e6fa723c600a9a8385d7c232d9fb967507817a9bc348"
[EROR] [03-10|13:20:26.696] failed to update fork choice             latestValidHash=0xa9c28ce2141b56c474f1dc504bee9b01eb1bd7d1a507580d5519d4437a97de1b err="fork choice update bad block: status=1, validationErr='updateForkChoice: invalid block, txnIdx=213, gas used by execution: 54093676, in header: 54196343, headerNum=83962765, 9b1cdf4822f91f017f47e6fa723c600a9a8385d7c232d9fb967507817a9bc348'"
[EROR] [03-10|13:20:26.696] [sync] waypoint execution err            lastCorrectTipNum=83959999 lastCorrectTipHash=0xabe48cec76ee5673bb72c8929808a218d37daead7aa1bd4e3a6d9b4d6e0c7375 execErr="fork choice update bad block: status=1, validationErr='updateForkChoice: invalid block, txnIdx=213, gas used by execution: 54093676, in header: 54196343, headerNum=83962765, 9b1cdf4822f91f017f47e6fa723c600a9a8385d7c232d9fb967507817a9bc348'"
[INFO] [03-10|13:20:26.696] [bridge] unwinding                       blockNum=83959999
[INFO] [03-10|13:20:29.701] [sync] sync service component stopped
[EROR] [03-10|13:20:29.701] [polygon.sync] crashed - stopping node   err="pos sync failed: unexpected bad block at finalized waypoint: fork choice update bad block: status=1, validationErr='updateForkChoice: invalid block, txnIdx=213, gas used by execution: 54093676, in header: 54196343, headerNum=83962765, 9b1cdf4822f91f017f47e6fa723c600a9a8385d7c232d9fb967507817a9bc348'"

Node is stuck in a bootloop afterwards.

Steps to reproduce the behaviour

Nothing out of the ordinary, just sync a normal node. I have also tried clearing polygon-bridge, heimdall and chaindata folders but to no avail.

Is there any way we can solve this error without having to resync? A resync takes well over a month at this point, and even when it gets back in sync, it is prone to this encountering this error. This particular node was freshly resynced after it previously encountered this issue, and now it encountered it again.