DLT silently exits on merges

[astronautas]

Absolutely not sure about the root cause, but noticed that dlt silently exits pipeline.run anytime you merge into already existing data source. Rest catalog. First run is fine, subsequent run fails. Append works just fine.

[astronautas]

@nicosuave Any ideas?

[nicosuave]

Do you have any more details so I can reproduce? @astronautas

[astronautas]

Found some context, but not the root cause :/. Try upserting hundreds of thousands of rows on a million rows or smth at the destination, something along those lines. delete-upsert takes ages - profiled it's some loop of building unique key pair-tuples. Upsert (the pyiceberg native one) sort of runs, but then halts and eventually exits (138). All flies on small data though. All did on my local M4.

[nicosuave]

Thanks for the details @astronautas