[Just for sharing] Local Spark-driven repro of concurrent-insert snapshot loss

[dushyantk1509]

Mirrors TablesConcurrencyTest.concurrentInsertTest from the LinkedIn acceptance suite, but runs fully in-process against the embedded OpenHouse server via OpenHouseSparkITest. Two futures fire 5 INSERT INTO statements each against a fresh table; each returns HTTP 200, so the client-side committedRecord stays at 10. The final SELECT row count must equal committedRecord — any mismatch means a snapshot was silently dropped by the catalog's subtractive-merge commit path during the race.

Reproduces on first run today (rows=9, expected=10), matching the production data-loss signature observed on WAR (BaseTransaction:466 WARN, "Failed to load metadata for a committed snapshot, skipping clean-up") and the chronic failures of TablesConcurrencyTest on WAR DR.

Summary

Issue] Briefly discuss the summary of the changes made in this
pull request in 2-3 lines.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

[mkuchenbecker]

Why this test isn't useful for confirming a prod bug

The test's plumbing

OpenHouseSparkITest boots SpringH2TestApplication which does NOT scan com.linkedin.openhouse.housetables.* — the real HTS service is absent. In its place,
HouseTablesH2Repository (in tables-test-fixtures, @primary) extends HouseTableRepository directly and delegates to JPA. So houseTableRepository.save() in the test goes:

doCommit → HouseTablesH2Repository.save() → JPA.merge() on H2 → done

Production goes:

doCommit → HouseTableRepositoryImpl.save() → HTTP PUT → HTS service →
UserTablesServiceImpl.putUserTable() → UserTableVersionMapper.toVersion() (string-CAS on metadataLocation) →
UserTableRow JPA.merge() on MySQL (with @Version optimistic lock) → done

Two completely different code paths past houseTableRepository.save(). The test skips the HTS service, the version mapper, the HTTP boundary, the real DB.

What the race in the test actually proves

Two threads both reach HouseTablesH2Repository.save() at the same (tableId, databaseId). JPA merges both rows. No CAS, last writer wins. The first writer's metadata.json is
orphaned. That's the 8-of-10 you see.

This proves: the H2 fixture is missing the CAS that production's HTS service has. It does not prove that the production path has a race — because the production path has
UserTableVersionMapper.toVersion() plus @Version on UserTableRow, both of which are in the call chain only when the real HTS service runs.