Fix missing _dlt_loads creation in class-based destination#14
Fix missing _dlt_loads creation in class-based destination#14
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 240637eff5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| expected_schema = schema_to_pyarrow(loads_table.schema()) | ||
| casted_row = cast_table_safe(load_row, expected_schema, strict=False) | ||
| loads_table.append(casted_row) |
There was a problem hiding this comment.
complete_load clears pending files and commits user tables before this append, but _store_completed_load performs a single non-retried metadata write; if a transient catalog/network error happens here, the load is reported as failed even though data is already committed, which can leave recovery in an inconsistent state and may trigger replay behavior on retry. This path should use the same retry/idempotency guarantees as table commits.
Useful? React with 👍 / 👎.
| catalog.create_table( | ||
| identifier=identifier, | ||
| schema=loads_schema, | ||
| partition_spec=PartitionSpec(), | ||
| ) |
There was a problem hiding this comment.
When _dlt_loads is created on first use, the code bypasses _get_table_location and always relies on catalog defaults, unlike normal table creation; with table_location_layout configured, this creates internal metadata in a different path (and can fail in environments where only the configured layout path is writable). The same location-resolution logic used for data tables should be applied here.
Useful? React with 👍 / 👎.
1 participant
Summary
complete_load()by writing a row to_dlt_loadsnamespace._dlt_loadson first use if missing_dlt_loadsexists and stores expected load ids across runsContext
_dlt_loadswas not being materialized for this destination, which breaks dlt internal load tracking.Fixes #13