Skip to content

Minor trace chemistry transfer should log row count after dedupe #637

@kbighorse

Description

@kbighorse

Transfer logs show e.g. "Loaded 29376 cleaned" but "Upserting 1000 MinorTraceChemistry records". The only way to get 1000 rows is either (1) deduplication by nma_GlobalID leaving 1000 unique IDs, or (2) many rows skipped in _row_to_dict. There is no logging that distinguishes these.

Requested action: In transfers/minor_trace_chemistry_transfer.py, after building row_dicts and calling _dedupe_rows(rows), log both len(row_dicts) and len(rows) (e.g. "Converted N rows to dicts, M unique by nma_GlobalID"). This makes it clear whether the reduction is from dedupe or from skipped rows and avoids confusion about whether the transfer is truncating data.

Metadata

Metadata

Assignees

Labels

technical debtTechnical debt and cleanup tasks

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions