Skip to content

flat↔REE SchemaAdapterFactory in beacon-datafusion-ext #280

@robinskil

Description

@robinskil

Part of #276 (Phase 2).

Goal

A boundary-safe SchemaAdapterFactory in beacon-datafusion-ext that coerces a column between flat and run-end-encoded representations so a single scan can absorb mixed-encoding files.

Tasks

  • Custom SchemaAdapterFactory / SchemaAdapter that, when source and target differ only by REE vs flat, routes through arrow::compute::cast (both directions).
  • Integrate with the existing batch-adapter usage in the openers (e.g. beacon-formats/src/bbf/opener.rs uses BatchAdapterFactory).

Acceptance

  • A scan over a mix of REE and flat files emits a single declared encoding with correct data.
  • Round-trip flat→REE→flat is lossless.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestrustPull requests that update rust code

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions