Skip to content

Support coalescing of partitions #718

@yruslan

Description

@yruslan

Background

Currently, Pramen supports repartitioning of parquet files in metastore partitions based on record count or by the explicit number. But repartitioning involves shuffling.

It would be beneficial to add support for coalesce as well because it does not involve shuffling so users can require only 1 files per info date partition, for example.

Feature

Support coalescing of partitions.

Example [Optional]

--

Proposed Solution [Optional]

--

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions