Validating assumption on children partition count

[edmondop]

In the query stage, we take the output partition count of the first child if the plan has children, assuming all the plans have the same partition count.

While this is effectively true, if a bug is introduced in planning, this assumption will not hold anymore, and the program might start behaving in surprising ways that are difficult to troubleshoot. Rather than taking the value from the first child, the last child, or a random child, ensuring that if there are children they all have the same output partition count is a more robust approach

[edmondop]

I think this should be merged before #54 @andygrove

[andygrove]

We can do this check without creating additional data structures:

    let partition_count = self.plan.children()[0].output_partitioning().partition_count();
    for i in 1..self.plan.children().len() {
        if self.plan.children()[i].output_partitioning().partition_count() != partition_count {
            return Err(...)
        }
    }

[edmondop]

Makes sense, do you think we should r change the signature to return an error ? That was my original idea, I think there is a 'plan_err!' macro in datafusion we can use

[andygrove]

PR #54 no longer looks at the plan's children in this section of code, so perhaps this check (which is a good check) should move somewhere else. I'd suggest adding it in QueryStage::new and have it return Result<Self>.

[edmondop]

Makes sense, so let's postpone this as an improvement on #54

[edmondop]

I converted it into Draft in the meantime