KAFKA-19193 Support rack-aware partitioning for Kafka producer by ivanyu · Pull Request #19850 · apache/kafka

ivanyu · 2025-05-29T17:02:10Z

According to KIP-1123, this commit adds the support for rack-aware partitioning to BuiltInPartitioner. It comes with two new configs for the producer: partitioner.rack.aware and client.rack, which allows enabling the new behavior.

Apart from the added unit tests, the desired behavior was tested by kafka-producer-perf-test.sh with an existing and a non-existing rack against a 4 node cluster with two racks and 12-partition topic:

./kafka_2.13-4.1.0-SNAPSHOT/bin/kafka-producer-perf-test.sh \
  --topic test-topic --num-records 100000 --throughput -1 --record-size 1 \
  --producer-props bootstrap.servers=127.0.0.1:9092 \
  client.rack=rack0 partitioner.rack.aware=true

Delete this text and replace it with a detailed description of your change. The
PR title and body will become the squashed commit message.

If you would like to tag individuals, add some commentary, upload images, or
include other supplemental information that should not be part of the eventual
commit message, please use a separate comment.

If applicable, please include a summary of the testing strategy (including
rationale) for the proposed change. Unit and/or integration tests are expected
for any behavior change and system tests should be considered for larger
changes.

github-actions · 2025-06-06T03:28:00Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

marcospassos · 2025-08-21T18:56:01Z

Is there any chance this feature will be included in the next release?

ivanyu · 2025-08-22T14:24:50Z

@marcospassos I'm waiting for 4.1 release to be cut, after that I'll ping the mailing list for review.

github-actions · 2025-08-23T03:21:42Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-09-18T03:17:50Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

LiamClarkeNZ

Love the tests.

LiamClarkeNZ · 2025-10-04T04:30:32Z

clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java

+    private final String rack;

-    private volatile PartitionLoadStats partitionLoadStats = null;
+    private volatile PartitionLoadStatsHolder partitionLoadStats = null;


Suggested change

private volatile PartitionLoadStatsHolder partitionLoadStats = null;

private volatile PartitionLoadStatsHolder partitionLoadStatsHolder = null;

As it got a bit confusing when later the code is retrieving partitionLoadStats.inThisRack which is a member of PartitionLoadStatsHolder of type PartitionLoadStats. :D

LiamClarkeNZ · 2025-10-04T04:37:03Z

clients/src/test/java/org/apache/kafka/clients/producer/internals/BuiltInPartitionerTest.java


        // Create partitions with "sticky" batch size to accommodate 3 records.
-        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3);
+        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");


Suggested change

BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");

final boolean rackAware = false;

final String clientRackId = "";

BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, rackAware, clientRackId);

LiamClarkeNZ · 2025-10-04T04:37:52Z

clients/src/test/java/org/apache/kafka/clients/producer/internals/BuiltInPartitionerTest.java


        // Check that switching works even when there is one partition.
-        BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1);
+        BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");


Suggested change

BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");

BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, rackAware, clientRackId);

Just to make the meaning of false and "" obvious at a glance :)

LiamClarkeNZ · 2025-10-04T07:57:18Z

clients/src/test/java/org/apache/kafka/clients/producer/internals/BuiltInPartitionerTest.java

+                partitionRacks[i] = leader.rack();
+            }
+            allPartitions.add(new PartitionInfo(TOPIC_A, i, leader, NODES, NODES));
            expectedFrequencies[i] = 6 - queueSizes[i];  // 6 is max(queueSizes) + 1


Love the comment :)

github-actions · 2026-01-03T03:48:55Z

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

github-actions · 2026-02-02T04:17:43Z

This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.

SoerenHenning · 2026-02-06T10:30:21Z

Hi, unfortunately this was closed. Are there any plan to continue this effort or anything we could do to support it?

marcospassos · 2026-02-06T12:33:08Z

This issue would definitely benefit the community and should not be closed.

kirktrue · 2026-03-19T19:08:19Z

@ivanyu—are you still able to work on this? Or do you need the community to follow up? LMK. Thanks!

ivanyu · 2026-03-19T19:51:33Z

@kirktrue sure, I will review the PR soon!

ivanyu · 2026-03-28T16:25:33Z

For some reason, I cannot reopen the PR (the button isn't even there). Maybe someone else can?

ivanyu · 2026-03-30T08:45:52Z

Thank you @LiamClarkeNZ, I addressed your comments

According to KIP-1123, this commit adds the support for rack-aware partitioning to `BuiltInPartitioner`. It comes with two new configs for the producer: `partitioner.rack.aware` and `client.rack`, which allows enabling the new behavior. Apart from the added unit tests, the desired behavior was validated by `kafka-producer-perf-test.sh` with an existing and a non-existing rack against a 4 node cluster with two racks and 12-partition topic: ```shell ./kafka_2.13-4.1.0-SNAPSHOT/bin/kafka-producer-perf-test.sh \ --topic test-topic --num-records 100000 --throughput -1 --record-size 1 \ --producer-props bootstrap.servers=127.0.0.1:9092 \ client.rack=rack0 partitioner.rack.aware=true ```

github-actions bot added triage PRs from the community producer clients labels May 29, 2025

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch 11 times, most recently from 1af9189 to 564e60f Compare May 31, 2025 06:14

ivanyu marked this pull request as ready for review May 31, 2025 06:20

github-actions bot added the needs-attention label Jun 6, 2025

zornhsu mentioned this pull request Jun 16, 2025

feat(producer): rack-aware partitioner IBM/sarama#3198

Closed

github-actions bot removed the needs-attention label Aug 22, 2025

github-actions bot added the needs-attention label Aug 23, 2025

showuon self-assigned this Sep 17, 2025

github-actions bot removed the needs-attention label Sep 17, 2025

github-actions bot added the needs-attention label Sep 18, 2025

LiamClarkeNZ approved these changes Oct 4, 2025

View reviewed changes

github-actions bot removed needs-attention triage PRs from the community labels Oct 5, 2025

github-actions bot added the stale Stale PRs label Jan 3, 2026

github-actions bot added the closed-stale PRs that were closed due to inactivity label Feb 2, 2026

github-actions bot closed this Feb 2, 2026

viktorsomogyi reopened this Mar 30, 2026

github-actions bot added the triage PRs from the community label Mar 30, 2026

viktorsomogyi removed stale Stale PRs closed-stale PRs that were closed due to inactivity labels Mar 30, 2026

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 564e60f to 6816a21 Compare March 30, 2026 08:45

viktorsomogyi added ci-approved and removed triage PRs from the community labels Mar 30, 2026

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 6816a21 to 605e049 Compare March 30, 2026 08:59

	private volatile PartitionLoadStatsHolder partitionLoadStats = null;
	private volatile PartitionLoadStatsHolder partitionLoadStatsHolder = null;

-        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");
+        final boolean rackAware = false;
+        final String clientRackId = "";
+        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, rackAware, clientRackId);

	BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");
	BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, rackAware, clientRackId);

Conversation

ivanyu commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 6, 2025

Uh oh!

marcospassos commented Aug 21, 2025

Uh oh!

ivanyu commented Aug 22, 2025

Uh oh!

github-actions bot commented Aug 23, 2025

Uh oh!

github-actions bot commented Sep 18, 2025

Uh oh!

LiamClarkeNZ left a comment

Choose a reason for hiding this comment

Uh oh!

LiamClarkeNZ Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

LiamClarkeNZ Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

LiamClarkeNZ Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

LiamClarkeNZ Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 3, 2026

Uh oh!

github-actions bot commented Feb 2, 2026

Uh oh!

SoerenHenning commented Feb 6, 2026

Uh oh!

marcospassos commented Feb 6, 2026

Uh oh!

kirktrue commented Mar 19, 2026

Uh oh!

ivanyu commented Mar 19, 2026

Uh oh!

ivanyu commented Mar 28, 2026

Uh oh!

ivanyu commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ivanyu commented May 29, 2025 •

edited

Loading