KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms for group configs by majialoong · Pull Request #21855 · apache/kafka

majialoong · 2026-03-23T16:22:31Z

createGroupConfigEntry used the group config name (e.g. consumer.session.timeout.ms) directly
to look up broker synonyms, which could produce incorrect synonym chains.

This change prepends GROUP_PREFIX to derive the correct broker config name
(e.g. group.consumer.session.timeout.ms) for synonym lookup, and validates with
configKeys().containsKey() to fall back to GroupConfig defaults for configs without
broker-level counterparts (e.g. share.auto.offset.reset).

It also reads current broker config values via KafkaConfig.get() instead of startup-frozen
defaults, so that dynamically updated broker configs are correctly reflected.

… for group configs

majialoong · 2026-03-23T16:27:35Z

Below is the output of kafka-configs.sh --describe for a consumer group after the fix:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --describe --entity-name my-group --all                                                                                                                
All configs for group my-group are:
  consumer.assignment.interval.ms=1555 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1555, DYNAMIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1333, STATIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1111, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
  consumer.assignor.offload.enable=true sensitive=false synonyms={DEFAULT_CONFIG:group.consumer.assignor.offload.enable=true}
  consumer.heartbeat.interval.ms=5555 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.heartbeat.interval.ms=5555, DEFAULT_CONFIG:group.consumer.heartbeat.interval.ms=5000}
  consumer.session.timeout.ms=46000 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.session.timeout.ms=46000, STATIC_BROKER_CONFIG:group.consumer.session.timeout.ms=47000, DEFAULT_CONFIG:group.consumer.session.timeout.ms=45000}
  share.assignment.interval.ms=1999 sensitive=false synonyms={STATIC_BROKER_CONFIG:group.share.assignment.interval.ms=1999, DEFAULT_CONFIG:group.share.assignment.interval.ms=1000}
  share.assignor.offload.enable=true sensitive=false synonyms={DEFAULT_CONFIG:group.share.assignor.offload.enable=true}
  share.auto.offset.reset=earliest sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:share.auto.offset.reset=earliest, DEFAULT_CONFIG:share.auto.offset.reset=latest}
  share.delivery.count.limit=5 sensitive=false synonyms={DEFAULT_CONFIG:group.share.delivery.count.limit=5}
  share.heartbeat.interval.ms=5000 sensitive=false synonyms={DEFAULT_CONFIG:group.share.heartbeat.interval.ms=5000}
  share.isolation.level=read_uncommitted sensitive=false synonyms={DEFAULT_CONFIG:share.isolation.level=read_uncommitted}
  share.partition.max.record.locks=2000 sensitive=false synonyms={DEFAULT_CONFIG:group.share.partition.max.record.locks=2000}
  share.record.lock.duration.ms=30000 sensitive=false synonyms={DEFAULT_CONFIG:group.share.record.lock.duration.ms=30000}
  share.renew.acknowledge.enable=true sensitive=false synonyms={DEFAULT_CONFIG:share.renew.acknowledge.enable=true}
  share.session.timeout.ms=45000 sensitive=false synonyms={DEFAULT_CONFIG:group.share.session.timeout.ms=45000}
  streams.assignment.interval.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.assignment.interval.ms=1000}
  streams.assignor.offload.enable=true sensitive=false synonyms={DEFAULT_CONFIG:group.streams.assignor.offload.enable=true}
  streams.heartbeat.interval.ms=5000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.heartbeat.interval.ms=5000}
  streams.initial.rebalance.delay.ms=3000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.initial.rebalance.delay.ms=3000}
  streams.num.standby.replicas=0 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.num.standby.replicas=0}
  streams.session.timeout.ms=45000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.session.timeout.ms=45000}
  streams.task.offset.interval.ms=60000 sensitive=false synonyms={DEFAULT_CONFIG:group.streams.task.offset.interval.ms=60000}

dajac · 2026-03-23T18:15:56Z

@majialoong Thanks for the patch! Could you please attach the output before the patch too?

chia7712 · 2026-03-24T01:34:59Z

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java

        STREAMS_GROUP_ASSIGNOR_OFFLOAD_ENABLE_CONFIG
    );
+
+    public static final String GROUP_PREFIX = "group.";


I'm a bit concerned about using an implicit string prefix to connect to server-level configurations. It can be error-prone and less flexible. Could we follow the pattern of ALL_TOPIC_CONFIG_SYNONYMS and introduce something like ALL_GROUP_CONFIG_SYNONYMS

Thanks for the review. Both approaches are valid.

I chose the prefix approach because Group configs have a uniform naming convention — every broker config name is "group." + name, with no exceptions. Given the current convention, future group configs would likely follow the same naming rule as well, so new configs wouldn't require maintaining a separate mapping table.

Topic needs ALL_TOPIC_CONFIG_SYNONYMS because its naming is irregular — different prefixes (log., log.cleaner.), different names (flush.messages → log.flush.interval.messages), and unit conversions. Group configs don't have this complexity.

I'm also open to introducing ALL_GROUP_CONFIG_SYNONYMS if we think the explicitness of a mapping table outweighs the cost of maintaining it for each new group config.

I chose the prefix approach because Group configs have a uniform naming convention — every broker config name is "group." + name, with no exceptions. Given the current convention, future group configs would likely follow the same naming rule as well, so new configs wouldn't require maintaining a separate mapping table.

My primary reservation is that predicating this logic on the group. prefix introduces a tacit constraint that is highly susceptible to oversight by future developers. For example, in #21801 (comment), there was an attempt to add "group.streams.max.num.warmup.replicas" and "streams.max.warmup.replicas", which directly contravenes the exact mapping convention assumed in this PR.

I'm amenable to adopting the group. prefix approach, but would you mind adding a unit test to codify this consistency? we could leverage reflection to cross-reference all configs between GroupConfig and GroupCoordinatorConfig, coorborating that the mapping always holds.

Thanks for the suggestion. I've added testGroupConfigBrokerSynonymNamingConvention in GroupCoordinatorConfigTest.

It iterates all GroupConfig entries and verifies each has a corresponding broker config named GROUP_PREFIX + groupConfigName. Entries without broker-level synonyms are maintained in an explicit exclusion list, and a reverse check ensures the exclusion list stays accurate.

Any deviation from the naming convention will fail CI with a clear message.

I am also a bit on the fence here. I think that having an explicit mapping is better and less error prone. We could have ALL_GROUP_CONFIG_SYNONYMS, as suggested by @chia7712, in GroupConfigs. I would also add a test to ensure that all configs have an entry in the Map to ensure that future configs are considered.

I’ll introduce an explicit mapping for this and add a test to ensure all group configs are covered.

majialoong · 2026-03-24T14:22:38Z

@majialoong Thanks for the patch! Could you please attach the output before the patch too?

@dajac Thanks for the review. Here are the before/after comparisons for kafka-configs.sh --describe across different scenarios, using consumer.assignment.interval.ms as the example config:

Invalid broker param in server.properties

Adding consumer.assignment.interval.ms=1111 (missing group. prefix) to server.properties.

Before:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:consumer.assignment.interval.ms=1000, STATIC_BROKER_CONFIG:consumer.assignment.interval.ms=1111}
  ...

After:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
  ...

The invalid config in server.properties no longer produces a spurious STATIC_BROKER_CONFIG synonym.

Valid broker param in server.properties

Adding group.consumer.assignment.interval.ms=1222 to server.properties.

Before:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:consumer.assignment.interval.ms=1000}
  ...

After:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1222 sensitive=false synonyms={STATIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1222, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
  ...

The broker override is now correctly reflected in both the displayed value and the synonym chain.

Group-level override only

No server.properties changes. Setting group config via:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type groups --entity-name test-group --add-config consumer.assignment.interval.ms=1333

Before:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe -all
All configs for group test-group are:
  consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333}
  ...

After:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
  ...

The DEFAULT_CONFIG synonym with the correct broker config name is now present.

Valid broker param + group-level override

Adding group.consumer.assignment.interval.ms=1222 to server.properties,

then setting group config via:
bin/kafka-configs.sh --alter --entity-type groups --entity-name test-group --add-config consumer.assignment.interval.ms=1333

Before:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333}
  ...

After:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333, STATIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1222, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
  ...

The full synonym chain DYNAMIC_GROUP_CONFIG → STATIC_BROKER_CONFIG → DEFAULT_CONFIG is now displayed correctly.

Valid broker param + group-level override + dynamic broker config

Adding group.consumer.assignment.interval.ms=1222 to server.properties,

then:
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type brokers --entity-default --add-config group.consumer.assignment.interval.ms=1444

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type groups --entity-name test-group --add-config consumer.assignment.interval.ms=1333

Before:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333}
  ...

After:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  consumer.assignment.interval.ms=1333 sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:consumer.assignment.interval.ms=1333, DYNAMIC_DEFAULT_BROKER_CONFIG:group.consumer.assignment.interval.ms=1444, STATIC_BROKER_CONFIG:group.consumer.assignment.interval.ms=1222, DEFAULT_CONFIG:group.consumer.assignment.interval.ms=1000}
  ...

The complete synonym chain DYNAMIC_GROUP_CONFIG → DYNAMIC_DEFAULT_BROKER_CONFIG → STATIC_BROKER_CONFIG → DEFAULT_CONFIG is now displayed correctly.

Config without broker synonym (share.auto.offset.reset)

Setting group config via:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type groups --entity-name test-group --add-config share.auto.offset.reset=latest

Before:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  ...
  share.auto.offset.reset=latest sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:share.auto.offset.reset=latest}
  ...

After:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups --entity-name test-group --describe --all
All configs for group test-group are:
  ...
  share.auto.offset.reset=latest sensitive=false synonyms={DYNAMIC_GROUP_CONFIG:share.auto.offset.reset=latest, DEFAULT_CONFIG:share.auto.offset.reset=latest}
  ...

Configs without a broker-level synonym correctly fall back to GroupConfig defaults.

dajac · 2026-03-27T14:03:48Z

@majialoong Thanks for all the cases! Do we have test cases for each of them?

dajac · 2026-03-27T14:05:49Z

core/src/main/scala/kafka/server/ConfigHelper.scala

    val allSynonyms = {
-      val list = configSynonyms(name, allNames, isSensitive)
+      val brokerConfigName = GroupCoordinatorConfig.GROUP_PREFIX + name
+      val list = if (KafkaConfig.configDef.configKeys().containsKey(brokerConfigName))


nit: Should we just remove configDef() and make CONFIG public?

Just to confirm, is it GroupConfig.configDef() you'd like removed (making CONFIG public)?

dajac · 2026-03-27T14:06:15Z

core/src/main/scala/kafka/server/ConfigHelper.scala

+      else
+        // No broker synonym, fall back to GroupConfig defaults
+        Option(GroupConfig.configDef().defaultValues().get(name))
+          .map(v => List(new DescribeConfigsResponseData.DescribeConfigsSynonym().setName(name)


nit: Should we put .setName on new line too?

dajac · 2026-03-27T14:07:29Z

core/src/main/scala/kafka/server/ConfigHelper.scala

    val allSynonyms = {
-      val list = configSynonyms(name, allNames, isSensitive)
+      val brokerConfigName = GroupCoordinatorConfig.GROUP_PREFIX + name
+      val list = if (KafkaConfig.configDef.configKeys().containsKey(brokerConfigName))


While we are here, would it be possible to not return internal configs? We don't have any at the moment but it would be good to cover it as may have one coming soon.

As for internal configs, createResponseConfig already uses AbstractConfig.nonInternalValues() to filter them out, so future internal group configs would be excluded automatically.

dajac · 2026-03-27T14:08:40Z

core/src/main/scala/kafka/server/ConfigHelper.scala

+  /**
+   * Build the current effective group config defaults by reading from KafkaConfig.
+   */
+  private def currentGroupConfigDefaults: java.util.Map[String, Object] = {


I wonder if we should rather place this one in KafkaConfig. What do you think?

Makes sense. I'll move it to KafkaConfig and name it extractGroupConfigMap to be consistent with extractLogConfigMap.

majialoong · 2026-03-27T18:15:50Z

@majialoong Thanks for all the cases! Do we have test cases for each of them?

@dajac The current testDescribeConfigsGroupSynonyms covers some scenarios. I'll add more test cases for the remaining ones and split them into separate methods so each test focuses on a single synonym chain pattern.

dajac · 2026-03-27T20:13:48Z

@majialoong Thanks for all the cases! Do we have test cases for each of them?

@dajac The current testDescribeConfigsGroupSynonyms covers some scenarios. I'll add more test cases for the remaining ones and split them into separate methods so each test focuses on a single synonym chain pattern.

Sounds good. I wonder whether we could add integration tests too. What do you think?

dajac · 2026-03-29T16:13:40Z

group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupConfigTest.java

+            GroupConfig.SHARE_AUTO_OFFSET_RESET_CONFIG,
+            GroupConfig.SHARE_ISOLATION_LEVEL_CONFIG,
+            GroupConfig.SHARE_RENEW_ACKNOWLEDGE_ENABLE_CONFIG
+        );


I would suggest a different approach. Should we make ALL_GROUP_CONFIG_SYNONYMS a Map<String, Optional<String>> and ensure that all configs have an entry in the map? Those three could use Optional.empty() to signal that they have no synonyms.

Agreed, I'll update it.

dajac · 2026-03-29T16:18:54Z

core/src/test/scala/unit/kafka/server/KafkaApisTest.scala

  }

+  @Test
+  def testDescribeConfigsGroupSynonyms(): Unit = {


Do we cover all the cases from your comment? It may be worth splitting it into smaller test cases.

We could also add them as integration tests. What do you think?

Makes sense. I'll split it into smaller test cases for each scenario and add corresponding integration tests as well.

chia7712 · 2026-03-30T03:05:39Z

core/src/main/scala/kafka/server/KafkaConfig.scala

    logProps
  }
+
+  def extractGroupConfigMap: java.util.Map[String, Object] = {


Could you please move this to AbstractKafkaConfig? We are trying to phase out KafkaConfig, so it would be better to avoid adding new code to it

chia7712 · 2026-03-30T03:57:24Z

core/src/main/scala/kafka/server/ConfigHelper.scala

+        configSynonyms(brokerConfigName, brokerSynonyms(brokerConfigName), isSensitive)
+      else
+        // No broker synonym, fall back to GroupConfig defaults
+        Option(GroupConfig.CONFIG_DEF.defaultValues().get(name))


Should we apply this fallback to topic?

KAFKA-20289: Fix DescribeConfigs to correctly resolve broker synonyms…

001916d

… for group configs

github-actions bot added triage PRs from the community core Kafka Broker group-coordinator labels Mar 23, 2026

chia7712 added the ci-approved label Mar 23, 2026

dajac self-requested a review March 23, 2026 18:10

chia7712 reviewed Mar 24, 2026

View reviewed changes

github-actions bot removed the triage PRs from the community label Mar 24, 2026

Fix test

b39fe33

github-actions bot added the tools label Mar 24, 2026

Add naming convention test for group config broker synonyms

03c9bf2

dajac reviewed Mar 27, 2026

View reviewed changes

majialoong added 2 commits March 28, 2026 00:47

Merge remote-tracking branch 'origin/trunk' into KAFKA-20289

8162ded

Address review comments

e0b1c99

Rename CONFIG to CONFIG_DEF and make public

04dfa51

github-actions bot added the clients label Mar 29, 2026

dajac reviewed Mar 29, 2026

View reviewed changes

chia7712 reviewed Mar 30, 2026

View reviewed changes

Conversation

majialoong commented Mar 23, 2026

Uh oh!

majialoong commented Mar 23, 2026

Uh oh!

dajac commented Mar 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

majialoong commented Mar 24, 2026

Uh oh!

dajac commented Mar 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

majialoong commented Mar 27, 2026

Uh oh!

dajac commented Mar 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants