diff --git a/docs/apis/_index.md b/docs/apis/_index.md index c73b0ca16d251..eeafa904a2d2a 100644 --- a/docs/apis/_index.md +++ b/docs/apis/_index.md @@ -43,13 +43,14 @@ The Producer API allows applications to send streams of data to topics in the Ka Examples of using the producer are shown in the [javadocs](/{version}/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html "Kafka 4.3 Javadoc"). To use the producer, add the following Maven dependency to your project: - - - - org.apache.kafka - kafka-clients - 4.3.0 - + +```xml + + org.apache.kafka + kafka-clients + 4.3.0 + +``` # Consumer API @@ -58,13 +59,14 @@ The Consumer API allows applications to read streams of data from topics in the Examples of using the consumer are shown in the [javadocs](/{version}/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html "Kafka 4.3 Javadoc"). To use the consumer, add the following Maven dependency to your project: - - - - org.apache.kafka - kafka-clients - 4.3.0 - + +```xml + + org.apache.kafka + kafka-clients + 4.3.0 + +``` # Share Consumer API @@ -73,13 +75,14 @@ The Share Consumer API enables applications in a share group to cooperatively co Examples of using the share consumer are shown in the [javadocs](/{version}/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaShareConsumer.html "Kafka 4.3 Javadoc"). To use the share consumer, add the following Maven dependency to your project: - - - - org.apache.kafka - kafka-clients - 4.3.0 - + +```xml + + org.apache.kafka + kafka-clients + 4.3.0 + +``` # Streams API @@ -90,13 +93,14 @@ Examples of using this library are shown in the [javadocs](/{version}/javadoc/in Additional documentation on using the Streams API is available [here](/43/documentation/streams). To use Kafka Streams, add the following Maven dependency to your project: - - - - org.apache.kafka - kafka-streams - 4.3.0 - + +```xml + + org.apache.kafka + kafka-streams + 4.3.0 + +``` When using Scala you may optionally include the `kafka-streams-scala` library. Additional documentation on using the Kafka Streams DSL for Scala is available [in the developer guide](/43/documentation/streams/developer-guide/dsl-api.html#scala-dsl). @@ -105,12 +109,14 @@ To use Kafka Streams DSL for Scala 2.13, add the following Maven dependency to y > **⚠️ DEPRECATION NOTICE**: The `kafka-streams-scala` library is deprecated as of Kafka 4.3 > and will be removed in Kafka 5.0. Please migrate to using the Java Streams API directly from Scala. > See the [migration guide](/{version}/streams/developer-guide/scala-migration) for details. - - - org.apache.kafka - kafka-streams-scala_2.13 - 4.3.0 - + +```xml + + org.apache.kafka + kafka-streams-scala_2.13 + 4.3.0 + +``` # Connect API @@ -125,12 +131,13 @@ Those who want to implement custom connectors can see the [javadoc](/{version}/j The Admin API supports managing and inspecting topics, brokers, acls, and other Kafka objects. To use the Admin API, add the following Maven dependency to your project: - - - - org.apache.kafka - kafka-clients - 4.3.0 - + +```xml + + org.apache.kafka + kafka-clients + 4.3.0 + +``` For more information about the Admin APIs, see the [javadoc](/{version}/javadoc/index.html?org/apache/kafka/clients/admin/Admin.html "Kafka 4.3 Javadoc"). diff --git a/docs/configuration/broker-configs.md b/docs/configuration/broker-configs.md index 86b85f74ac8b8..bca732b31f835 100644 --- a/docs/configuration/broker-configs.md +++ b/docs/configuration/broker-configs.md @@ -47,34 +47,40 @@ From Kafka version 1.1 onwards, some of the broker configs can be updated withou * `cluster-wide`: May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing. To alter the current broker configs for broker id 0 (for example, the number of log cleaner threads): - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --add-config log.cleaner.threads=2 +``` To describe the current dynamic broker configs for broker id 0: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --describe + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --describe +``` To delete a config override and revert to the statically configured or default value for broker id 0 (for example, the number of log cleaner threads): - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --delete-config log.cleaner.threads -To update the log level for a logger on broker id 0: +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 0 --alter --delete-config log.cleaner.threads +``` +To update the log level for a logger on broker id 0: - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --broker-logger 0 --add-config org.apache.kafka.server.quota.ClientQuotaManager\$ThrottledChannelReaper=DEBUG --alter +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --broker-logger 0 --add-config org.apache.kafka.server.quota.ClientQuotaManager\$ThrottledChannelReaper=DEBUG --alter +``` Some configs may be configured as a cluster-wide default to maintain consistent values across the whole cluster. All brokers in the cluster will process the cluster default update. For example, to update log cleaner threads on all brokers: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=2 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=2 +``` To describe the currently configured dynamic cluster-wide default configs: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe +``` All configs that are configurable at cluster level may also be configured at per-broker level (e.g. for testing). If a config value is defined at different levels, the following order of precedence is used: diff --git a/docs/configuration/configuration-providers.md b/docs/configuration/configuration-providers.md index 5a6eb24e6a7fe..3d02912ca71f6 100644 --- a/docs/configuration/configuration-providers.md +++ b/docs/configuration/configuration-providers.md @@ -45,26 +45,29 @@ To use a configuration provider, specify it in your configuration using the `con Configuration providers allow you to pass parameters and retrieve configuration data from various sources. To specify configuration providers, you use a comma-separated list of aliases and the fully-qualified class names that implement the configuration providers: - - - config.providers=provider1,provider2 - config.providers.provider1.class=com.example.Provider1 - config.providers.provider2.class=com.example.Provider2 + +```properties +config.providers=provider1,provider2 +config.providers.provider1.class=com.example.Provider1 +config.providers.provider2.class=com.example.Provider2 +``` Each provider can have its own set of parameters, which are passed in a specific format: - - - config.providers..param.= + +```properties +config.providers..param.= +``` The `ConfigProvider` interface serves as a base for all configuration providers. Custom implementations of this interface can be created to retrieve configuration data from various sources. You can package the implementation as a JAR file, add the JAR to your classpath, and reference the provider's class in your configuration. **Example custom provider configuration** - - - config.providers=customProvider - config.providers.customProvider.class=com.example.customProvider - config.providers.customProvider.param.param1=value1 - config.providers.customProvider.param.param2=value2 + +```properties +config.providers=customProvider +config.providers.customProvider.class=com.example.customProvider +config.providers.customProvider.param.param1=value1 +config.providers.customProvider.param.param2=value2 +``` ## DirectoryConfigProvider @@ -75,16 +78,18 @@ Each file represents a key, and its content is the value. This provider is usefu To restrict the files that the `DirectoryConfigProvider` can access, use the `allowed.paths` parameter. This parameter accepts a comma-separated list of paths that the provider is allowed to access. If not set, all paths are allowed. **Example`DirectoryConfigProvider` configuration** - - - config.providers=dirProvider - config.providers.dirProvider.class=org.apache.kafka.common.config.provider.DirectoryConfigProvider - config.providers.dirProvider.param.allowed.paths=/path/to/dir1,/path/to/dir2 + +```properties +config.providers=dirProvider +config.providers.dirProvider.class=org.apache.kafka.common.config.provider.DirectoryConfigProvider +config.providers.dirProvider.param.allowed.paths=/path/to/dir1,/path/to/dir2 +``` To reference a value supplied by the `DirectoryConfigProvider`, use the correct placeholder syntax: - - - ${dirProvider::} + +```text +${dirProvider::} +``` ## EnvVarConfigProvider @@ -97,16 +102,18 @@ This provider is useful for configuring applications running in containers, for To restrict which environment variables the `EnvVarConfigProvider` can access, use the `allowlist.pattern` parameter. This parameter accepts a regular expression that environment variable names must match to be used by the provider. **Example`EnvVarConfigProvider` configuration** - - - config.providers=envVarProvider - config.providers.envVarProvider.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider - config.providers.envVarProvider.param.allowlist.pattern=^MY_ENVAR1_.* + +```properties +config.providers=envVarProvider +config.providers.envVarProvider.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider +config.providers.envVarProvider.param.allowlist.pattern=^MY_ENVAR1_.* +``` To reference a value supplied by the `EnvVarConfigProvider`, use the correct placeholder syntax: - - - ${envVarProvider:} + +```text +${envVarProvider:} +``` ## FileConfigProvider @@ -117,41 +124,46 @@ This provider is useful for loading configuration data from mounted files. To restrict the file paths that the `FileConfigProvider` can access, use the `allowed.paths` parameter. This parameter accepts a comma-separated list of paths that the provider is allowed to access. If not set, all paths are allowed. **Example`FileConfigProvider` configuration** - - - config.providers=fileProvider - config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider - config.providers.fileProvider.param.allowed.paths=/path/to/config1,/path/to/config2 + +```properties +config.providers=fileProvider +config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider +config.providers.fileProvider.param.allowed.paths=/path/to/config1,/path/to/config2 +``` To reference a value supplied by the `FileConfigProvider`, use the correct placeholder syntax: - - - ${fileProvider::} + +```text +${fileProvider::} +``` ## Example: Referencing files Here’s an example that uses a file configuration provider with Kafka Connect to provide authentication credentials to a database for a connector. First, create a `connector-credentials.properties` configuration file with the following credentials: - - - dbUsername=my-username - dbPassword=my-password + +```properties +dbUsername=my-username +dbPassword=my-password +``` Specify a `FileConfigProvider` in the Kafka Connect configuration: **Example Kafka Connect configuration with a`FileConfigProvider`** - - - config.providers=fileProvider - config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider + +```properties +config.providers=fileProvider +config.providers.fileProvider.class=org.apache.kafka.common.config.provider.FileConfigProvider +``` Next, reference the properties from the file in the connector configuration. **Example connector configuration referencing file properties** - - - database.user=${fileProvider:/path/to/connector-credentials.properties:dbUsername} - database.password=${fileProvider:/path/to/connector-credentials.properties:dbPassword} + +```properties +database.user=${fileProvider:/path/to/connector-credentials.properties:dbUsername} +database.password=${fileProvider:/path/to/connector-credentials.properties:dbPassword} +``` At runtime, the configuration provider reads and extracts the values from the properties file. diff --git a/docs/configuration/system-properties.md b/docs/configuration/system-properties.md index 2637441ffbfa7..6a82c1410416b 100644 --- a/docs/configuration/system-properties.md +++ b/docs/configuration/system-properties.md @@ -33,8 +33,10 @@ Kafka supports some configuration that can be enabled through Java system proper This system property is used to determine which files, if any, are allowed to be read by the SASL OAUTHBEARER plugin. This property accepts comma-separated list of files. By default the value is an empty list. If users want to enable some files, users need to explicitly set the system property like below. - - -Dorg.apache.kafka.sasl.oauthbearer.allowed.files=/tmp/token,/tmp/private_key.pem + +```bash +-Dorg.apache.kafka.sasl.oauthbearer.allowed.files=/tmp/token,/tmp/private_key.pem +``` @@ -61,8 +63,10 @@ Default Value: This system property is used to set the allowed URLs as SASL OAUTHBEARER token or jwks endpoints. This property accepts comma-separated list of URLs. By default the value is an empty list. If users want to enable some URLs, users need to explicitly set the system property like below. - - -Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=https://www.example.com,file:///tmp/token + +```bash +-Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=https://www.example.com,file:///tmp/token +```
@@ -89,12 +93,16 @@ Default Value: This system property is used to disable the problematic login modules usage in SASL JAAS configuration. This property accepts comma-separated list of loginModule names. By default **com.sun.security.auth.module.JndiLoginModule** and **com.sun.security.auth.module.LdapLoginModule** loginModule is disabled. If users want to enable JndiLoginModule or LdapLoginModule, users need to explicitly reset the system property like below. We advise the users to validate configurations and only allow trusted JNDI configurations. For more details [CVE-2023-25194](/community/cve-list/#CVE-2023-25194). - - -Dorg.apache.kafka.disallowed.login.modules= + +```bash +-Dorg.apache.kafka.disallowed.login.modules= +``` To disable more loginModules, update the system property with comma-separated loginModule names. Make sure to explicitly add **JndiLoginModule** module name to the comma-separated list like below. - - -Dorg.apache.kafka.disallowed.login.modules=com.sun.security.auth.module.JndiLoginModule,com.ibm.security.auth.module.LdapLoginModule,com.ibm.security.auth.module.Krb5LoginModule + +```bash +-Dorg.apache.kafka.disallowed.login.modules=com.sun.security.auth.module.JndiLoginModule,com.ibm.security.auth.module.LdapLoginModule,com.ibm.security.auth.module.Krb5LoginModule +``` The configuration is deprecated and will be removed in a future release. Please use **org.apache.kafka.allowed.login.modules** instead.
@@ -153,16 +161,22 @@ Default Value: This system property controls the automatic loading of ConfigProvider implementations in Apache Kafka. ConfigProviders are used to dynamically supply configuration values from sources such as files, directories, or environment variables. This property accepts a comma-separated list of ConfigProvider names. By default, all built-in ConfigProviders are enabled, including **FileConfigProvider** , **DirectoryConfigProvider** , and **EnvVarConfigProvider**. If users want to disable all automatic ConfigProviders, they need to explicitly set the system property as shown below. Disabling automatic ConfigProviders is recommended in environments where configuration data comes from untrusted sources or where increased security is required. For more details, see [CVE-2024-31141](/community/cve-list/#CVE-2024-31141). - - -Dorg.apache.kafka.automatic.config.providers=none + +```bash +-Dorg.apache.kafka.automatic.config.providers=none +``` To allow specific ConfigProviders, update the system property with a comma-separated list of fully qualified ConfigProvider class names. For example, to enable only the **EnvVarConfigProvider** , set the property as follows: - - -Dorg.apache.kafka.automatic.config.providers=org.apache.kafka.common.config.provider.EnvVarConfigProvider + +```bash +-Dorg.apache.kafka.automatic.config.providers=org.apache.kafka.common.config.provider.EnvVarConfigProvider +``` To use multiple ConfigProviders, include their names in a comma-separated list as shown below: - - -Dorg.apache.kafka.automatic.config.providers=org.apache.kafka.common.config.provider.FileConfigProvider,org.apache.kafka.common.config.provider.EnvVarConfigProvider + +```bash +-Dorg.apache.kafka.automatic.config.providers=org.apache.kafka.common.config.provider.FileConfigProvider,org.apache.kafka.common.config.provider.EnvVarConfigProvider +```
diff --git a/docs/configuration/tiered-storage-configs.md b/docs/configuration/tiered-storage-configs.md index ac6041b660be5..a6d5d38bb2f0e 100644 --- a/docs/configuration/tiered-storage-configs.md +++ b/docs/configuration/tiered-storage-configs.md @@ -39,16 +39,17 @@ All configurations here should start with the prefix defined by `remote.log.meta The implementation of `TopicBasedRemoteLogMetadataManager` needs to create admin, producer, and consumer clients for the internal topic `__remote_log_metadata`. Additional configurations can be provided for different types of clients using the following configuration properties: - - - # Configs for admin, producer, and consumer clients - .remote.log.metadata.common.client. = - - # Configs only for admin client - .remote.log.metadata.admin. = - - # Configs only for producer client - .remote.log.metadata.producer. = - - # Configs only for consumer client - .remote.log.metadata.consumer. = + +```properties +# Configs for admin, producer, and consumer clients +.remote.log.metadata.common.client. = + +# Configs only for admin client +.remote.log.metadata.admin. = + +# Configs only for producer client +.remote.log.metadata.producer. = + +# Configs only for consumer client +.remote.log.metadata.consumer. = +``` diff --git a/docs/configuration/topic-configs.md b/docs/configuration/topic-configs.md index d95936c002370..dffba75e8d87f 100644 --- a/docs/configuration/topic-configs.md +++ b/docs/configuration/topic-configs.md @@ -27,26 +27,30 @@ type: docs Configurations pertinent to topics have both a server default as well an optional per-topic override. If no per-topic configuration is given the server default is used. The override can be set at topic creation time by giving one or more `--config` options. This example creates a topic named _my-topic_ with a custom max message size and flush rate: - - - $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 \ - --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1 + +```bash +$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 1 \ + --replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1 +``` Overrides can also be changed or set later using the alter configs command. This example updates the max message size for _my-topic_ : - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic - --alter --add-config max.message.bytes=128000 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic + --alter --add-config max.message.bytes=128000 +``` To check overrides set on the topic you can do - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --describe + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --describe +``` To remove an override you can do - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic - --alter --delete-config max.message.bytes + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic + --alter --delete-config max.message.bytes +``` Below is the topic configuration. The server's default configuration for this property is given under the Server Default Property heading. A given server default config value only applies to a topic if it does not have an explicit topic config override. {{< include-html file="/static/{version}/generated/topic_config.html" >}} diff --git a/docs/design/design.md b/docs/design/design.md index 24d66c198627c..e5eb380826394 100644 --- a/docs/design/design.md +++ b/docs/design/design.md @@ -370,17 +370,18 @@ Log compaction ensures that Kafka will always retain at least the last known val So far we have described only the simpler approach to data retention where old log data is discarded after a fixed period of time or when the log reaches some predetermined size. This works well for temporal event data such as logging where each record stands alone. However an important class of data streams are the log of changes to keyed, mutable data (for example, the changes to a database table). Let's discuss a concrete example of such a stream. Say we have a topic containing user email addresses; every time a user updates their email address we send a message to this topic using their user id as the primary key. Now say we send the following messages over some time period for a user with id 123, each message corresponding to a change in email address (messages for other ids are omitted): - - - 123 => bill@microsoft.com - . - . - . - 123 => bill@gatesfoundation.org - . - . - . - 123 => bill@gmail.com + +```text +123 => bill@microsoft.com + . + . + . +123 => bill@gatesfoundation.org + . + . + . +123 => bill@gmail.com +``` Log compaction gives us a more granular retention mechanism so that we are guaranteed to retain at least the last update for each primary key (e.g. `bill@gmail.com`). By doing this we guarantee that the log contains a full snapshot of the final value for every key not just keys that changed recently. This means downstream consumers can restore their own state off this topic without us having to retain a complete log of all changes. @@ -436,19 +437,22 @@ Log compaction is handled by the log cleaner, a pool of background threads that ### Configuring The Log Cleaner The log cleaner is enabled by default. This will start the pool of cleaner threads. To enable log cleaning on a particular topic, add the log-specific property - - - log.cleanup.policy=compact + +```properties +log.cleanup.policy=compact +``` The `log.cleanup.policy` property is a broker configuration setting defined in the broker's `server.properties` file; it affects all of the topics in the cluster that do not have a configuration override in place as documented [here](/documentation.html#brokerconfigs). The log cleaner can be configured to retain a minimum amount of the uncompacted "head" of the log. This is enabled by setting the compaction time lag. - - - log.cleaner.min.compaction.lag.ms + +```properties +log.cleaner.min.compaction.lag.ms +``` This can be used to prevent messages newer than a minimum message age from being subject to compaction. If not set, all log segments are eligible for compaction except for the last segment, i.e. the one currently being written to. The active segment will not be compacted even if all of its messages are older than the minimum compaction time lag. The log cleaner can be configured to ensure a maximum delay after which the uncompacted "head" of the log becomes eligible for log compaction. - - - log.cleaner.max.compaction.lag.ms + +```properties +log.cleaner.max.compaction.lag.ms +``` This can be used to prevent log with low produce rate from remaining ineligible for compaction for an unbounded duration. If not set, logs that do not exceed min.cleanable.dirty.ratio are not compacted. Note that this compaction deadline is not a hard guarantee since it is still subjected to the availability of log cleaner threads and the actual compaction time. You will want to monitor the uncleanable-partitions-count, max-clean-time-secs and max-compaction-delay-secs metrics. diff --git a/docs/design/protocol.md b/docs/design/protocol.md index 87966feee10f3..8807e5ebbbcf7 100644 --- a/docs/design/protocol.md +++ b/docs/design/protocol.md @@ -145,11 +145,12 @@ The [BNF](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form)s below give an ### Common Request and Response Structure All requests and responses originate from the following grammar which will be incrementally describe through the rest of this document: - - - RequestOrResponse => Size (RequestMessage | ResponseMessage) - Size => int32 - + +```text +RequestOrResponse => Size (RequestMessage | ResponseMessage) + Size => int32 +``` +
@@ -197,10 +198,11 @@ The following are the numeric codes that the stable ApiKey in the request can ta This section gives details on each of the individual API Messages, their usage, their binary format, and the meaning of their fields. The message consists of the header and body: - - - Message => RequestOrResponseHeader Body - + +```text +Message => RequestOrResponseHeader Body +``` + `RequestOrResponseHeader` is the versioned request or response header. `Body` is the message-specific body. diff --git a/docs/implementation/log.md b/docs/implementation/log.md index ef54104964824..e7f6c257906fc 100644 --- a/docs/implementation/log.md +++ b/docs/implementation/log.md @@ -47,24 +47,25 @@ The actual process of reading from an offset requires first locating the log seg The log provides the capability of getting the most recently written message to allow clients to start subscribing as of "right now". This is also useful in the case the consumer fails to consume its data within its SLA-specified number of days. In this case when the client attempts to consume a non-existent offset it is given an OutOfRangeException and can either reset itself or fail as appropriate to the use case. The following is the format of the results sent to the consumer. - - - MessageSetSend (fetch result) - - total length : 4 bytes - error code : 2 bytes - message 1 : x bytes - ... - message n : x bytes - - - MultiMessageSetSend (multiFetch result) - - total length : 4 bytes - error code : 2 bytes - messageSetSend 1 - ... - messageSetSend n + +```text +MessageSetSend (fetch result) + +total length : 4 bytes +error code : 2 bytes +message 1 : x bytes +... +message n : x bytes + + +MultiMessageSetSend (multiFetch result) + +total length : 4 bytes +error code : 2 bytes +messageSetSend 1 +... +messageSetSend n +``` ## Deletes diff --git a/docs/implementation/message-format.md b/docs/implementation/message-format.md index d93cc025b32c5..29078b759c4b7 100644 --- a/docs/implementation/message-format.md +++ b/docs/implementation/message-format.md @@ -31,33 +31,34 @@ Messages (aka Records) are always written in batches. The technical term for a b ## Record Batch The following is the on-disk format of a RecordBatch. - - - baseOffset: int64 - batchLength: int32 - partitionLeaderEpoch: int32 - magic: int8 (current magic value is 2) - crc: uint32 - attributes: int16 - bit 0~2: - 0: no compression - 1: gzip - 2: snappy - 3: lz4 - 4: zstd - bit 3: timestampType - bit 4: isTransactional (0 means not transactional) - bit 5: isControlBatch (0 means not a control batch) - bit 6: hasDeleteHorizonMs (0 means baseTimestamp is not set as the delete horizon for compaction) - bit 7~15: unused - lastOffsetDelta: int32 - baseTimestamp: int64 - maxTimestamp: int64 - producerId: int64 - producerEpoch: int16 - baseSequence: int32 - recordsCount: int32 - records: [Record] + +```text +baseOffset: int64 +batchLength: int32 +partitionLeaderEpoch: int32 +magic: int8 (current magic value is 2) +crc: uint32 +attributes: int16 + bit 0~2: + 0: no compression + 1: gzip + 2: snappy + 3: lz4 + 4: zstd + bit 3: timestampType + bit 4: isTransactional (0 means not transactional) + bit 5: isControlBatch (0 means not a control batch) + bit 6: hasDeleteHorizonMs (0 means baseTimestamp is not set as the delete horizon for compaction) + bit 7~15: unused +lastOffsetDelta: int32 +baseTimestamp: int64 +maxTimestamp: int64 +producerId: int64 +producerEpoch: int16 +baseSequence: int32 +recordsCount: int32 +records: [Record] +``` Note that when compression is enabled, the compressed record data is serialized directly following the count of the number of records. @@ -74,37 +75,40 @@ Compaction may also modify the baseTimestamp if the record batch contains record A control batch contains a single record called the control record. Control records should not be passed on to applications. Instead, they are used by consumers to filter out aborted transactional messages. The key of a control record conforms to the following schema: - - - version: int16 (current version is 0) - type: int16 (0 indicates an abort marker, 1 indicates a commit) + +```text +version: int16 (current version is 0) +type: int16 (0 indicates an abort marker, 1 indicates a commit) +``` The schema for the value of a control record is dependent on the type. The value is opaque to clients. ## Record The on-disk format of each record is delineated below. - - - length: varint - attributes: int8 - bit 0~7: unused - timestampDelta: varlong - offsetDelta: varint - keyLength: varint - key: byte[] - valueLength: varint - value: byte[] - headersCount: varint - Headers => [Header] + +```text +length: varint +attributes: int8 + bit 0~7: unused +timestampDelta: varlong +offsetDelta: varint +keyLength: varint +key: byte[] +valueLength: varint +value: byte[] +headersCount: varint +Headers => [Header] +``` ### Record Header - - - headerKeyLength: varint - headerKey: String - headerValueLength: varint - Value: byte[] + +```text +headerKeyLength: varint +headerKey: String +headerValueLength: varint +Value: byte[] +``` The key of a record header is guaranteed to be non-null, while the value of a record header may be null. The order of headers in a record is preserved when producing and consuming. diff --git a/docs/kafka-connect/administration.md b/docs/kafka-connect/administration.md index 742a531742d40..0c7da8dfec683 100644 --- a/docs/kafka-connect/administration.md +++ b/docs/kafka-connect/administration.md @@ -39,22 +39,23 @@ If a Connect worker leaves the group, intentionally or due to a failure, Connect The new Connect protocol is enabled when all the workers that form the Connect cluster are configured with `connect.protocol=compatible`, which is also the default value when this property is missing. Therefore, upgrading to the new Connect protocol happens automatically when all the workers upgrade to 2.3.0. A rolling upgrade of the Connect cluster will activate incremental cooperative rebalancing when the last worker joins on version 2.3.0. You can use the REST API to view the current status of a connector and its tasks, including the ID of the worker to which each was assigned. For example, the `GET /connectors/file-source/status` request shows the status of a connector named `file-source`: - - - { - "name": "file-source", - "connector": { + +```json +{ + "name": "file-source", + "connector": { + "state": "RUNNING", + "worker_id": "192.168.1.208:8083" + }, + "tasks": [ + { + "id": 0, "state": "RUNNING", - "worker_id": "192.168.1.208:8083" - }, - "tasks": [ - { - "id": 0, - "state": "RUNNING", - "worker_id": "192.168.1.209:8083" - } - ] - } + "worker_id": "192.168.1.209:8083" + } + ] +} +``` Connectors and their tasks publish status updates to a shared topic (configured with `status.storage.topic`) which all workers in the cluster monitor. Because the workers consume this topic asynchronously, there is typically a (short) delay before a state change is visible through the status API. The following states are possible for a connector or one of its tasks: diff --git a/docs/kafka-connect/connector-development-guide.md b/docs/kafka-connect/connector-development-guide.md index d8544666e9620..6b8ab3d910cd7 100644 --- a/docs/kafka-connect/connector-development-guide.md +++ b/docs/kafka-connect/connector-development-guide.md @@ -57,58 +57,63 @@ The rest of this section will walk through some code to demonstrate the key step ### Connector Example We'll cover the `SourceConnector` as a simple example. `SinkConnector` implementations are very similar. Pick a package and class name, these examples will use the `FileStreamSourceConnector` but substitute your own class name where appropriate. In order to make the plugin discoverable at runtime, add a ServiceLoader manifest to your resources in `META-INF/services/org.apache.kafka.connect.source.SourceConnector` with your fully-qualified class name on a single line: - - - com.example.FileStreamSourceConnector + +```text +com.example.FileStreamSourceConnector +``` Create a class that inherits from `SourceConnector` and add a field that will store the configuration information to be propagated to the task(s) (the topic to send data to, and optionally - the filename to read from and the maximum batch size): - - - package com.example; - - public class FileStreamSourceConnector extends SourceConnector { - private Map props; + +```java +package com.example; + +public class FileStreamSourceConnector extends SourceConnector { + private Map props; +``` The easiest method to fill in is `taskClass()`, which defines the class that should be instantiated in worker processes to actually read the data: - - - @Override - public Class taskClass() { - return FileStreamSourceTask.class; - } + +```java +@Override +public Class taskClass() { + return FileStreamSourceTask.class; +} +``` We will define the `FileStreamSourceTask` class below. Next, we add some standard lifecycle methods, `start()` and `stop()`: - - - @Override - public void start(Map props) { - // Initialization logic and setting up of resources can take place in this method. - // This connector doesn't need to do any of that, but we do log a helpful message to the user. - - this.props = props; - AbstractConfig config = new AbstractConfig(CONFIG_DEF, props); - String filename = config.getString(FILE_CONFIG); - filename = (filename == null || filename.isEmpty()) ? "standard input" : config.getString(FILE_CONFIG); - log.info("Starting file source connector reading from {}", filename); - } - - @Override - public void stop() { - // Nothing to do since no background monitoring is required. - } + +```java +@Override +public void start(Map props) { + // Initialization logic and setting up of resources can take place in this method. + // This connector doesn't need to do any of that, but we do log a helpful message to the user. + + this.props = props; + AbstractConfig config = new AbstractConfig(CONFIG_DEF, props); + String filename = config.getString(FILE_CONFIG); + filename = (filename == null || filename.isEmpty()) ? "standard input" : config.getString(FILE_CONFIG); + log.info("Starting file source connector reading from {}", filename); +} + +@Override +public void stop() { + // Nothing to do since no background monitoring is required. +} +``` Finally, the real core of the implementation is in `taskConfigs()`. In this case we are only handling a single file, so even though we may be permitted to generate more tasks as per the `maxTasks` argument, we return a list with only one entry: - - - @Override - public List> taskConfigs(int maxTasks) { - // Note that the task configs could contain configs additional to or different from the connector configs if needed. For instance, - // if different tasks have different responsibilities, or if different tasks are meant to process different subsets of the source data stream). - ArrayList> configs = new ArrayList<>(); - // Only one input stream makes sense. - configs.add(props); - return configs; - } + +```java +@Override +public List> taskConfigs(int maxTasks) { + // Note that the task configs could contain configs additional to or different from the connector configs if needed. For instance, + // if different tasks have different responsibilities, or if different tasks are meant to process different subsets of the source data stream). + ArrayList> configs = new ArrayList<>(); + // Only one input stream makes sense. + configs.add(props); + return configs; +} +``` Even with multiple tasks, this method implementation is usually pretty simple. It just has to determine the number of input tasks, which may require contacting the remote service it is pulling data from, and then divvy them up. Because some patterns for splitting work among tasks are so common, some utilities are provided in `ConnectorUtils` to simplify these cases. @@ -119,57 +124,59 @@ Note that this simple example does not include dynamic input. See the discussion Next we'll describe the implementation of the corresponding `SourceTask`. The implementation is short, but too long to cover completely in this guide. We'll use pseudo-code to describe most of the implementation, but you can refer to the source code for the full example. Just as with the connector, we need to create a class inheriting from the appropriate base `Task` class. It also has some standard lifecycle methods: - - - public class FileStreamSourceTask extends SourceTask { - private String filename; - private InputStream stream; - private String topic; - private int batchSize; - - @Override - public void start(Map props) { - filename = props.get(FileStreamSourceConnector.FILE_CONFIG); - stream = openOrThrowError(filename); - topic = props.get(FileStreamSourceConnector.TOPIC_CONFIG); - batchSize = props.get(FileStreamSourceConnector.TASK_BATCH_SIZE_CONFIG); - } - - @Override - public synchronized void stop() { - stream.close(); - } + +```java +public class FileStreamSourceTask extends SourceTask { + private String filename; + private InputStream stream; + private String topic; + private int batchSize; + + @Override + public void start(Map props) { + filename = props.get(FileStreamSourceConnector.FILE_CONFIG); + stream = openOrThrowError(filename); + topic = props.get(FileStreamSourceConnector.TOPIC_CONFIG); + batchSize = props.get(FileStreamSourceConnector.TASK_BATCH_SIZE_CONFIG); } + @Override + public synchronized void stop() { + stream.close(); + } +} +``` + These are slightly simplified versions, but show that these methods should be relatively simple and the only work they should perform is allocating or freeing resources. There are two points to note about this implementation. First, the `start()` method does not yet handle resuming from a previous offset, which will be addressed in a later section. Second, the `stop()` method is synchronized. This will be necessary because `SourceTasks` are given a dedicated thread which they can block indefinitely, so they need to be stopped with a call from a different thread in the Worker. Next, we implement the main functionality of the task, the `poll()` method which gets events from the input system and returns a `List`: - - - @Override - public List poll() throws InterruptedException { - try { - ArrayList records = new ArrayList<>(); - while (streamValid(stream) && records.isEmpty()) { - LineAndOffset line = readToNextLine(stream); - if (line != null) { - Map sourcePartition = Collections.singletonMap("filename", filename); - Map sourceOffset = Collections.singletonMap("position", streamOffset); - records.add(new SourceRecord(sourcePartition, sourceOffset, topic, Schema.STRING_SCHEMA, line)); - if (records.size() >= batchSize) { - return records; - } - } else { - Thread.sleep(1); + +```java +@Override +public List poll() throws InterruptedException { + try { + ArrayList records = new ArrayList<>(); + while (streamValid(stream) && records.isEmpty()) { + LineAndOffset line = readToNextLine(stream); + if (line != null) { + Map sourcePartition = Collections.singletonMap("filename", filename); + Map sourceOffset = Collections.singletonMap("position", streamOffset); + records.add(new SourceRecord(sourcePartition, sourceOffset, topic, Schema.STRING_SCHEMA, line)); + if (records.size() >= batchSize) { + return records; } + } else { + Thread.sleep(1); } - return records; - } catch (IOException e) { - // Underlying stream was killed, probably as a result of calling stop. Allow to return - // null, and driving thread will handle any shutdown if necessary. } - return null; + return records; + } catch (IOException e) { + // Underlying stream was killed, probably as a result of calling stop. Allow to return + // null, and driving thread will handle any shutdown if necessary. } + return null; +} +``` Again, we've omitted some details, but we can see the important steps: the `poll()` method is going to be called repeatedly, and for each call it will loop trying to read records from the file. For each line it reads, it also tracks the file offset. It uses this information to create an output `SourceRecord` with four pieces of information: the source partition (there is only one, the single file being read), source offset (byte offset in the file), output topic name, and output value (the line, and we include a schema indicating this value will always be a string). Other variants of the `SourceRecord` constructor can also include a specific output partition, a key, and headers. @@ -180,19 +187,20 @@ Although not used in the example, `SourceTask` also provides two APIs to commit ### Sink Tasks The previous section described how to implement a simple `SourceTask`. Unlike `SourceConnector` and `SinkConnector`, `SourceTask` and `SinkTask` have very different interfaces because `SourceTask` uses a pull interface and `SinkTask` uses a push interface. Both share the common lifecycle methods, but the `SinkTask` interface is quite different: - - - public abstract class SinkTask implements Task { - public void initialize(SinkTaskContext context) { - this.context = context; - } - - public abstract void put(Collection records); - - public void flush(Map currentOffsets) { - } + +```java +public abstract class SinkTask implements Task { + public void initialize(SinkTaskContext context) { + this.context = context; } + public abstract void put(Collection records); + + public void flush(Map currentOffsets) { + } +} +``` + The `SinkTask` documentation contains full details, but this interface is nearly as simple as the `SourceTask`. The `put()` method should contain most of the implementation, accepting sets of `SinkRecords`, performing any required translation, and storing them in the destination system. This method does not need to ensure the data has been fully written to the destination system before returning. In fact, in many cases internal buffering will be useful so an entire batch of records can be sent at once, reducing the overhead of inserting events into the downstream data store. The `SinkRecords` contain essentially the same information as `SourceRecords`: Kafka topic, partition, offset, the event key and value, and optional headers. The `flush()` method is used during the offset commit process, which allows tasks to recover from failures and resume from a safe point such that no events will be missed. The method should push any outstanding data to the destination system and then block until the write has been acknowledged. The `offsets` parameter can often be ignored, but is useful in some cases where implementations want to store offset information in the destination store to provide exactly-once delivery. For example, an HDFS connector could do this and use atomic move operations to make sure the `flush()` operation atomically commits the data and offsets to a final location in HDFS. @@ -200,53 +208,55 @@ The `flush()` method is used during the offset commit process, which allows task ### Errant Record Reporter When error reporting is enabled for a connector, the connector can use an `ErrantRecordReporter` to report problems with individual records sent to a sink connector. The following example shows how a connector's `SinkTask` subclass might obtain and use the `ErrantRecordReporter`, safely handling a null reporter when the DLQ is not enabled or when the connector is installed in an older Connect runtime that doesn't have this reporter feature: - - - private ErrantRecordReporter reporter; - - @Override - public void start(Map props) { - ... - try { - reporter = context.errantRecordReporter(); // may be null if DLQ not enabled - } catch (NoSuchMethodException | NoClassDefFoundError e) { - // Will occur in Connect runtimes earlier than 2.6 - reporter = null; - } + +```java +private ErrantRecordReporter reporter; + +@Override +public void start(Map props) { + ... + try { + reporter = context.errantRecordReporter(); // may be null if DLQ not enabled + } catch (NoSuchMethodException | NoClassDefFoundError e) { + // Will occur in Connect runtimes earlier than 2.6 + reporter = null; } - - @Override - public void put(Collection records) { - for (SinkRecord record: records) { - try { - // attempt to process and send record to data sink - process(record); - } catch(Exception e) { - if (reporter != null) { - // Send errant record to error reporter - reporter.report(record, e); - } else { - // There's no error reporter, so fail - throw new ConnectException("Failed on record", e); - } +} + +@Override +public void put(Collection records) { + for (SinkRecord record: records) { + try { + // attempt to process and send record to data sink + process(record); + } catch(Exception e) { + if (reporter != null) { + // Send errant record to error reporter + reporter.report(record, e); + } else { + // There's no error reporter, so fail + throw new ConnectException("Failed on record", e); } } } +} +``` ### Resuming from Previous Offsets The `SourceTask` implementation included a stream ID (the input filename) and offset (position in the file) with each record. The framework uses this to commit offsets periodically so that in the case of a failure, the task can recover and minimize the number of events that are reprocessed and possibly duplicated (or to resume from the most recent offset if Kafka Connect was stopped gracefully, e.g. in standalone mode or due to a job reconfiguration). This commit process is completely automated by the framework, but only the connector knows how to seek back to the right position in the input stream to resume from that location. To correctly resume upon startup, the task can use the `SourceContext` passed into its `initialize()` method to access the offset data. In `initialize()`, we would add a bit more code to read the offset (if it exists) and seek to that position: - - - stream = new FileInputStream(filename); - Map offset = context.offsetStorageReader().offset(Collections.singletonMap(FILENAME_FIELD, filename)); - if (offset != null) { - Long lastRecordedOffset = (Long) offset.get("position"); - if (lastRecordedOffset != null) - seekToOffset(stream, lastRecordedOffset); - } + +```java +stream = new FileInputStream(filename); +Map offset = context.offsetStorageReader().offset(Collections.singletonMap(FILENAME_FIELD, filename)); +if (offset != null) { + Long lastRecordedOffset = (Long) offset.get("position"); + if (lastRecordedOffset != null) + seekToOffset(stream, lastRecordedOffset); +} +``` Of course, you might need to read many keys for each of the input streams. The `OffsetStorageReader` interface also allows you to issue bulk reads to efficiently load all offsets, then apply them by seeking each input stream to the appropriate position. @@ -263,51 +273,53 @@ By default, the Kafka Connect framework will create and commit a new Kafka trans If enabled, the connector's tasks will have access to a `TransactionContext` from their `SourceTaskContext`, which they can use to control when transactions are aborted and committed. For example, to commit a transaction at least every ten records: - - - private int recordsSent; - - @Override - public void start(Map props) { - this.recordsSent = 0; - } - - @Override - public List poll() { - List records = fetchRecords(); - boolean shouldCommit = false; - for (SourceRecord record : records) { - if (++this.recordsSent >= 10) { - shouldCommit = true; - } - } - if (shouldCommit) { - this.recordsSent = 0; - this.context.transactionContext().commitTransaction(); + +```java +private int recordsSent; + +@Override +public void start(Map props) { + this.recordsSent = 0; +} + +@Override +public List poll() { + List records = fetchRecords(); + boolean shouldCommit = false; + for (SourceRecord record : records) { + if (++this.recordsSent >= 10) { + shouldCommit = true; } - return records; } - -Or to commit a transaction for exactly every tenth record: - - - private int recordsSent; - - @Override - public void start(Map props) { + if (shouldCommit) { this.recordsSent = 0; + this.context.transactionContext().commitTransaction(); } - - @Override - public List poll() { - List records = fetchRecords(); - for (SourceRecord record : records) { - if (++this.recordsSent % 10 == 0) { - this.context.transactionContext().commitTransaction(record); - } + return records; +} +``` + +Or to commit a transaction for exactly every tenth record: + +```java +private int recordsSent; + +@Override +public void start(Map props) { + this.recordsSent = 0; +} + +@Override +public List poll() { + List records = fetchRecords(); + for (SourceRecord record : records) { + if (++this.recordsSent % 10 == 0) { + this.context.transactionContext().commitTransaction(record); } - return records; } + return records; +} +``` Most connectors do not need to define their own transaction boundaries. However, it may be useful if files or objects in the source system are broken up into multiple source records, but should be delivered atomically. Additionally, it may be useful if it is impossible to give each source record a unique source offset, if every record with a given offset is delivered within a single transaction. @@ -320,32 +332,36 @@ A few additional preflight validation APIs can be implemented by source connecto Some users may require exactly-once semantics from a connector. In this case, they may set the `exactly.once.support` property to `required` in the configuration for the connector. When this happens, the Kafka Connect framework will ask the connector whether it can provide exactly-once semantics with the specified configuration. This is done by invoking the `exactlyOnceSupport` method on the connector. If a connector doesn't support exactly-once semantics, it should still implement this method to let users know for certain that it cannot provide exactly-once semantics: - - - @Override - public ExactlyOnceSupport exactlyOnceSupport(Map props) { - // This connector cannot provide exactly-once semantics under any conditions - return ExactlyOnceSupport.UNSUPPORTED; - } - + +```java +@Override +public ExactlyOnceSupport exactlyOnceSupport(Map props) { + // This connector cannot provide exactly-once semantics under any conditions + return ExactlyOnceSupport.UNSUPPORTED; +} +``` + + Otherwise, a connector should examine the configuration, and return `ExactlyOnceSupport.SUPPORTED` if it can provide exactly-once semantics: - - - @Override - public ExactlyOnceSupport exactlyOnceSupport(Map props) { - // This connector can always provide exactly-once semantics - return ExactlyOnceSupport.SUPPORTED; - } + +```java +@Override +public ExactlyOnceSupport exactlyOnceSupport(Map props) { + // This connector can always provide exactly-once semantics + return ExactlyOnceSupport.SUPPORTED; +} +``` Additionally, if the user has configured the connector to define its own transaction boundaries, the Kafka Connect framework will ask the connector whether it can define its own transaction boundaries with the specified configuration, using the `canDefineTransactionBoundaries` method: - - - @Override - public ConnectorTransactionBoundaries canDefineTransactionBoundaries(Map props) { - // This connector can always define its own transaction boundaries - return ConnectorTransactionBoundaries.SUPPORTED; - } + +```java +@Override +public ConnectorTransactionBoundaries canDefineTransactionBoundaries(Map props) { + // This connector can always define its own transaction boundaries + return ConnectorTransactionBoundaries.SUPPORTED; +} +``` This method should only be implemented for connectors that can define their own transaction boundaries in some cases. If a connector is never able to define its own transaction boundaries, it does not need to implement this method. @@ -354,10 +370,11 @@ This method should only be implemented for connectors that can define their own Kafka Connect is intended to define bulk data copying jobs, such as copying an entire database rather than creating many jobs to copy each table individually. One consequence of this design is that the set of input or output streams for a connector can vary over time. Source connectors need to monitor the source system for changes, e.g. table additions/deletions in a database. When they pick up changes, they should notify the framework via the `ConnectorContext` object that reconfiguration is necessary. For example, in a `SourceConnector`: - - - if (inputsChanged()) - this.context.requestTaskReconfiguration(); + +```java +if (inputsChanged()) + this.context.requestTaskReconfiguration(); +``` The framework will promptly request new configuration information and update the tasks, allowing them to gracefully commit their progress before reconfiguring them. Note that in the `SourceConnector` this monitoring is currently left up to the connector implementation. If an extra thread is required to perform this monitoring, the connector must allocate it itself. @@ -370,17 +387,18 @@ Ideally this code for monitoring changes would be isolated to the `Connector` an Kafka Connect allows you to validate connector configurations before submitting a connector to be executed and can provide feedback about errors and recommended values. To take advantage of this, connector developers need to provide an implementation of `config()` to expose the configuration definition to the framework. The following code in `FileStreamSourceConnector` defines the configuration and exposes it to the framework. - - - static final ConfigDef CONFIG_DEF = new ConfigDef() - .define(FILE_CONFIG, Type.STRING, null, Importance.HIGH, "Source filename. If not specified, the standard input will be used") - .define(TOPIC_CONFIG, Type.STRING, ConfigDef.NO_DEFAULT_VALUE, new ConfigDef.NonEmptyString(), Importance.HIGH, "The topic to publish data to") - .define(TASK_BATCH_SIZE_CONFIG, Type.INT, DEFAULT_TASK_BATCH_SIZE, Importance.LOW, - "The maximum number of records the source task can read from the file each time it is polled"); - - public ConfigDef config() { - return CONFIG_DEF; - } + +```java +static final ConfigDef CONFIG_DEF = new ConfigDef() + .define(FILE_CONFIG, Type.STRING, null, Importance.HIGH, "Source filename. If not specified, the standard input will be used") + .define(TOPIC_CONFIG, Type.STRING, ConfigDef.NO_DEFAULT_VALUE, new ConfigDef.NonEmptyString(), Importance.HIGH, "The topic to publish data to") + .define(TASK_BATCH_SIZE_CONFIG, Type.INT, DEFAULT_TASK_BATCH_SIZE, Importance.LOW, + "The maximum number of records the source task can read from the file each time it is polled"); + +public ConfigDef config() { + return CONFIG_DEF; +} +``` `ConfigDef` class is used for specifying the set of expected configurations. For each configuration, you can specify the name, the type, the default value, the documentation, the group information, the order in the group, the width of the configuration value and the name suitable for display in the UI. Plus, you can provide special validation logic used for single configuration validation by overriding the `Validator` class. Moreover, as there may be dependencies between configurations, for example, the valid values and visibility of a configuration may change according to the values of other configurations. To handle this, `ConfigDef` allows you to specify the dependents of a configuration and to provide an implementation of `Recommender` to get valid values and set visibility of a configuration given the current configuration values. @@ -393,17 +411,18 @@ The FileStream connectors are good examples because they are simple, but they al To create more complex data, you'll need to work with the Kafka Connect `data` API. Most structured records will need to interact with two classes in addition to primitive types: `Schema` and `Struct`. The API documentation provides a complete reference, but here is a simple example creating a `Schema` and `Struct`: - - - Schema schema = SchemaBuilder.struct().name(NAME) - .field("name", Schema.STRING_SCHEMA) - .field("age", Schema.INT_SCHEMA) - .field("admin", SchemaBuilder.bool().defaultValue(false).build()) - .build(); - - Struct struct = new Struct(schema) - .put("name", "Barbara Liskov") - .put("age", 75); + +```java +Schema schema = SchemaBuilder.struct().name(NAME) + .field("name", Schema.STRING_SCHEMA) + .field("age", Schema.INT_SCHEMA) + .field("admin", SchemaBuilder.bool().defaultValue(false).build()) + .build(); + +Struct struct = new Struct(schema) + .put("name", "Barbara Liskov") + .put("age", 75); +``` If you are implementing a source connector, you'll need to decide when and how to create schemas. Where possible, you should avoid recomputing them as much as possible. For example, if your connector is guaranteed to have a fixed schema, create it statically and reuse a single instance. diff --git a/docs/kafka-connect/user-guide.md b/docs/kafka-connect/user-guide.md index d859a0ba6e4df..924f727fda67d 100644 --- a/docs/kafka-connect/user-guide.md +++ b/docs/kafka-connect/user-guide.md @@ -33,9 +33,10 @@ The [quickstart](../getting-started/quickstart) provides a brief example of how Kafka Connect currently supports two modes of execution: standalone (single process) and distributed. In standalone mode all work is performed in a single process. This configuration is simpler to setup and get started with and may be useful in situations where only one worker makes sense (e.g. collecting log files), but it does not benefit from some of the features of Kafka Connect such as fault tolerance. You can start a standalone process with the following command: - - - $ bin/connect-standalone.sh config/connect-standalone.properties [connector1.properties connector2.json …] + +```bash +$ bin/connect-standalone.sh config/connect-standalone.properties [connector1.properties connector2.json …] +``` The first parameter is the configuration for the worker. This includes settings such as the Kafka connection parameters, serialization format, and how frequently to commit offsets. The provided example should work well with a local cluster running with the default configuration provided by `config/server.properties`. It will require tweaking to use with a different configuration or production deployment. All workers (both standalone and distributed) require a few configs: @@ -59,9 +60,10 @@ Client configuration overrides can be configured individually per connector by u The remaining parameters are connector configuration files. Each file may either be a Java Properties file or a JSON file containing an object with the same structure as the request body of either the `POST /connectors` endpoint or the `PUT /connectors/{name}/config` endpoint (see the [OpenAPI documentation](/43/generated/connect_rest.yaml)). You may include as many as you want, but all will execute within the same process (on different threads). You can also choose not to specify any connector configuration files on the command line, and instead use the REST API to create connectors at runtime after your standalone worker starts. Distributed mode handles automatic balancing of work, allows you to scale up (or down) dynamically, and offers fault tolerance both in the active tasks and for configuration and offset commit data. Execution is very similar to standalone mode: - - - $ bin/connect-distributed.sh config/connect-distributed.properties + +```bash +$ bin/connect-distributed.sh config/connect-distributed.properties +``` The difference is in the class which is started and the configuration parameters which change how the Kafka Connect process decides where to store configurations, how to assign work, and where to store offsets and task statuses. In the distributed mode, Kafka Connect stores the offsets, configs and task statuses in Kafka topics. It is recommended to manually create the topics for offset, configs and statuses to achieve the desired number of partitions and replication factors. If the topics are not yet created when starting Kafka Connect, the topics will be auto created with default number of partitions and replication factor, which may not be best suited for its usage. @@ -116,10 +118,11 @@ A transformation chain can be specified in the connector configuration. For example, lets take the built-in file source connector and use a transformation to add a static field. Throughout the example we'll use schemaless JSON data format. To use schemaless format, we changed the following two lines in `connect-standalone.properties` from true to false: - - - key.converter.schemas.enable - value.converter.schemas.enable + +```properties +key.converter.schemas.enable +value.converter.schemas.enable +``` The file source connector reads each line as a String. We will wrap each line in a Map and then add a second field to identify the origin of the event. To do this, we use two transformations: @@ -129,35 +132,38 @@ The file source connector reads each line as a String. We will wrap each line in After adding the transformations, `connect-file-source.properties` file looks as following: - - - name=local-file-source - connector.class=FileStreamSource - tasks.max=1 - file=test.txt - topic=connect-test - transforms=MakeMap, InsertSource - transforms.MakeMap.type=org.apache.kafka.connect.transforms.HoistField$Value - transforms.MakeMap.field=line - transforms.InsertSource.type=org.apache.kafka.connect.transforms.InsertField$Value - transforms.InsertSource.static.field=data_source - transforms.InsertSource.static.value=test-file-source + +```properties +name=local-file-source +connector.class=FileStreamSource +tasks.max=1 +file=test.txt +topic=connect-test +transforms=MakeMap, InsertSource +transforms.MakeMap.type=org.apache.kafka.connect.transforms.HoistField$Value +transforms.MakeMap.field=line +transforms.InsertSource.type=org.apache.kafka.connect.transforms.InsertField$Value +transforms.InsertSource.static.field=data_source +transforms.InsertSource.static.value=test-file-source +``` All the lines starting with `transforms` were added for the transformations. You can see the two transformations we created: "InsertSource" and "MakeMap" are aliases that we chose to give the transformations. The transformation types are based on the list of built-in transformations you can see below. Each transformation type has additional configuration: HoistField requires a configuration called "field", which is the name of the field in the map that will include the original String from the file. InsertField transformation lets us specify the field name and the value that we are adding. When we ran the file source connector on my sample file without the transformations, and then read them using `kafka-console-consumer.sh`, the results were: - - - "foo" - "bar" - "hello world" + +```text +"foo" +"bar" +"hello world" +``` We then create a new file connector, this time after adding the transformations to the configuration file. This time, the results will be: - - - {"line":"foo","data_source":"test-file-source"} - {"line":"bar","data_source":"test-file-source"} - {"line":"hello world","data_source":"test-file-source"} + +```json +{"line":"foo","data_source":"test-file-source"} +{"line":"bar","data_source":"test-file-source"} +{"line":"hello world","data_source":"test-file-source"} +``` You can see that the lines we've read are now part of a JSON map, and there is an extra field with the static value we specified. This is just one example of what you can do with transformations. @@ -210,34 +216,36 @@ For example, suppose you have a source connector which produces messages to many To do this we need first to filter out the records destined for the topic 'foo'. The Filter transformation removes records from further processing, and can use the TopicNameMatches predicate to apply the transformation only to records in topics which match a certain regular expression. TopicNameMatches's only configuration property is `pattern` which is a Java regular expression for matching against the topic name. The configuration would look like this: - - - transforms=Filter - transforms.Filter.type=org.apache.kafka.connect.transforms.Filter - transforms.Filter.predicate=IsFoo - - predicates=IsFoo - predicates.IsFoo.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches - predicates.IsFoo.pattern=foo + +```properties +transforms=Filter +transforms.Filter.type=org.apache.kafka.connect.transforms.Filter +transforms.Filter.predicate=IsFoo + +predicates=IsFoo +predicates.IsFoo.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches +predicates.IsFoo.pattern=foo +``` Next we need to apply ExtractField only when the topic name of the record is not 'bar'. We can't just use TopicNameMatches directly, because that would apply the transformation to matching topic names, not topic names which do _not_ match. The transformation's implicit `negate` config properties allows us to invert the set of records which a predicate matches. Adding the configuration for this to the previous example we arrive at: - - - transforms=Filter,Extract - transforms.Filter.type=org.apache.kafka.connect.transforms.Filter - transforms.Filter.predicate=IsFoo - - transforms.Extract.type=org.apache.kafka.connect.transforms.ExtractField$Key - transforms.Extract.field=other_field - transforms.Extract.predicate=IsBar - transforms.Extract.negate=true - - predicates=IsFoo,IsBar - predicates.IsFoo.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches - predicates.IsFoo.pattern=foo - - predicates.IsBar.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches - predicates.IsBar.pattern=bar + +```properties +transforms=Filter,Extract +transforms.Filter.type=org.apache.kafka.connect.transforms.Filter +transforms.Filter.predicate=IsFoo + +transforms.Extract.type=org.apache.kafka.connect.transforms.ExtractField$Key +transforms.Extract.field=other_field +transforms.Extract.predicate=IsBar +transforms.Extract.negate=true + +predicates=IsFoo,IsBar +predicates.IsFoo.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches +predicates.IsFoo.pattern=foo + +predicates.IsBar.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches +predicates.IsBar.pattern=bar +``` Kafka Connect includes the following predicates: @@ -254,9 +262,10 @@ Details on how to configure each predicate are listed below: ## REST API Since Kafka Connect is intended to be run as a service, it also provides a REST API for managing connectors. This REST API is available in both standalone and distributed mode. The REST API server can be configured using the `listeners` configuration option. This field should contain a list of listeners in the following format: `protocol://host:port,protocol2://host2:port2`. Currently supported protocols are `http` and `https`. For example: - - - listeners=http://localhost:8080,https://localhost:8443 + +```properties +listeners=http://localhost:8080,https://localhost:8443 +``` By default, if no `listeners` are specified, the REST server runs on port 8083 using the HTTP protocol. When using HTTPS, the configuration has to include the SSL configuration. By default, it will use the `ssl.*` settings. In case it is needed to use different configuration for the REST API than for connecting to Kafka brokers, the fields can be prefixed with `listeners.https`. When using the prefix, only the prefixed options will be used and the `ssl.*` options without the prefix will be ignored. Following fields can be used to configure HTTPS for the REST API: @@ -306,42 +315,46 @@ The following are the currently supported REST API endpoints: * `GET /connectors/{name}/offsets` \- get the current offsets for a connector * `DELETE /connectors/{name}/offsets` \- reset the offsets for a connector. The connector must exist and must be in the stopped state (see `PUT /connectors/{name}/stop`) * `PATCH /connectors/{name}/offsets` \- alter the offsets for a connector. The connector must exist and must be in the stopped state (see `PUT /connectors/{name}/stop`). The request body should be a JSON object containing a JSON array `offsets` field, similar to the response body of the `GET /connectors/{name}/offsets` endpoint. An example request body for the `FileStreamSourceConnector`: - + + ```json + { + "offsets": [ { - "offsets": [ - { - "partition": { - "filename": "test.txt" - }, - "offset": { - "position": 30 - } - } - ] + "partition": { + "filename": "test.txt" + }, + "offset": { + "position": 30 + } } + ] + } + ``` An example request body for the `FileStreamSinkConnector`: - - { - "offsets": [ - { - "partition": { - "kafka_topic": "test", - "kafka_partition": 0 - }, - "offset": { - "kafka_offset": 5 - } - }, - { - "partition": { - "kafka_topic": "test", - "kafka_partition": 1 - }, - "offset": null - } - ] - } + +```json +{ + "offsets": [ + { + "partition": { + "kafka_topic": "test", + "kafka_partition": 0 + }, + "offset": { + "kafka_offset": 5 + } + }, + { + "partition": { + "kafka_topic": "test", + "kafka_partition": 1 + }, + "offset": null + } + ] +} +``` The "offset" field may be null to reset the offset for a specific partition (applicable to both source and sink connectors). Note that the request body format depends on the connector implementation in the case of source connectors, whereas there is a common format across all sink connectors. @@ -361,9 +374,10 @@ The following is a supported REST request at the top-level (root) endpoint: The `admin.listeners` configuration can be used to configure admin REST APIs on Kafka Connect's REST API server. Similar to the `listeners` configuration, this field should contain a list of listeners in the following format: `protocol://host:port,protocol2://host2:port2`. Currently supported protocols are `http` and `https`. For example: - - - admin.listeners=http://localhost:8080,https://localhost:8443 + +```properties +admin.listeners=http://localhost:8080,https://localhost:8443 +``` By default, if `admin.listeners` is not configured, the admin REST APIs will be available on the regular listeners. @@ -388,36 +402,38 @@ To report errors within a connector's converter, transforms, or within the sink To report errors within a connector's converter, transforms, or within the sink connector itself to a dead letter queue topic, set `errors.deadletterqueue.topic.name`, and optionally `errors.deadletterqueue.context.headers.enable=true`. By default connectors exhibit "fail fast" behavior immediately upon an error or exception. This is equivalent to adding the following configuration properties with their defaults to a connector configuration: - - - # disable retries on failure - errors.retry.timeout=0 - - # do not log the error and their contexts - errors.log.enable=false - - # do not record errors in a dead letter queue topic - errors.deadletterqueue.topic.name= - - # Fail on first error - errors.tolerance=none + +```properties +# disable retries on failure +errors.retry.timeout=0 + +# do not log the error and their contexts +errors.log.enable=false + +# do not record errors in a dead letter queue topic +errors.deadletterqueue.topic.name= + +# Fail on first error +errors.tolerance=none +``` These and other related connector configuration properties can be changed to provide different behavior. For example, the following configuration properties can be added to a connector configuration to setup error handling with multiple retries, logging to the application logs and the `my-connector-errors` Kafka topic, and tolerating all errors by reporting them rather than failing the connector task: - - - # retry for at most 10 minutes times waiting up to 30 seconds between consecutive failures - errors.retry.timeout=600000 - errors.retry.delay.max.ms=30000 - - # log error context along with application logs, but do not include configs and messages - errors.log.enable=true - errors.log.include.messages=false - - # produce error context into the Kafka topic - errors.deadletterqueue.topic.name=my-connector-errors - - # Tolerate all errors. - errors.tolerance=all + +```properties +# retry for at most 10 minutes times waiting up to 30 seconds between consecutive failures +errors.retry.timeout=600000 +errors.retry.delay.max.ms=30000 + +# log error context along with application logs, but do not include configs and messages +errors.log.enable=true +errors.log.include.messages=false + +# produce error context into the Kafka topic +errors.deadletterqueue.topic.name=my-connector-errors + +# Tolerate all errors. +errors.tolerance=all +``` ## Exactly-once support @@ -489,10 +505,11 @@ In order for a plugin to be compatible, it must appear as a line in a manifest c For example, if you only have one connector with the fully-qualified name `com.example.MySinkConnector`, then only one manifest file must be added to resources in `META-INF/services/org.apache.kafka.connect.sink.SinkConnector`, and the contents should be similar to the following: - - - # license header or comment - com.example.MySinkConnector + +```text +# license header or comment +com.example.MySinkConnector +``` You should then verify that your manifests are correct by using the verification steps with a pre-release artifact. If the verification succeeds, you can then release the plugin normally, and operators can upgrade to the compatible version. diff --git a/docs/operations/basic-kafka-operations.md b/docs/operations/basic-kafka-operations.md index dd042d0fcab4b..4e5686a3fa158 100644 --- a/docs/operations/basic-kafka-operations.md +++ b/docs/operations/basic-kafka-operations.md @@ -33,10 +33,11 @@ This section will review the most common operations you will perform on your Kaf You have the option of either adding topics manually or having them be created automatically when data is first published to a non-existent topic. If topics are auto-created then you may want to tune the default topic configurations used for auto-created topics. Topics are added and modified using the topic tool: - - - $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_topic_name \ - --partitions 20 --replication-factor 3 --config x=y + +```bash +$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_topic_name \ + --partitions 20 --replication-factor 3 --config x=y +``` The replication factor controls how many servers will replicate each message that is written. If you have a replication factor of 3 then up to 2 servers can fail before you will lose access to your data. We recommend you use a replication factor of 2 or 3 so that you can transparently bounce machines without interrupting data consumption. @@ -51,10 +52,11 @@ The configurations added on the command line override the default settings the s You can change the configuration or partitioning of a topic using the same topic tool. To add partitions you can do - - - $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic my_topic_name \ - --partitions 40 + +```bash +$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic my_topic_name \ + --partitions 40 +``` **Note:** Dynamically increasing the number of partitions for a topic has several important considerations and potential side effects: @@ -64,19 +66,22 @@ To add partitions you can do * **Risks with Internal Topics**: Users should **never** manually increase partitions for Kafka's internal state topics such as `__consumer_offsets`, `__transaction_state`, `__share_group_state`, or `__cluster_metadata`. Doing so can break coordinator mapping logic, cause state inconsistencies, and lead to data corruption or system failures. These topics are managed automatically by Kafka and should not be modified manually. To add configs: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my_topic_name --alter --add-config x=y + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my_topic_name --alter --add-config x=y +``` To remove a config: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my_topic_name --alter --delete-config x + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my_topic_name --alter --delete-config x +``` And finally deleting a topic: - - - $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic my_topic_name + +```bash +$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic my_topic_name +``` Kafka does not currently support reducing the number of partitions for a topic. @@ -89,9 +94,10 @@ The Kafka cluster will automatically detect any broker shutdown or failure and e 1. It will sync all its logs to disk to avoid needing to do any log recovery when it restarts (i.e. validating the checksum for all messages in the tail of the log). Log recovery takes time so this speeds up intentional restarts. 2. It will migrate any partitions the server is the leader for to other replicas prior to shutting down. This will make the leadership transfer faster and minimize the time each partition is unavailable to a few milliseconds. Syncing the logs will happen automatically whenever the server is stopped other than by a hard kill, but the controlled leadership migration requires using a special setting: - - - controlled.shutdown.enable=true + +```properties +controlled.shutdown.enable=true +``` Note that controlled shutdown will only succeed if _all_ the partitions hosted on the broker have replicas (i.e. the replication factor is greater than 1 _and_ at least one of these replicas is alive). This is generally what you want since shutting down the last replica would make that topic partition unavailable. @@ -100,23 +106,26 @@ Note that controlled shutdown will only succeed if _all_ the partitions hosted o Whenever a broker stops or crashes, leadership for that broker's partitions transfers to other replicas. When the broker is restarted it will only be a follower for all its partitions, meaning it will not be used for client reads and writes. To avoid this imbalance, Kafka has a notion of preferred replicas. If the list of replicas for a partition is 1,5,9 then node 1 is preferred as the leader to either node 5 or 9 because it is earlier in the replica list. By default the Kafka cluster will try to restore leadership to the preferred replicas. This behaviour is configured with: - - - auto.leader.rebalance.enable=true + +```properties +auto.leader.rebalance.enable=true +``` You can also set this to false, but you will then need to manually restore leadership to the restored replicas by running the command: - - - $ bin/kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type preferred --all-topic-partitions + +```bash +$ bin/kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type preferred --all-topic-partitions +``` ## Balancing replicas across racks The rack awareness feature spreads replicas of the same partition across different racks. This extends the guarantees Kafka provides for broker-failure to cover rack-failure, limiting the risk of data loss should all the brokers on a rack fail at once. The feature can also be applied to other broker groupings such as availability zones in EC2. You can specify that a broker belongs to a particular rack by adding a property to the broker config: - - - broker.rack=my-rack-id + +```properties +broker.rack=my-rack-id +``` When a topic is created, modified or replicas are redistributed, the rack constraint will be honoured, ensuring replicas span as many racks as they can (a partition will span min(#racks, replication-factor) different racks). @@ -131,77 +140,88 @@ Kafka administrators can define data flows that cross the boundaries of individu ## Checking consumer position Sometimes it's useful to see the position of your consumers. We have a tool that will show the position of all consumers in a consumer group as well as how far behind the end of the log they are. To run this tool on a consumer group named _my-group_ consuming a topic named _my-topic_ would look like this: - - - $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group - TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID - my-topic 0 2 4 2 consumer-1-029af89c-873c-4751-a720-cefd41a669d6 /127.0.0.1 consumer-1 - my-topic 1 2 3 1 consumer-1-029af89c-873c-4751-a720-cefd41a669d6 /127.0.0.1 consumer-1 - my-topic 2 2 3 1 consumer-2-42c1abd4-e3b2-425d-a8bb-e1ea49b29bb2 /127.0.0.1 consumer-2 + +```bash +$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group +TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID +my-topic 0 2 4 2 consumer-1-029af89c-873c-4751-a720-cefd41a669d6 /127.0.0.1 consumer-1 +my-topic 1 2 3 1 consumer-1-029af89c-873c-4751-a720-cefd41a669d6 /127.0.0.1 consumer-1 +my-topic 2 2 3 1 consumer-2-42c1abd4-e3b2-425d-a8bb-e1ea49b29bb2 /127.0.0.1 consumer-2 +``` ## Managing groups With the GroupCommand tool, we can list groups of all types, including consumer groups, share groups and streams groups. Each type of group has its own tool for administering groups of that type. For example, to list all groups in the cluster: - - - $ bin/kafka-groups.sh --bootstrap-server localhost:9092 --list - GROUP TYPE PROTOCOL - my-consumer-group Consumer consumer - my-share-group Share share + +```bash +$ bin/kafka-groups.sh --bootstrap-server localhost:9092 --list +GROUP TYPE PROTOCOL +my-consumer-group Consumer consumer +my-share-group Share share +``` ## Managing consumer groups With the ConsumerGroupCommand tool, we can list, describe, or delete the consumer groups. The consumer group can be deleted manually, or automatically when the last committed offset for that group expires. Manual deletion works only if the group does not have any active members. For example, to list all consumer groups across all topics: - - - $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list - test-consumer-group + +```bash +$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list +test-consumer-group +``` To view offsets, as mentioned earlier, we "describe" the consumer group like this: - - - $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group - TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID - topic3 0 241019 395308 154289 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 - topic2 1 520678 803288 282610 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 - topic3 1 241018 398817 157799 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 - topic1 0 854144 855809 1665 consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 - topic2 0 460537 803290 342753 consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 - topic3 2 243655 398812 155157 consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 + +```bash +$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group +TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID +topic3 0 241019 395308 154289 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 +topic2 1 520678 803288 282610 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 +topic3 1 241018 398817 157799 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 +topic1 0 854144 855809 1665 consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 +topic2 0 460537 803290 342753 consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 +topic3 2 243655 398812 155157 consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 +``` Note that if the consumer group uses the consumer protocol, the admin client needs DESCRIBE access to all the topics used in the group (topics the members are subscribed to). In contrast, the classic protocol does not require all topics DESCRIBE authorization. There are a number of additional "describe" options that can be used to provide more detailed information about a consumer group: * \--members: This option provides the list of all active members in the consumer group. - - $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members - CONSUMER-ID HOST CLIENT-ID #PARTITIONS - consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 2 - consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 1 - consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 3 - consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee /127.0.0.1 consumer3 0 + + ```bash + $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members + CONSUMER-ID HOST CLIENT-ID #PARTITIONS + consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 2 + consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 1 + consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 3 + consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee /127.0.0.1 consumer3 0 + ``` * \--members --verbose: On top of the information reported by the "--members" options above, this option also provides the partitions assigned to each member. - - $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose - CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT - consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 2 topic1(0), topic2(0) - consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 1 topic3(2) - consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 3 topic2(1), topic3(0,1) - consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee /127.0.0.1 consumer3 0 - + + ```bash + $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose + CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT + consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 2 topic1(0), topic2(0) + consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 1 topic3(2) + consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 3 topic2(1), topic3(0,1) + consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee /127.0.0.1 consumer3 0 - + ``` * \--offsets: This is the default describe option and provides the same output as the "--describe" option. * \--state: This option provides useful group-level information. - - $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state - COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS - localhost:9092 (0) range Stable 4 + + ```bash + $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state + COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS + localhost:9092 (0) range Stable 4 + ``` To manually delete one or multiple consumer groups, the "--delete" option can be used: - - - $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group - Deletion of requested consumer groups ('my-group', 'my-other-group') was successful. + +```bash +$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group +Deletion of requested consumer groups ('my-group', 'my-other-group') was successful. +``` To reset offsets of a consumer group, "--reset-offsets" option can be used. This option supports one consumer group at the time. It requires defining following scopes: --all-topics or --topic. One scope must be selected, unless you use '--from-file' scenario. Also, first make sure that the consumer instances are inactive. See [KIP-122](https://cwiki.apache.org/confluence/x/_iEIB) for more details. @@ -227,43 +247,50 @@ It has 3 execution options: Please note, that out of range offsets will be adjusted to available offset end. For example, if offset end is at 10 and offset shift request is of 15, then, offset at 10 will actually be selected. For example, to reset offsets of a consumer group to the latest offset: - - - $ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group my-group --topic topic1 --to-latest - TOPIC PARTITION NEW-OFFSET - topic1 0 0 + +```bash +$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group my-group --topic topic1 --to-latest +TOPIC PARTITION NEW-OFFSET +topic1 0 0 +``` ## Managing share groups Use the ShareGroupCommand tool to list, describe, or delete the share groups. Only share groups without any active members can be deleted. For example, to list all share groups in a cluster: - - - $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --list - my-share-group + +```bash +$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --list +my-share-group +``` To view the current start offset and lag, use the "--describe" option: - - - $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --describe --group my-share-group - GROUP TOPIC PARTITION START-OFFSET LAG - my-share-group topic1 0 4 0 + +```bash +$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --describe --group my-share-group +GROUP TOPIC PARTITION START-OFFSET LAG +my-share-group topic1 0 4 0 +``` The start offset is the earliest offset for in-flight records being evaluated for delivery to share consumers. Some records after the start offset may already have completed delivery. NOTE: The admin client needs DESCRIBE access to all the topics used in the group. There are many --describe options that provide more detailed information about a share group: * \--members: Describes active members in the share group. - - bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --describe --group my-share-group --members - GROUP CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT - my-share-group 94wrSQNmRda9Q6sk6jMO6Q /127.0.0.1 console-share-consumer 1 topic1:0 - my-share-group EfI0sha8QSKSrL_-I_zaTA /127.0.0.1 console-share-consumer 1 topic1:0 + + ```bash + bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --describe --group my-share-group --members + GROUP CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT + my-share-group 94wrSQNmRda9Q6sk6jMO6Q /127.0.0.1 console-share-consumer 1 topic1:0 + my-share-group EfI0sha8QSKSrL_-I_zaTA /127.0.0.1 console-share-consumer 1 topic1:0 + ``` You can see that both members have been assigned the same partition which they are sharing. * \--offsets: The default describe option. This provides the same output as the "--describe" option. * \--state: Describes a summary of the state of the share group. - - bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --describe --group my-share-group --state - GROUP COORDINATOR (ID) STATE #MEMBERS - my-share-group localhost:9092 (1) Stable 2 + + ```bash + bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --describe --group my-share-group --state + GROUP COORDINATOR (ID) STATE #MEMBERS + my-share-group localhost:9092 (1) Stable 2 + ``` @@ -286,24 +313,27 @@ It has 2 execution options: For example, to reset offsets of a share group to the latest offset: - - - $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group my-share-group --topic topic1 --to-latest --execute - GROUP TOPIC PARTITION NEW-OFFSET - my-share-group topic1 0 10 + +```bash +$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group my-share-group --topic topic1 --to-latest --execute +GROUP TOPIC PARTITION NEW-OFFSET +my-share-group topic1 0 10 +``` To delete the offsets of individual topics in the share group, use the "--delete-offsets" option: - - - $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --delete-offsets --group my-share-group --topic topic1 - TOPIC STATUS - topic1 Successful + +```bash +$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --delete-offsets --group my-share-group --topic topic1 +TOPIC STATUS +topic1 Successful +``` To delete one or more share groups, use "--delete" option: - - - $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --delete --group my-share-group - Deletion of requested share groups ('my-share-group') was successful. + +```bash +$ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --delete --group my-share-group +Deletion of requested share groups ('my-share-group') was successful. +``` ## Expanding your cluster @@ -328,70 +358,74 @@ The partition reassignment tool can be used to move some topics off of the curre For instance, the following example will move all partitions for topics foo1,foo2 to the new set of brokers 5,6. At the end of this move, all partitions for topics foo1 and foo2 will _only_ exist on brokers 5,6. Since the tool accepts the input list of topics as a json file, you first need to identify the topics you want to move and create the json file as follows: - - - $ cat topics-to-move.json - { - "topics": [ - { "topic": "foo1" }, - { "topic": "foo2" } - ], - "version": 1 - } + +```bash +$ cat topics-to-move.json +{ + "topics": [ + { "topic": "foo1" }, + { "topic": "foo2" } + ], + "version": 1 +} +``` Once the json file is ready, use the partition reassignment tool to generate a candidate assignment: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate - Current partition replica assignment - {"version":1, - "partitions":[{"topic":"foo1","partition":0,"replicas":[2,1],"log_dirs":["any"]}, - {"topic":"foo1","partition":1,"replicas":[1,3],"log_dirs":["any"]}, - {"topic":"foo1","partition":2,"replicas":[3,4],"log_dirs":["any"]}, - {"topic":"foo2","partition":0,"replicas":[4,2],"log_dirs":["any"]}, - {"topic":"foo2","partition":1,"replicas":[2,1],"log_dirs":["any"]}, - {"topic":"foo2","partition":2,"replicas":[1,3],"log_dirs":["any"]}] - } - - Proposed partition reassignment configuration - {"version":1, - "partitions":[{"topic":"foo1","partition":0,"replicas":[6,5],"log_dirs":["any"]}, - {"topic":"foo1","partition":1,"replicas":[5,6],"log_dirs":["any"]}, - {"topic":"foo1","partition":2,"replicas":[6,5],"log_dirs":["any"]}, - {"topic":"foo2","partition":0,"replicas":[5,6],"log_dirs":["any"]}, - {"topic":"foo2","partition":1,"replicas":[6,5],"log_dirs":["any"]}, - {"topic":"foo2","partition":2,"replicas":[5,6],"log_dirs":["any"]}] - } + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate +Current partition replica assignment +{"version":1, + "partitions":[{"topic":"foo1","partition":0,"replicas":[2,1],"log_dirs":["any"]}, + {"topic":"foo1","partition":1,"replicas":[1,3],"log_dirs":["any"]}, + {"topic":"foo1","partition":2,"replicas":[3,4],"log_dirs":["any"]}, + {"topic":"foo2","partition":0,"replicas":[4,2],"log_dirs":["any"]}, + {"topic":"foo2","partition":1,"replicas":[2,1],"log_dirs":["any"]}, + {"topic":"foo2","partition":2,"replicas":[1,3],"log_dirs":["any"]}] +} + +Proposed partition reassignment configuration +{"version":1, + "partitions":[{"topic":"foo1","partition":0,"replicas":[6,5],"log_dirs":["any"]}, + {"topic":"foo1","partition":1,"replicas":[5,6],"log_dirs":["any"]}, + {"topic":"foo1","partition":2,"replicas":[6,5],"log_dirs":["any"]}, + {"topic":"foo2","partition":0,"replicas":[5,6],"log_dirs":["any"]}, + {"topic":"foo2","partition":1,"replicas":[6,5],"log_dirs":["any"]}, + {"topic":"foo2","partition":2,"replicas":[5,6],"log_dirs":["any"]}] +} +``` The tool generates a candidate assignment that will move all partitions from topics foo1,foo2 to brokers 5,6. Note, however, that at this point, the partition movement has not started, it merely tells you the current assignment and the proposed new assignment. The current assignment should be saved in case you want to rollback to it. The new assignment should be saved in a json file (e.g. expand-cluster-reassignment.json) to be input to the tool with the --execute option as follows: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expand-cluster-reassignment.json --execute - Current partition replica assignment - - {"version":1, - "partitions":[{"topic":"foo1","partition":0,"replicas":[2,1],"log_dirs":["any"]}, - {"topic":"foo1","partition":1,"replicas":[1,3],"log_dirs":["any"]}, - {"topic":"foo1","partition":2,"replicas":[3,4],"log_dirs":["any"]}, - {"topic":"foo2","partition":0,"replicas":[4,2],"log_dirs":["any"]}, - {"topic":"foo2","partition":1,"replicas":[2,1],"log_dirs":["any"]}, - {"topic":"foo2","partition":2,"replicas":[1,3],"log_dirs":["any"]}] - } - - Save this to use as the --reassignment-json-file option during rollback - Successfully started partition reassignments for foo1-0,foo1-1,foo1-2,foo2-0,foo2-1,foo2-2 + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expand-cluster-reassignment.json --execute +Current partition replica assignment + +{"version":1, + "partitions":[{"topic":"foo1","partition":0,"replicas":[2,1],"log_dirs":["any"]}, + {"topic":"foo1","partition":1,"replicas":[1,3],"log_dirs":["any"]}, + {"topic":"foo1","partition":2,"replicas":[3,4],"log_dirs":["any"]}, + {"topic":"foo2","partition":0,"replicas":[4,2],"log_dirs":["any"]}, + {"topic":"foo2","partition":1,"replicas":[2,1],"log_dirs":["any"]}, + {"topic":"foo2","partition":2,"replicas":[1,3],"log_dirs":["any"]}] +} + +Save this to use as the --reassignment-json-file option during rollback +Successfully started partition reassignments for foo1-0,foo1-1,foo1-2,foo2-0,foo2-1,foo2-2 +``` Finally, the --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json (used with the --execute option) should be used with the --verify option: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expand-cluster-reassignment.json --verify - Status of partition reassignment: - Reassignment of partition [foo1,0] is completed - Reassignment of partition [foo1,1] is still in progress - Reassignment of partition [foo1,2] is still in progress - Reassignment of partition [foo2,0] is completed - Reassignment of partition [foo2,1] is completed - Reassignment of partition [foo2,2] is completed + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expand-cluster-reassignment.json --verify +Status of partition reassignment: +Reassignment of partition [foo1,0] is completed +Reassignment of partition [foo1,1] is still in progress +Reassignment of partition [foo1,2] is still in progress +Reassignment of partition [foo2,0] is completed +Reassignment of partition [foo2,1] is completed +Reassignment of partition [foo2,2] is completed +``` ### Custom partition assignment and migration @@ -400,32 +434,35 @@ The partition reassignment tool can also be used to selectively move replicas of For instance, the following example moves partition 0 of topic foo1 to brokers 5,6 and partition 1 of topic foo2 to brokers 2,3: The first step is to hand craft the custom reassignment plan in a json file: - - - $ cat custom-reassignment.json - {"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]} + +```bash +$ cat custom-reassignment.json +{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]} +``` Then, use the json file with the --execute option to start the reassignment process: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file custom-reassignment.json --execute - Current partition replica assignment - - {"version":1, - "partitions":[{"topic":"foo1","partition":0,"replicas":[1,2],"log_dirs":["any"]}, - {"topic":"foo2","partition":1,"replicas":[3,4],"log_dirs":["any"]}] - } - - Save this to use as the --reassignment-json-file option during rollback - Successfully started partition reassignments for foo1-0,foo2-1 + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file custom-reassignment.json --execute +Current partition replica assignment + +{"version":1, + "partitions":[{"topic":"foo1","partition":0,"replicas":[1,2],"log_dirs":["any"]}, + {"topic":"foo2","partition":1,"replicas":[3,4],"log_dirs":["any"]}] +} + +Save this to use as the --reassignment-json-file option during rollback +Successfully started partition reassignments for foo1-0,foo2-1 +``` The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same custom-reassignment.json (used with the --execute option) should be used with the --verify option: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file custom-reassignment.json --verify - Status of partition reassignment: - Reassignment of partition [foo1,0] is completed - Reassignment of partition [foo2,1] is completed + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file custom-reassignment.json --verify +Status of partition reassignment: +Reassignment of partition [foo1,0] is completed +Reassignment of partition [foo2,1] is completed +``` ## Decommissioning brokers and log directories @@ -435,8 +472,10 @@ The first step to decommission brokers is to mark them as cordoned via the Admin For example to cordon broker 1: - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config cordoned.log.dirs="*" --entity-type brokers --entity-name 1 - Completed updating config for broker 1. +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config cordoned.log.dirs="*" --entity-type brokers --entity-name 1 +Completed updating config for broker 1. +``` Then reassign all the partitions from that broker to other brokers in the cluster. The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet. @@ -447,7 +486,9 @@ Once all the reassignment is done, shutdown the broker and unregister it to remo For example to unregister broker 1: - $ bin/kafka-cluster.sh unregister --bootstrap-server localhost:9092 --id 1 +```bash +$ bin/kafka-cluster.sh unregister --bootstrap-server localhost:9092 --id 1 +``` ### Decommissioning log directories @@ -455,8 +496,10 @@ The first step to decommission log directories is to mark them as cordoned via t For example to cordon /data/dir1 from broker 1: - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config cordoned.log.dirs=/data/dir1 --entity-type brokers --entity-name 1 - Completed updating config for broker 1. +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config cordoned.log.dirs=/data/dir1 --entity-type brokers --entity-name 1 +Completed updating config for broker 1. +``` Then reassign all the partitions from the log directory to decommission to other log directories or brokers in the cluster. The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning a log directory yet. @@ -468,18 +511,24 @@ Then uncordon the log directory. Since the broker hosting that directory is offl For example: - $ bin/kafka-configs.sh --bootstrap-controller localhost:9093 --alter --delete-config cordoned.log.dirs --entity-type brokers --entity-name 1 - Completed updating config for broker 1. +```bash +$ bin/kafka-configs.sh --bootstrap-controller localhost:9093 --alter --delete-config cordoned.log.dirs --entity-type brokers --entity-name 1 +Completed updating config for broker 1. +``` Update the configuration for the broker and remove the log directory to decommission from log.dir or log.dirs. For example if the broker configuration contained: - log.dirs=/data/dir1,/data/dir2 +```properties +log.dirs=/data/dir1,/data/dir2 +``` Update it to: - log.dirs=/data/dir2 +```properties +log.dirs=/data/dir2 +``` Finally restart the broker. @@ -490,37 +539,41 @@ Increasing the replication factor of an existing partition is easy. Just specify For instance, the following example increases the replication factor of partition 0 of topic foo from 1 to 3. Before increasing the replication factor, the partition's only replica existed on broker 5. As part of increasing the replication factor, we will add more replicas on brokers 6 and 7. The first step is to hand craft the custom reassignment plan in a json file: - - - $ cat increase-replication-factor.json - {"version":1, - "partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]} + +```bash +$ cat increase-replication-factor.json +{"version":1, + "partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]} +``` Then, use the json file with the --execute option to start the reassignment process: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase-replication-factor.json --execute - Current partition replica assignment - - {"version":1, - "partitions":[{"topic":"foo","partition":0,"replicas":[5],"log_dirs":["any"]}]} - - Save this to use as the --reassignment-json-file option during rollback - Successfully started partition reassignment for foo-0 + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase-replication-factor.json --execute +Current partition replica assignment + +{"version":1, + "partitions":[{"topic":"foo","partition":0,"replicas":[5],"log_dirs":["any"]}]} + +Save this to use as the --reassignment-json-file option during rollback +Successfully started partition reassignment for foo-0 +``` The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same increase-replication-factor.json (used with the --execute option) should be used with the --verify option: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase-replication-factor.json --verify - Status of partition reassignment: - Reassignment of partition [foo,0] is completed + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase-replication-factor.json --verify +Status of partition reassignment: +Reassignment of partition [foo,0] is completed +``` You can also verify the increase in replication factor with the kafka-topics.sh tool: - - - $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic foo --describe - Topic:foo PartitionCount:1 ReplicationFactor:3 Configs: - Topic: foo Partition: 0 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7 + +```bash +$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic foo --describe +Topic:foo PartitionCount:1 ReplicationFactor:3 Configs: + Topic: foo Partition: 0 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7 +``` ## Limiting bandwidth usage during data migration @@ -529,68 +582,76 @@ Kafka lets you apply a throttle to replication traffic, setting an upper bound o There are two interfaces that can be used to engage a throttle. The simplest, and safest, is to apply a throttle when invoking the kafka-reassign-partitions.sh, but kafka-configs.sh can also be used to view and alter the throttle values directly. So for example, if you were to execute a rebalance, with the below command, it would move partitions at no more than 50MB/s between brokers, and at no more than 100MB/s between disks on a broker. - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --throttle 50000000 --replica-alter-log-dirs-throttle 100000000 + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --throttle 50000000 --replica-alter-log-dirs-throttle 100000000 +``` When you execute this script you will see the throttle engage: - - - The inter-broker throttle limit was set to 50000000 B/s - The replica-alter-dir throttle limit was set to 100000000 B/s - Successfully started partition reassignment for foo1-0 + +```text +The inter-broker throttle limit was set to 50000000 B/s +The replica-alter-dir throttle limit was set to 100000000 B/s +Successfully started partition reassignment for foo1-0 +``` Should you wish to alter the throttle, during a rebalance, say to increase the inter-broker throughput so it completes quicker, you can do this by re-running the execute command with the --additional option passing the same reassignment-json-file: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --additional --execute --reassignment-json-file bigger-cluster.json --throttle 700000000 - The inter-broker throttle limit was set to 700000000 B/s + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --additional --execute --reassignment-json-file bigger-cluster.json --throttle 700000000 +The inter-broker throttle limit was set to 700000000 B/s +``` Once the rebalance completes the administrator can check the status of the rebalance using the --verify option. If the rebalance has completed, the throttle will be removed via the --verify command. It is important that administrators remove the throttle in a timely manner once rebalancing completes by running the command with the --verify option. Failure to do so could cause regular replication traffic to be throttled. When the --verify option is executed, and the reassignment has completed, the script will confirm that the throttle was removed: - - - $ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --verify --reassignment-json-file bigger-cluster.json - Status of partition reassignment: - Reassignment of partition [my-topic,1] is completed - Reassignment of partition [my-topic,0] is completed - - Clearing broker-level throttles on brokers 1,2,3 - Clearing topic-level throttles on topic my-topic + +```bash +$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --verify --reassignment-json-file bigger-cluster.json +Status of partition reassignment: +Reassignment of partition [my-topic,1] is completed +Reassignment of partition [my-topic,0] is completed + +Clearing broker-level throttles on brokers 1,2,3 +Clearing topic-level throttles on topic my-topic +``` The administrator can also validate the assigned configs using the kafka-configs.sh. There are two sets of throttle configuration used to manage the throttling process. First set refers to the throttle value itself. This is configured, at a broker level, using the dynamic properties: - - - leader.replication.throttled.rate - follower.replication.throttled.rate - replica.alter.log.dirs.io.max.bytes.per.second + +```properties +leader.replication.throttled.rate +follower.replication.throttled.rate +replica.alter.log.dirs.io.max.bytes.per.second +``` Then there is the configuration pair of enumerated sets of throttled replicas: - - - leader.replication.throttled.replicas - follower.replication.throttled.replicas + +```properties +leader.replication.throttled.replicas +follower.replication.throttled.replicas +``` Which are configured per topic. All five config values are automatically assigned by kafka-reassign-partitions.sh (discussed below). To view the throttle limit configuration: - - - $ bin/kafka-configs.sh --describe --bootstrap-server localhost:9092 --entity-type brokers - Configs for brokers '2' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000,replica.alter.log.dirs.io.max.bytes.per.second=1000000000 - Configs for brokers '1' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000,replica.alter.log.dirs.io.max.bytes.per.second=1000000000 + +```bash +$ bin/kafka-configs.sh --describe --bootstrap-server localhost:9092 --entity-type brokers +Configs for brokers '2' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000,replica.alter.log.dirs.io.max.bytes.per.second=1000000000 +Configs for brokers '1' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000,replica.alter.log.dirs.io.max.bytes.per.second=1000000000 +``` This shows the throttle applied to both leader and follower side of the replication protocol (by default both sides are assigned the same throttled throughput value), as well as the disk throttle. To view the list of throttled replicas: - - - $ bin/kafka-configs.sh --describe --bootstrap-server localhost:9092 --entity-type topics - Configs for topic 'my-topic' are leader.replication.throttled.replicas=1:102,0:101, - follower.replication.throttled.replicas=1:101,0:102 + +```bash +$ bin/kafka-configs.sh --describe --bootstrap-server localhost:9092 --entity-type topics +Configs for topic 'my-topic' are leader.replication.throttled.replicas=1:102,0:101, + follower.replication.throttled.replicas=1:101,0:102 +``` Here we see the leader throttle is applied to partition 1 on broker 102 and partition 0 on broker 101. Likewise the follower throttle is applied to partition 1 on broker 101 and partition 0 on broker 102. @@ -609,16 +670,18 @@ The throttle should be removed in a timely manner once reassignment completes (b _(2) Ensuring Progress:_ If the throttle is set too low, in comparison to the incoming write rate, it is possible for replication to not make progress. This occurs when: - - - max(BytesInPerSec) > throttle + +```text +max(BytesInPerSec) > throttle +``` Where BytesInPerSec is the metric that monitors the write throughput of producers into each broker. The administrator can monitor whether replication is making progress, during the rebalance, using the metric: - - - kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+) + +```text +kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+) +``` The lag should constantly decrease during replication. If the metric does not decrease the administrator should increase the throttle throughput as described above. @@ -627,83 +690,96 @@ The lag should constantly decrease during replication. If the metric does not de Quotas overrides and defaults may be configured at (user, client-id), user or client-id levels as described here. By default, clients receive an unlimited quota. It is possible to set custom quotas for each (user, client-id), user or client-id group. Configure custom quota for (user=user1, client-id=clientA): - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 --entity-type clients --entity-name clientA - Updated config for entity: user-principal 'user1', client-id 'clientA'. + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 --entity-type clients --entity-name clientA +Updated config for entity: user-principal 'user1', client-id 'clientA'. +``` Configure custom quota for user=user1: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 - Updated config for entity: user-principal 'user1'. + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 +Updated config for entity: user-principal 'user1'. +``` Configure custom quota for client-id=clientA: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-name clientA - Updated config for entity: client-id 'clientA'. + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-name clientA +Updated config for entity: client-id 'clientA'. +``` It is possible to set default quotas for each (user, client-id), user or client-id group by specifying _\--entity-default_ option instead of _\--entity-name_. Configure default client-id quota for user=user1: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 --entity-type clients --entity-default - Updated config for entity: user-principal 'user1', default client-id. + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 --entity-type clients --entity-default +Updated config for entity: user-principal 'user1', default client-id. +``` Configure default quota for user: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-default - Updated config for entity: default user-principal. + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-default +Updated config for entity: default user-principal. +``` Configure default quota for client-id: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-default - Updated config for entity: default client-id. + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-default +Updated config for entity: default client-id. +``` Here's how to describe the quota for a given (user, client-id): - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users --entity-name user1 --entity-type clients --entity-name clientA - Configs for user-principal 'user1', client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users --entity-name user1 --entity-type clients --entity-name clientA +Configs for user-principal 'user1', client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 +``` Describe quota for a given user: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users --entity-name user1 - Configs for user-principal 'user1' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users --entity-name user1 +Configs for user-principal 'user1' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 +``` Describe quota for a given client-id: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type clients --entity-name clientA - Configs for client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type clients --entity-name clientA +Configs for client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 +``` Describe default quota for user: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users --entity-default - Quota configs for the default user-principal are consumer_byte_rate=2048.0, request_percentage=200.0, producer_byte_rate=1024.0 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users --entity-default +Quota configs for the default user-principal are consumer_byte_rate=2048.0, request_percentage=200.0, producer_byte_rate=1024.0 +``` Describe default quota for client-id: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type clients --entity-default - Quota configs for the default client-id are consumer_byte_rate=2048.0, request_percentage=200.0, producer_byte_rate=1024.0 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type clients --entity-default +Quota configs for the default client-id are consumer_byte_rate=2048.0, request_percentage=200.0, producer_byte_rate=1024.0 +``` If entity name is not specified, all entities of the specified type are described. For example, describe all users: - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users - Configs for user-principal 'user1' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 - Configs for default user-principal are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users +Configs for user-principal 'user1' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 +Configs for default user-principal are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 +``` Similarly for (user, client): - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users --entity-type clients - Configs for user-principal 'user1', default client-id are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 - Configs for user-principal 'user1', client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type users --entity-type clients +Configs for user-principal 'user1', default client-id are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 +Configs for user-principal 'user1', client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 +``` diff --git a/docs/operations/geo-replication-(cross-cluster-data-mirroring).md b/docs/operations/geo-replication-(cross-cluster-data-mirroring).md index f68f09b71d1f9..8a0284c76db90 100644 --- a/docs/operations/geo-replication-(cross-cluster-data-mirroring).md +++ b/docs/operations/geo-replication-(cross-cluster-data-mirroring).md @@ -72,16 +72,17 @@ Here are some example patterns: By default, a flow replicates all topics and consumer groups (except excluded ones). However, each replication flow can be configured independently. For instance, you can define that only specific topics or consumer groups are replicated from the source cluster to the target cluster. Here is a first example on how to configure data replication from a `primary` cluster to a `secondary` cluster (an active/passive setup): - - - # Basic settings - clusters = primary, secondary - primary.bootstrap.servers = broker3-primary:9092 - secondary.bootstrap.servers = broker5-secondary:9092 - - # Define replication flows - primary->secondary.enabled = true - primary->secondary.topics = foobar-topic, quux-.* + +```properties +# Basic settings +clusters = primary, secondary +primary.bootstrap.servers = broker3-primary:9092 +secondary.bootstrap.servers = broker5-secondary:9092 + +# Define replication flows +primary->secondary.enabled = true +primary->secondary.topics = foobar-topic, quux-.* +``` ## Configuring Geo-Replication @@ -106,36 +107,39 @@ The MirrorMaker configuration file is typically named `connect-mirror-maker.prop Example: Define MirrorMaker settings (explained in more detail later). - - - # Global settings - clusters = us-west, us-east # defines cluster aliases - us-west.bootstrap.servers = broker3-west:9092 - us-east.bootstrap.servers = broker5-east:9092 - - topics = .* # all topics to be replicated by default - - # Specific replication flow settings (here: flow from us-west to us-east) - us-west->us-east.enabled = true - us-west->us.east.topics = foo.*, bar.* # override the default above + +```properties +# Global settings +clusters = us-west, us-east # defines cluster aliases +us-west.bootstrap.servers = broker3-west:9092 +us-east.bootstrap.servers = broker5-east:9092 + +topics = .* # all topics to be replicated by default + +# Specific replication flow settings (here: flow from us-west to us-east) +us-west->us-east.enabled = true +us-west->us.east.topics = foo.*, bar.* # override the default above +``` MirrorMaker is based on the Kafka Connect framework. Any Kafka Connect, source connector, and sink connector settings as described in the documentation chapter on Kafka Connect can be used directly in the MirrorMaker configuration, without having to change or prefix the name of the configuration setting. Example: Define custom Kafka Connect settings to be used by MirrorMaker. - - - # Setting Kafka Connect defaults for MirrorMaker - tasks.max = 5 + +```properties +# Setting Kafka Connect defaults for MirrorMaker +tasks.max = 5 +``` Most of the default Kafka Connect settings work well for MirrorMaker out-of-the-box, with the exception of `tasks.max`. In order to evenly distribute the workload across more than one MirrorMaker process, it is recommended to set `tasks.max` to at least `2` (preferably higher) depending on the available hardware resources and the total number of topic-partitions to be replicated. You can further customize MirrorMaker's Kafka Connect settings _per source or target cluster_ (more precisely, you can specify Kafka Connect worker-level configuration settings "per connector"). Use the format of `{cluster}.{config_name}` in the MirrorMaker configuration file. Example: Define custom connector settings for the `us-west` cluster. - - - # us-west custom settings - us-west.offset.storage.topic = my-mirrormaker-offsets + +```properties +# us-west custom settings +us-west.offset.storage.topic = my-mirrormaker-offsets +``` MirrorMaker internally uses the Kafka producer, consumer, and admin clients. Custom settings for these clients are often needed. To override the defaults, use the following format in the MirrorMaker configuration file: @@ -146,40 +150,44 @@ MirrorMaker internally uses the Kafka producer, consumer, and admin clients. Cus Example: Define custom producer, consumer, admin client settings. - - - # us-west cluster (from which to consume) - us-west.consumer.isolation.level = read_committed - us-west.admin.bootstrap.servers = broker57-primary:9092 - - # us-east cluster (to which to produce) - us-east.producer.compression.type = gzip - us-east.producer.buffer.memory = 32768 - us-east.admin.bootstrap.servers = broker8-secondary:9092 + +```properties +# us-west cluster (from which to consume) +us-west.consumer.isolation.level = read_committed +us-west.admin.bootstrap.servers = broker57-primary:9092 + +# us-east cluster (to which to produce) +us-east.producer.compression.type = gzip +us-east.producer.buffer.memory = 32768 +us-east.admin.bootstrap.servers = broker8-secondary:9092 +``` ### Exactly once Exactly-once semantics are supported for dedicated MirrorMaker clusters as of version 3.5.0. For new MirrorMaker clusters, set the `exactly.once.source.support` property to enabled for all targeted Kafka clusters that should be written to with exactly-once semantics. For example, to enable exactly-once for writes to cluster `us-east`, the following configuration can be used: - - - us-east.exactly.once.source.support = enabled + +```properties +us-east.exactly.once.source.support = enabled +``` For existing MirrorMaker clusters, a two-step upgrade is necessary. Instead of immediately setting the `exactly.once.source.support` property to enabled, first set it to `preparing` on all nodes in the cluster. Once this is complete, it can be set to `enabled` on all nodes in the cluster, in a second round of restarts. In either case, it is also necessary to enable intra-cluster communication between the MirrorMaker nodes, as described in [KIP-710](https://cwiki.apache.org/confluence/x/4g5RCg). To do this, the `dedicated.mode.enable.internal.rest` property must be set to `true`. In addition, many of the REST-related [configuration properties available for Kafka Connect](https://kafka.apache.org/documentation/#connectconfigs) can be specified in the MirrorMaker config. For example, to enable intra-cluster communication in MirrorMaker cluster with each node listening on port 8080 of their local machine, the following should be added to the MirrorMaker config file: - - - dedicated.mode.enable.internal.rest = true - listeners = http://localhost:8080 + +```properties +dedicated.mode.enable.internal.rest = true +listeners = http://localhost:8080 +``` **Note that, if intra-cluster communication is enabled in production environments, it is highly recommended to secure the REST servers brought up by each MirrorMaker node. See the [configuration properties for Kafka Connect](https://kafka.apache.org/documentation/#connectconfigs) for information on how this can be accomplished.** It is also recommended to filter records from aborted transactions out from replicated data when running MirrorMaker. To do this, ensure that the consumer used to read from source clusters is configured with `isolation.level` set to `read_committed`. If replicating data from cluster `us-west`, this can be done for all replication flows that read from that cluster by adding the following to the MirrorMaker config file: - - - us-west.consumer.isolation.level = read_committed + +```properties +us-west.consumer.isolation.level = read_committed +``` As a final note, under the hood, MirrorMaker uses Kafka Connect source connectors to replicate data. For more information on exactly-once support for these kinds of connectors, see the [relevant docs page](https://kafka.apache.org/documentation/#connect_exactlyoncesource). @@ -192,17 +200,19 @@ To define a replication flow, you must first define the respective source and ta Example: Define two cluster aliases `primary` and `secondary`, including their connection information. - - - clusters = primary, secondary - primary.bootstrap.servers = broker10-primary:9092,broker-11-primary:9092 - secondary.bootstrap.servers = broker5-secondary:9092,broker6-secondary:9092 + +```properties +clusters = primary, secondary +primary.bootstrap.servers = broker10-primary:9092,broker-11-primary:9092 +secondary.bootstrap.servers = broker5-secondary:9092,broker6-secondary:9092 +``` Secondly, you must explicitly enable individual replication flows with `{source}->{target}.enabled = true` as needed. Remember that flows are directional: if you need two-way (bidirectional) replication, you must enable flows in both directions. - - - # Enable replication from primary to secondary - primary->secondary.enabled = true + +```properties +# Enable replication from primary to secondary +primary->secondary.enabled = true +``` By default, a replication flow will replicate all but a few special topics and consumer groups from the source cluster to the target cluster, and automatically detect any newly created topics and groups. The names of replicated topics in the target cluster will be prefixed with the name of the source cluster (see section further below). For example, the topic `foo` in the source cluster `us-west` would be replicated to a topic named `us-west.foo` in the target cluster `us-east`. @@ -222,19 +232,20 @@ The most important settings are: Example: - - - # Custom top-level defaults that apply to all replication flows - topics = .* - groups = consumer-group1, consumer-group2 - - # Don't forget to enable a flow! - us-west->us-east.enabled = true - - # Custom settings for specific replication flows - us-west->us-east.topics = foo.* - us-west->us-east.groups = bar.* - us-west->us-east.emit.heartbeats = false + +```properties +# Custom top-level defaults that apply to all replication flows +topics = .* +groups = consumer-group1, consumer-group2 + +# Don't forget to enable a flow! +us-west->us-east.enabled = true + +# Custom settings for specific replication flows +us-west->us-east.topics = foo.* +us-west->us-east.groups = bar.* +us-west->us-east.emit.heartbeats = false +``` Additional configuration settings are supported which can be left with their default values in most cases. See [MirrorMaker Configs](/documentation/#mirrormakerconfigs). @@ -243,30 +254,33 @@ Additional configuration settings are supported which can be left with their def MirrorMaker supports the same security settings as Kafka Connect, so please refer to the linked section for further information. Example: Encrypt communication between MirrorMaker and the `us-east` cluster. - - - us-east.security.protocol=SSL - us-east.ssl.truststore.location=/path/to/truststore.jks - us-east.ssl.truststore.password=my-secret-password - us-east.ssl.keystore.location=/path/to/keystore.jks - us-east.ssl.keystore.password=my-secret-password - us-east.ssl.key.password=my-secret-password + +```properties +us-east.security.protocol=SSL +us-east.ssl.truststore.location=/path/to/truststore.jks +us-east.ssl.truststore.password=my-secret-password +us-east.ssl.keystore.location=/path/to/keystore.jks +us-east.ssl.keystore.password=my-secret-password +us-east.ssl.key.password=my-secret-password +``` ### Custom Naming of Replicated Topics in Target Clusters Replicated topics in a target cluster-sometimes called _remote_ topics-are renamed according to a replication policy. MirrorMaker uses this policy to ensure that events (aka records, messages) from different clusters are not written to the same topic-partition. By default as per [DefaultReplicationPolicy](https://github.com/apache/kafka/blob/trunk/connect/mirror-client/src/main/java/org/apache/kafka/connect/mirror/DefaultReplicationPolicy.java), the names of replicated topics in the target clusters have the format `{source}.{source_topic_name}`: - - - us-west us-east - ========= ================= - bar-topic - foo-topic --> us-west.foo-topic + +```text +us-west us-east +========= ================= + bar-topic +foo-topic --> us-west.foo-topic +``` You can customize the separator (default: `.`) with the `replication.policy.separator` setting: - - - # Defining a custom separator - us-west->us-east.replication.policy.separator = _ + +```properties +# Defining a custom separator +us-west->us-east.replication.policy.separator = _ +``` If you need further control over how replicated topics are named, you can implement a custom `ReplicationPolicy` and override `replication.policy.class` (default is `DefaultReplicationPolicy`) in the MirrorMaker configuration. @@ -275,15 +289,16 @@ If you need further control over how replicated topics are named, you can implem MirrorMaker processes share configuration via their target Kafka clusters. This behavior may cause conflicts when configurations differ among MirrorMaker processes that operate against the same target cluster. For example, the following two MirrorMaker processes would be racy: - - - # Configuration of process 1 - A->B.enabled = true - A->B.topics = foo - - # Configuration of process 2 - A->B.enabled = true - A->B.topics = bar + +```properties +# Configuration of process 1 +A->B.enabled = true +A->B.topics = foo + +# Configuration of process 2 +A->B.enabled = true +A->B.topics = bar +``` In this case, the two processes will share configuration via cluster `B`, which causes a conflict. Depending on which of the two processes is the elected "leader", the result will be that either the topic `foo` or the topic `bar` is replicated, but not both. @@ -292,47 +307,51 @@ It is therefore important to keep the MirrorMaker configuration consistent acros ### Best Practice: Consume from Remote, Produce to Local To minimize latency ("producer lag"), it is recommended to locate MirrorMaker processes as close as possible to their target clusters, i.e., the clusters that it produces data to. That's because Kafka producers typically struggle more with unreliable or high-latency network connections than Kafka consumers. - - - First DC Second DC - ========== ========================= - primary --------- MirrorMaker --> secondary - (remote) (local) + +```text +First DC Second DC +========== ========================= +primary --------- MirrorMaker --> secondary +(remote) (local) +``` To run such a "consume from remote, produce to local" setup, run the MirrorMaker processes close to and preferably in the same location as the target clusters, and explicitly set these "local" clusters in the `--clusters` command line parameter (blank-separated list of cluster aliases): - - - # Run in secondary's data center, reading from the remote `primary` cluster - $ bin/connect-mirror-maker.sh connect-mirror-maker.properties --clusters secondary + +```bash +# Run in secondary's data center, reading from the remote `primary` cluster +$ bin/connect-mirror-maker.sh connect-mirror-maker.properties --clusters secondary +``` The `--clusters secondary` tells the MirrorMaker process that the given cluster(s) are nearby, and prevents it from replicating data or sending configuration to clusters at other, remote locations. ### Example: Active/Passive High Availability Deployment The following example shows the basic settings to replicate topics from a primary to a secondary Kafka environment, but not from the secondary back to the primary. Please be aware that most production setups will need further configuration, such as security settings. - - - # Unidirectional flow (one-way) from primary to secondary cluster - primary.bootstrap.servers = broker1-primary:9092 - secondary.bootstrap.servers = broker2-secondary:9092 - - primary->secondary.enabled = true - secondary->primary.enabled = false - - primary->secondary.topics = foo.* # only replicate some topics + +```properties +# Unidirectional flow (one-way) from primary to secondary cluster +primary.bootstrap.servers = broker1-primary:9092 +secondary.bootstrap.servers = broker2-secondary:9092 + +primary->secondary.enabled = true +secondary->primary.enabled = false + +primary->secondary.topics = foo.* # only replicate some topics +``` ### Example: Active/Active High Availability Deployment The following example shows the basic settings to replicate topics between two clusters in both ways. Please be aware that most production setups will need further configuration, such as security settings. - - - # Bidirectional flow (two-way) between us-west and us-east clusters - clusters = us-west, us-east - us-west.bootstrap.servers = broker1-west:9092,broker2-west:9092 - Us-east.bootstrap.servers = broker3-east:9092,broker4-east:9092 - - us-west->us-east.enabled = true - us-east->us-west.enabled = true + +```properties +# Bidirectional flow (two-way) between us-west and us-east clusters +clusters = us-west, us-east +us-west.bootstrap.servers = broker1-west:9092,broker2-west:9092 +Us-east.bootstrap.servers = broker3-east:9092,broker4-east:9092 + +us-west->us-east.enabled = true +us-east->us-west.enabled = true +``` _Note on preventing replication "loops" (where topics will be originally replicated from A to B, then the replicated topics will be replicated yet again from B to A, and so forth)_ : As long as you define the above flows in the same MirrorMaker configuration file, you do not need to explicitly add `topics.exclude` settings to prevent replication loops between the two clusters. @@ -341,48 +360,50 @@ _Note on preventing replication "loops" (where topics will be originally replica Let's put all the information from the previous sections together in a larger example. Imagine there are three data centers (west, east, north), with two Kafka clusters in each data center (e.g., `west-1`, `west-2`). The example in this section shows how to configure MirrorMaker (1) for Active/Active replication within each data center, as well as (2) for Cross Data Center Replication (XDCR). First, define the source and target clusters along with their replication flows in the configuration: - - - # Basic settings - clusters: west-1, west-2, east-1, east-2, north-1, north-2 - west-1.bootstrap.servers = ... - west-2.bootstrap.servers = ... - east-1.bootstrap.servers = ... - east-2.bootstrap.servers = ... - north-1.bootstrap.servers = ... - north-2.bootstrap.servers = ... - - # Replication flows for Active/Active in West DC - west-1->west-2.enabled = true - west-2->west-1.enabled = true - - # Replication flows for Active/Active in East DC - east-1->east-2.enabled = true - east-2->east-1.enabled = true - - # Replication flows for Active/Active in North DC - north-1->north-2.enabled = true - north-2->north-1.enabled = true - - # Replication flows for XDCR via west-1, east-1, north-1 - west-1->east-1.enabled = true - west-1->north-1.enabled = true - east-1->west-1.enabled = true - east-1->north-1.enabled = true - north-1->west-1.enabled = true - north-1->east-1.enabled = true + +```properties +# Basic settings +clusters: west-1, west-2, east-1, east-2, north-1, north-2 +west-1.bootstrap.servers = ... +west-2.bootstrap.servers = ... +east-1.bootstrap.servers = ... +east-2.bootstrap.servers = ... +north-1.bootstrap.servers = ... +north-2.bootstrap.servers = ... + +# Replication flows for Active/Active in West DC +west-1->west-2.enabled = true +west-2->west-1.enabled = true + +# Replication flows for Active/Active in East DC +east-1->east-2.enabled = true +east-2->east-1.enabled = true + +# Replication flows for Active/Active in North DC +north-1->north-2.enabled = true +north-2->north-1.enabled = true + +# Replication flows for XDCR via west-1, east-1, north-1 +west-1->east-1.enabled = true +west-1->north-1.enabled = true +east-1->west-1.enabled = true +east-1->north-1.enabled = true +north-1->west-1.enabled = true +north-1->east-1.enabled = true +``` Then, in each data center, launch one or more MirrorMaker as follows: - - - # In West DC: - $ bin/connect-mirror-maker.sh connect-mirror-maker.properties --clusters west-1 west-2 - - # In East DC: - $ bin/connect-mirror-maker.sh connect-mirror-maker.properties --clusters east-1 east-2 - - # In North DC: - $ bin/connect-mirror-maker.sh connect-mirror-maker.properties --clusters north-1 north-2 + +```bash +# In West DC: +$ bin/connect-mirror-maker.sh connect-mirror-maker.properties --clusters west-1 west-2 + +# In East DC: +$ bin/connect-mirror-maker.sh connect-mirror-maker.properties --clusters east-1 east-2 + +# In North DC: +$ bin/connect-mirror-maker.sh connect-mirror-maker.properties --clusters north-1 north-2 +``` With this configuration, records produced to any cluster will be replicated within the data center, as well as across to other data centers. By providing the `--clusters` parameter, we ensure that each MirrorMaker process produces data to nearby clusters only. @@ -393,28 +414,30 @@ _Note:_ The `--clusters` parameter is, technically, not required here. MirrorMak You can run as few or as many MirrorMaker processes (think: nodes, servers) as needed. Because MirrorMaker is based on Kafka Connect, MirrorMaker processes that are configured to replicate the same Kafka clusters run in a distributed setup: They will find each other, share configuration (see section below), load balance their work, and so on. If, for example, you want to increase the throughput of replication flows, one option is to run additional MirrorMaker processes in parallel. To start a MirrorMaker process, run the command: - - - $ bin/connect-mirror-maker.sh connect-mirror-maker.properties + +```bash +$ bin/connect-mirror-maker.sh connect-mirror-maker.properties +``` After startup, it may take a few minutes until a MirrorMaker process first begins to replicate data. Optionally, as described previously, you can set the parameter `--clusters` to ensure that the MirrorMaker process produces data to nearby clusters only. - - - # Note: The cluster alias us-west must be defined in the configuration file - $ bin/connect-mirror-maker.sh connect-mirror-maker.properties \ - --clusters us-west - + +```bash +# Note: The cluster alias us-west must be defined in the configuration file +$ bin/connect-mirror-maker.sh connect-mirror-maker.properties \ + --clusters us-west +``` _Note when testing replication of consumer groups:_ By default, MirrorMaker does not replicate consumer groups created by the kafka-console-consumer.sh tool, which you might use to test your MirrorMaker setup on the command line. If you do want to replicate these consumer groups as well, set the `groups.exclude` configuration accordingly (default: `groups.exclude = console-consumer-.*, connect-.*, __.*`). Remember to update the configuration again once you completed your testing. ## Stopping Geo-Replication You can stop a running MirrorMaker process by sending a SIGTERM signal with the command: - - - $ kill + +```bash +$ kill +``` ## Applying Configuration Changes @@ -439,27 +462,28 @@ Metrics are tracked for each replicated topic. The source cluster can be inferre The following metrics are emitted: - - - # MBean: kafka.connect.mirror:type=MirrorSourceConnector,target=([-.w]+),topic=([-.w]+),partition=([0-9]+) - record-count # number of records replicated source -> target - record-rate # average number of records/sec in replicated records - record-age-ms # age of records when they are replicated - record-age-ms-min - record-age-ms-max - record-age-ms-avg - replication-latency-ms # time it takes records to propagate source->target - replication-latency-ms-min - replication-latency-ms-max - replication-latency-ms-avg - byte-rate # average number of bytes/sec in replicated records - byte-count # number of bytes replicated source -> target - - # MBean: kafka.connect.mirror:type=MirrorCheckpointConnector,source=([-.w]+),target=([-.w]+),group=([-.w]+),topic=([-.w]+),partition=([0-9]+) - - checkpoint-latency-ms # time it takes to replicate consumer offsets - checkpoint-latency-ms-min - checkpoint-latency-ms-max - checkpoint-latency-ms-avg + +```text +# MBean: kafka.connect.mirror:type=MirrorSourceConnector,target=([-.w]+),topic=([-.w]+),partition=([0-9]+) +record-count # number of records replicated source -> target +record-rate # average number of records/sec in replicated records +record-age-ms # age of records when they are replicated +record-age-ms-min +record-age-ms-max +record-age-ms-avg +replication-latency-ms # time it takes records to propagate source->target +replication-latency-ms-min +replication-latency-ms-max +replication-latency-ms-avg +byte-rate # average number of bytes/sec in replicated records +byte-count # number of bytes replicated source -> target + +# MBean: kafka.connect.mirror:type=MirrorCheckpointConnector,source=([-.w]+),target=([-.w]+),group=([-.w]+),topic=([-.w]+),partition=([0-9]+) + +checkpoint-latency-ms # time it takes to replicate consumer offsets +checkpoint-latency-ms-min +checkpoint-latency-ms-max +checkpoint-latency-ms-avg +``` These metrics do not differentiate between created-at and log-append timestamps. diff --git a/docs/operations/hardware-and-os.md b/docs/operations/hardware-and-os.md index 3e08cba33d7a8..e510e7f2d916b 100644 --- a/docs/operations/hardware-and-os.md +++ b/docs/operations/hardware-and-os.md @@ -76,9 +76,10 @@ In Linux, data written to the filesystem is maintained in [pagecache](https://en Pdflush has a configurable policy that controls how much dirty data can be maintained in cache and for how long before it must be written back to disk. This policy is described [here](https://web.archive.org/web/20160518040713/http://www.westnet.com/~gsmith/content/linux-pdflush.htm). When Pdflush cannot keep up with the rate of data being written it will eventually cause the writing process to block incurring latency in the writes to slow down the accumulation of data. You can see the current state of OS memory usage by doing - - - $ cat /proc/meminfo + +```bash +$ cat /proc/meminfo +``` The meaning of these values are described in the link above. @@ -129,22 +130,25 @@ EXT4 is a serviceable choice of filesystem for the Kafka data directories, howev When Kafka is configured to use KRaft, the controllers store the cluster metadata in the directory specified in `metadata.log.dir` \-- or the first log directory, if `metadata.log.dir` is not configured. See the documentation for `metadata.log.dir` for details. If the data in the cluster metadata directory is lost either because of hardware failure or the hardware needs to be replaced, care should be taken when provisioning the new controller node. The new controller node should not be formatted and started until the majority of the controllers have all of the committed data. To determine if the majority of the controllers have the committed data, run the kafka-metadata-quorum.sh tool to describe the replication status: - - - $ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication - NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status - 1 dDo1k_pRSD-VmReEpu383g 966 0 1732367153528 1732367153528 Leader - 2 wQWaQMJYpcifUPMBGeRHqg 966 0 1732367153304 1732367153304 Observer - ... ... ... ... ... ... + +```bash +$ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication +NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status +1 dDo1k_pRSD-VmReEpu383g 966 0 1732367153528 1732367153528 Leader +2 wQWaQMJYpcifUPMBGeRHqg 966 0 1732367153304 1732367153304 Observer +... ... ... ... ... ... +``` Check and wait until the `Lag` is small for a majority of the controllers. If the leader's end offset is not increasing, you can wait until the lag is 0 for a majority; otherwise, you can pick the latest leader end offset and wait until all replicas have reached it. Check and wait until the `LastFetchTimestamp` and `LastCaughtUpTimestamp` are close to each other for the majority of the controllers. At this point it is safer to format the controller's metadata log directory. This can be done by running the kafka-storage.sh command. - - - $ bin/kafka-storage.sh format --cluster-id uuid --config config/server.properties + +```bash +$ bin/kafka-storage.sh format --cluster-id uuid --config config/server.properties +``` It is possible for the `bin/kafka-storage.sh format` command above to fail with a message like `Log directory ... is already formatted`. This can happen when combined mode is used and only the metadata log directory was lost but not the others. In that case and only in that case, can you run the `bin/kafka-storage.sh format` command with the `--ignore-formatted` option. Start the KRaft controller after formatting the log directories. - - - $ bin/kafka-server-start.sh config/server.properties + +```bash +$ bin/kafka-server-start.sh config/server.properties +``` diff --git a/docs/operations/java-version.md b/docs/operations/java-version.md index 9cf05519c428a..d7edaefe195b7 100644 --- a/docs/operations/java-version.md +++ b/docs/operations/java-version.md @@ -31,11 +31,12 @@ Java 17, Java 21, and Java 25 are fully supported while Java 11 is supported for We generally recommend running Apache Kafka with the most recent LTS release (Java 25 at the time of writing) for performance, efficiency and support reasons. From a security perspective, we recommend the latest released patch version as older versions typically have disclosed security vulnerabilities. Typical arguments for running Kafka with OpenJDK-based Java implementations (including Oracle JDK) are: - - - -Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC - -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M - -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+ExplicitGCInvokesConcurrent + +```bash +-Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC +-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M +-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+ExplicitGCInvokesConcurrent +``` For reference, here are the stats for one of LinkedIn's busiest clusters (at peak) that uses said Java arguments: diff --git a/docs/operations/kraft.md b/docs/operations/kraft.md index 5281dc927039b..7d95e0824e2c3 100644 --- a/docs/operations/kraft.md +++ b/docs/operations/kraft.md @@ -47,18 +47,20 @@ In KRaft mode, specific Kafka servers are selected to be controllers. The server A Kafka admin will typically select 3 or 5 servers for this role, depending on factors like cost and the number of concurrent failures your system should withstand without availability impact. A majority of the controllers must be alive in order to maintain availability. With 3 controllers, the cluster can tolerate 1 controller failure; with 5 controllers, the cluster can tolerate 2 controller failures. All of the servers in a Kafka cluster discover the active controller using the `controller.quorum.bootstrap.servers` property. All the controllers should be enumerated in this property. Each controller is identified with their `host` and `port` information. For example: - - - controller.quorum.bootstrap.servers=host1:port1,host2:port2,host3:port3 + +```properties +controller.quorum.bootstrap.servers=host1:port1,host2:port2,host3:port3 +``` If a Kafka cluster has 3 controllers named controller1, controller2 and controller3, then controller1 may have the following configuration: - - - process.roles=controller - node.id=1 - listeners=CONTROLLER://controller1.example.com:9093 - controller.quorum.bootstrap.servers=controller1.example.com:9093,controller2.example.com:9093,controller3.example.com:9093 - controller.listener.names=CONTROLLER + +```properties +process.roles=controller +node.id=1 +listeners=CONTROLLER://controller1.example.com:9093 +controller.quorum.bootstrap.servers=controller1.example.com:9093,controller2.example.com:9093,controller3.example.com:9093 +controller.listener.names=CONTROLLER +``` Every broker and controller must set the `controller.quorum.bootstrap.servers` property. @@ -71,36 +73,40 @@ This feature upgrade is done by upgrading the KRaft feature version and updating ### Describe KRaft Version Dynamic controller cluster was added in `kraft.version=1` or `release-version 4.1`. To determine which kraft feature version the cluster is using you can execute the following CLI command: - - - $ bin/kafka-features.sh --bootstrap-controller localhost:9093 describe - ... - Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 0 Epoch: 7 - Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 4.0-IV3 FinalizedVersionLevel: 4.0-IV3 Epoch: 7 + +```bash +$ bin/kafka-features.sh --bootstrap-controller localhost:9093 describe +... +Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 0 Epoch: 7 +Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 4.0-IV3 FinalizedVersionLevel: 4.0-IV3 Epoch: 7 +``` If the `FinalizedVersionLevel` for `Feature: kraft.version` is `0`, the version needs to be upgraded to at least `1` to support a dynamic controller cluster. ### Upgrade KRaft Version The KRaft feature version can be upgraded to support dynamic controller clusters by using the `kafka-feature` CLI command. To upgrade all of the feature versions to the latest version: - - - $ bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --release-version 4.1 + +```bash +$ bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --release-version 4.1 +``` To upgrade just the KRaft feature version: - - - $ bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature kraft.version=1 + +```bash +$ bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature kraft.version=1 +``` ### Update KRaft Config KRaft version 1 deprecated the `controller.quorum.voters` property and added the `controller.quorum.bootstrap.servers` property. After checking that the KRaft version has been successfully upgraded to at least version `1`, remove the `controller.quorum.voters` property and add the `controller.quorum.bootstrap.servers` to all of the nodes (controllers and brokers) in the cluster. - - - process.roles=... - node.id=... - controller.quorum.bootstrap.servers=controller1.example.com:9093,controller2.example.com:9093,controller3.example.com:9093 - controller.listener.names=CONTROLLER + +```properties +process.roles=... +node.id=... +controller.quorum.bootstrap.servers=controller1.example.com:9093,controller2.example.com:9093,controller3.example.com:9093 +controller.listener.names=CONTROLLER +``` ## Provisioning Nodes @@ -111,35 +117,38 @@ This is different from how Kafka has operated in the past. Previously, Kafka wou ### Bootstrap a Standalone Controller The recommended method for creating a new KRaft controller cluster is to bootstrap it with one voter and dynamically add the rest of the controllers. Bootstrapping the first controller can be done with the following CLI command: - - - $ bin/kafka-storage.sh format --cluster-id --standalone --config config/controller.properties + +```bash +$ bin/kafka-storage.sh format --cluster-id --standalone --config config/controller.properties +``` This command will 1) create a meta.properties file in metadata.log.dir with a randomly generated directory.id, 2) create a snapshot at 00000000000000000000-0000000000.checkpoint with the necessary control records (KRaftVersionRecord and VotersRecord) to make this Kafka node the only voter for the quorum. ### Bootstrap with Multiple Controllers The KRaft cluster metadata partition can also be bootstrapped with more than one voter. This can be done by using the --initial-controllers flag: - - - CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" - CONTROLLER_0_UUID="$(bin/kafka-storage.sh random-uuid)" - CONTROLLER_1_UUID="$(bin/kafka-storage.sh random-uuid)" - CONTROLLER_2_UUID="$(bin/kafka-storage.sh random-uuid)" - - # In each controller execute - bin/kafka-storage.sh format --cluster-id ${CLUSTER_ID} \ - --initial-controllers "0@controller-0:1234:${CONTROLLER_0_UUID},1@controller-1:1234:${CONTROLLER_1_UUID},2@controller-2:1234:${CONTROLLER_2_UUID}" \ - --config config/controller.properties + +```bash +CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" +CONTROLLER_0_UUID="$(bin/kafka-storage.sh random-uuid)" +CONTROLLER_1_UUID="$(bin/kafka-storage.sh random-uuid)" +CONTROLLER_2_UUID="$(bin/kafka-storage.sh random-uuid)" + +# In each controller execute +bin/kafka-storage.sh format --cluster-id ${CLUSTER_ID} \ + --initial-controllers "0@controller-0:1234:${CONTROLLER_0_UUID},1@controller-1:1234:${CONTROLLER_1_UUID},2@controller-2:1234:${CONTROLLER_2_UUID}" \ + --config config/controller.properties +``` This command is similar to the standalone version but the snapshot at 00000000000000000000-0000000000.checkpoint will instead contain a VotersRecord that includes information for all of the controllers specified in --initial-controllers. It is important that the value of this flag is the same in all of the controllers with the same cluster id. In the replica description 0@controller-0:1234:3Db5QLSqSZieL3rJBUUegA, 0 is the replica id, 3Db5QLSqSZieL3rJBUUegA is the replica directory id, controller-0 is the replica's host and 1234 is the replica's port. ### Formatting Brokers and New Controllers When provisioning new broker and controller nodes that we want to add to an existing Kafka cluster, use the `kafka-storage.sh format` command with the --no-initial-controllers flag. - - - $ bin/kafka-storage.sh format --cluster-id --config config/server.properties --no-initial-controllers + +```bash +$ bin/kafka-storage.sh format --cluster-id --config config/server.properties --no-initial-controllers +``` ## Controller membership changes @@ -152,26 +161,30 @@ When using a dynamic quorum, `controller.quorum.voters` must not be set and `con When using a static quorum, the configuration file for each broker and controller must specify the IDs, hostnames, and ports of all controllers in `controller.quorum.voters`. If you are not sure whether you are using static or dynamic quorums, you can determine this by running something like the following: - - - $ bin/kafka-features.sh --bootstrap-controller localhost:9093 describe + +```bash +$ bin/kafka-features.sh --bootstrap-controller localhost:9093 describe +``` If the `kraft.version` field is level 0 or absent, you are using a static quorum. If it is 1 or above, you are using a dynamic quorum. For example, here is an example of a static quorum: - - - Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 0 Epoch: 5 - Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5 + +```text +Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 0 Epoch: 5 +Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5 +``` Here is another example of a static quorum: - - - Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.8-IV0 FinalizedVersionLevel: 3.8-IV0 Epoch: 5 + +```text +Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.8-IV0 FinalizedVersionLevel: 3.8-IV0 Epoch: 5 +``` Here is an example of a dynamic quorum: - - - Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 Epoch: 5 - Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5 + +```text +Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 Epoch: 5 +Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5 +``` The static versus dynamic nature of the quorum is determined at the time of formatting. Specifically, the quorum will be formatted as dynamic if `controller.quorum.voters` is **not** present, and one of --standalone, --initial-controllers, or --no-initial-controllers is set. If you have followed the instructions earlier in this document, you will get a dynamic quorum. @@ -180,85 +193,93 @@ Note: To migrate from static voter set to dynamic voter set, please refer to the ### Add New Controller If a dynamic controller cluster already exists, it can be expanded by first provisioning a new controller using the kafka-storage.sh tool and starting the controller. After starting the controller, the replication to the new controller can be monitored using the `bin/kafka-metadata-quorum.sh describe --replication` command. Once the new controller has caught up to the active controller, it can be added to the cluster using the `bin/kafka-metadata-quorum.sh add-controller` command. When using broker endpoints use the --bootstrap-server flag: - - - $ bin/kafka-metadata-quorum.sh --command-config config/controller.properties --bootstrap-server localhost:9092 add-controller + +```bash +$ bin/kafka-metadata-quorum.sh --command-config config/controller.properties --bootstrap-server localhost:9092 add-controller +``` When using controller endpoints use the --bootstrap-controller flag: - - - $ bin/kafka-metadata-quorum.sh --command-config config/controller.properties --bootstrap-controller localhost:9093 add-controller + +```bash +$ bin/kafka-metadata-quorum.sh --command-config config/controller.properties --bootstrap-controller localhost:9093 add-controller +``` Note that if there are any configs needed to be passed to the Admin Client, like the authentication configuration, please also include in the "controller.properties". ### Remove Controller If the dynamic controller cluster already exists, it can be shrunk using the `bin/kafka-metadata-quorum.sh remove-controller` command. Use the remove-controller command before shutting down the controller to have it removed from the quorum first. When using broker endpoints use the --bootstrap-server flag: - - - $ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 remove-controller --controller-id --controller-directory-id + +```bash +$ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 remove-controller --controller-id --controller-directory-id +``` When using controller endpoints use the --bootstrap-controller flag: - - - $ bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 remove-controller --controller-id --controller-directory-id + +```bash +$ bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 remove-controller --controller-id --controller-directory-id +``` ## Debugging ### Metadata Quorum Tool The kafka-metadata-quorum.sh tool can be used to describe the runtime state of the cluster metadata partition. For example, the following command displays a summary of the metadata quorum: - - - $ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status - ClusterId: fMCL8kv1SWm87L_Md-I2hg - LeaderId: 3002 - LeaderEpoch: 2 - HighWatermark: 10 - MaxFollowerLag: 0 - MaxFollowerLagTimeMs: -1 - CurrentVoters: [{"id": 3000, "directoryId": "ILZ5MPTeRWakmJu99uBJCA", "endpoints": ["CONTROLLER://localhost:9093"]}, - {"id": 3001, "directoryId": "b-DwmhtOheTqZzPoh52kfA", "endpoints": ["CONTROLLER://localhost:9094"]}, - {"id": 3002, "directoryId": "g42deArWBTRM5A1yuVpMCg", "endpoints": ["CONTROLLER://localhost:9095"]}] - CurrentObservers: [{"id": 0, "directoryId": "3Db5QLSqSZieL3rJBUUegA"}, - {"id": 1, "directoryId": "UegA3Db5QLSqSZieL3rJBU"}, - {"id": 2, "directoryId": "L3rJBUUegA3Db5QLSqSZie"}] + +```bash +$ bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status +ClusterId: fMCL8kv1SWm87L_Md-I2hg +LeaderId: 3002 +LeaderEpoch: 2 +HighWatermark: 10 +MaxFollowerLag: 0 +MaxFollowerLagTimeMs: -1 +CurrentVoters: [{"id": 3000, "directoryId": "ILZ5MPTeRWakmJu99uBJCA", "endpoints": ["CONTROLLER://localhost:9093"]}, + {"id": 3001, "directoryId": "b-DwmhtOheTqZzPoh52kfA", "endpoints": ["CONTROLLER://localhost:9094"]}, + {"id": 3002, "directoryId": "g42deArWBTRM5A1yuVpMCg", "endpoints": ["CONTROLLER://localhost:9095"]}] +CurrentObservers: [{"id": 0, "directoryId": "3Db5QLSqSZieL3rJBUUegA"}, + {"id": 1, "directoryId": "UegA3Db5QLSqSZieL3rJBU"}, + {"id": 2, "directoryId": "L3rJBUUegA3Db5QLSqSZie"}] +``` ### Dump Log Tool The kafka-dump-log.sh tool can be used to debug the log segments and snapshots for the cluster metadata directory. The tool will scan the provided files and decode the metadata records. For example, this command decodes and prints the records in the first log segment: - - - $ bin/kafka-dump-log.sh --cluster-metadata-decoder --files metadata_log_dir/__cluster_metadata-0/00000000000000000000.log + +```bash +$ bin/kafka-dump-log.sh --cluster-metadata-decoder --files metadata_log_dir/__cluster_metadata-0/00000000000000000000.log +``` This command decodes and prints the records in a cluster metadata snapshot: - - - $ bin/kafka-dump-log.sh --cluster-metadata-decoder --files metadata_log_dir/__cluster_metadata-0/00000000000000000100-0000000001.checkpoint + +```bash +$ bin/kafka-dump-log.sh --cluster-metadata-decoder --files metadata_log_dir/__cluster_metadata-0/00000000000000000100-0000000001.checkpoint +``` ### Metadata Shell The kafka-metadata-shell.sh tool can be used to interactively inspect the state of the cluster metadata partition: - - - $ bin/kafka-metadata-shell.sh --snapshot metadata_log_dir/__cluster_metadata-0/00000000000000007228-0000000001.checkpoint - >> ls / - brokers local metadataQuorum topicIds topics - >> ls /topics - foo - >> cat /topics/foo/0/data - { - "partitionId" : 0, - "topicId" : "5zoAlv-xEh9xRANKXt1Lbg", - "replicas" : [ 1 ], - "isr" : [ 1 ], - "removingReplicas" : null, - "addingReplicas" : null, - "leader" : 1, - "leaderEpoch" : 0, - "partitionEpoch" : 0 - } - >> exit + +```bash +$ bin/kafka-metadata-shell.sh --snapshot metadata_log_dir/__cluster_metadata-0/00000000000000007228-0000000001.checkpoint +>> ls / +brokers local metadataQuorum topicIds topics +>> ls /topics +foo +>> cat /topics/foo/0/data +{ + "partitionId" : 0, + "topicId" : "5zoAlv-xEh9xRANKXt1Lbg", + "replicas" : [ 1 ], + "isr" : [ 1 ], + "removingReplicas" : null, + "addingReplicas" : null, + "leader" : 1, + "leaderEpoch" : 0, + "partitionEpoch" : 0 +} +>> exit +``` Note: `00000000000000000000-0000000000.checkpoint` does not contain cluster metadata. Use a valid snapshot file when examining metadata with the `kafka-metadata-shell.sh` tool. diff --git a/docs/operations/monitoring.md b/docs/operations/monitoring.md index 9ecaa2ed51c2e..c7f22f3b1666f 100644 --- a/docs/operations/monitoring.md +++ b/docs/operations/monitoring.md @@ -3884,9 +3884,10 @@ A Kafka Streams instance contains all the producer and consumer metrics as well Note that the metrics have a 4-layer hierarchy. At the top level there are client-level metrics for each started Kafka Streams client. Each client has stream threads, with their own metrics. Each stream thread has tasks, with their own metrics. Each task has a number of processor nodes, with their own metrics. Each task also has a number of state stores and record caches, all with their own metrics. Use the following configuration option to specify which metrics you want collected: - - - metrics.recording.level="info" + +```properties +metrics.recording.level="info" +``` ### Client Metrics diff --git a/docs/operations/multi-tenancy.md b/docs/operations/multi-tenancy.md index ac07b1926b793..e0dfa6b828f35 100644 --- a/docs/operations/multi-tenancy.md +++ b/docs/operations/multi-tenancy.md @@ -97,14 +97,15 @@ Security settings for Kafka fall into three main categories, which are similar t When securing a multi-tenant Kafka environment, the most common administrative task is the third category (authorization), i.e., managing the user/client permissions that grant or deny access to certain topics and thus to the data stored by users within a cluster. This task is performed predominantly through the setting of access control lists (ACLs). Here, administrators of multi-tenant environments in particular benefit from putting a hierarchical topic naming structure in place as described in a previous section, because they can conveniently control access to topics through prefixed ACLs (`--resource-pattern-type Prefixed`). This significantly minimizes the administrative overhead of securing topics in multi-tenant environments: administrators can make their own trade-offs between higher developer convenience (more lenient permissions, using fewer and broader ACLs) vs. tighter security (more stringent permissions, using more and narrower ACLs). In the following example, user Alice-a new member of ACME corporation's InfoSec team-is granted write permissions to all topics whose names start with "acme.infosec.", such as "acme.infosec.telemetry.logins" and "acme.infosec.syslogs.events". - - - # Grant permissions to user Alice - $ bin/kafka-acls.sh \ - --bootstrap-server localhost:9092 \ - --add --allow-principal User:Alice \ - --producer \ - --resource-pattern-type prefixed --topic acme.infosec. + +```bash +# Grant permissions to user Alice +$ bin/kafka-acls.sh \ + --bootstrap-server localhost:9092 \ + --add --allow-principal User:Alice \ + --producer \ + --resource-pattern-type prefixed --topic acme.infosec. +``` You can similarly use this approach to isolate different customers on the same shared cluster. diff --git a/docs/operations/tiered-storage.md b/docs/operations/tiered-storage.md index 5da145952798a..8fdd092fb29a6 100644 --- a/docs/operations/tiered-storage.md +++ b/docs/operations/tiered-storage.md @@ -59,102 +59,111 @@ The configuration prefixed with `local` are to specify the time/size the "local" Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation. To have a preview of the tiered storage feature, the [LocalTieredStorage](https://github.com/apache/kafka/blob/trunk/storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java) implemented for integration test can be used, which will create a temporary directory in local storage to simulate the remote storage. -To adopt the `LocalTieredStorage`, the test library needs to be built locally - - - # please checkout to the specific version tag you're using before building it - # ex: `git checkout 4.3.0` - $ ./gradlew clean :storage:testJar +To adopt the `LocalTieredStorage`, the test library needs to be built locally: + +```bash +# please checkout to the specific version tag you're using before building it +# ex: `git checkout 4.3.0` +$ ./gradlew clean :storage:testJar +``` After build successfully, there should be a `kafka-storage-x.x.x-test.jar` file under `storage/build/libs`. Next, setting configurations in the broker side to enable tiered storage feature. - - - # Sample KRaft broker server.properties listening on PLAINTEXT://:9092 - remote.log.storage.system.enable=true - - # Setting the listener for the clients in RemoteLogMetadataManager to talk to the brokers. - remote.log.metadata.manager.listener.name=PLAINTEXT - - # Please provide the implementation info for remoteStorageManager. - # This is the mandatory configuration for tiered storage. - # Here, we use the `LocalTieredStorage` built above. - remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage - remote.log.storage.manager.class.path=/PATH/TO/kafka-storage-4.3.0-test.jar - - # These 2 prefix are default values, but customizable - remote.log.storage.manager.impl.prefix=rsm.config. - remote.log.metadata.manager.impl.prefix=rlmm.config. - - # Configure the directory used for `LocalTieredStorage` - # Note, please make sure the brokers need to have access to this directory - rsm.config.dir=/tmp/kafka-remote-storage - - # For single broker cluster, set this to 1. Default is 3 for clusters with 3 or more brokers. - rlmm.config.remote.log.metadata.topic.replication.factor=1 - - # The minimum number of replicas that must acknowledge a write to remote log metadata topic. - # Default value is 2. For single broker cluster (replication factor = 1), set this to 1. - rlmm.config.remote.log.metadata.topic.min.isr=1 - - # Try to speed up the log retention check interval for testing - log.retention.check.interval.ms=1000 + +```properties +# Sample KRaft broker server.properties listening on PLAINTEXT://:9092 +remote.log.storage.system.enable=true + +# Setting the listener for the clients in RemoteLogMetadataManager to talk to the brokers. +remote.log.metadata.manager.listener.name=PLAINTEXT + +# Please provide the implementation info for remoteStorageManager. +# This is the mandatory configuration for tiered storage. +# Here, we use the `LocalTieredStorage` built above. +remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage +remote.log.storage.manager.class.path=/PATH/TO/kafka-storage-4.3.0-test.jar + +# These 2 prefix are default values, but customizable +remote.log.storage.manager.impl.prefix=rsm.config. +remote.log.metadata.manager.impl.prefix=rlmm.config. + +# Configure the directory used for `LocalTieredStorage` +# Note, please make sure the brokers need to have access to this directory +rsm.config.dir=/tmp/kafka-remote-storage + +# For single broker cluster, set this to 1. Default is 3 for clusters with 3 or more brokers. +rlmm.config.remote.log.metadata.topic.replication.factor=1 + +# The minimum number of replicas that must acknowledge a write to remote log metadata topic. +# Default value is 2. For single broker cluster (replication factor = 1), set this to 1. +rlmm.config.remote.log.metadata.topic.min.isr=1 + +# Try to speed up the log retention check interval for testing +log.retention.check.interval.ms=1000 +``` Following quick start guide to start up the kafka environment. Then, create a topic with tiered storage enabled with configs: - - - # remote.storage.enable=true -> enables tiered storage on the topic - # local.retention.ms=1000 -> The number of milliseconds to keep the local log segment before it gets deleted. - # Note that a local log segment is eligible for deletion only after it gets uploaded to remote. - # retention.ms=3600000 -> when segments exceed this time, the segments in remote storage will be deleted - # segment.bytes=1048576 -> for test only, to speed up the log segment rolling interval - # file.delete.delay.ms=10000 -> for test only, to speed up the local-log segment file delete delay - - $ bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 \ - --config remote.storage.enable=true --config local.retention.ms=1000 --config retention.ms=3600000 \ - --config segment.bytes=1048576 --config file.delete.delay.ms=1000 + +```bash +# remote.storage.enable=true -> enables tiered storage on the topic +# local.retention.ms=1000 -> The number of milliseconds to keep the local log segment before it gets deleted. +# Note that a local log segment is eligible for deletion only after it gets uploaded to remote. +# retention.ms=3600000 -> when segments exceed this time, the segments in remote storage will be deleted +# segment.bytes=1048576 -> for test only, to speed up the log segment rolling interval +# file.delete.delay.ms=10000 -> for test only, to speed up the local-log segment file delete delay + +$ bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 \ +--config remote.storage.enable=true --config local.retention.ms=1000 --config retention.ms=3600000 \ +--config segment.bytes=1048576 --config file.delete.delay.ms=1000 +``` Try to send messages to the `tieredTopic` topic to roll the log segment: - - - $ bin/kafka-producer-perf-test.sh --bootstrap-server localhost:9092 --topic tieredTopic --num-records 1200 --record-size 1024 --throughput -1 + +```bash +$ bin/kafka-producer-perf-test.sh --bootstrap-server localhost:9092 --topic tieredTopic --num-records 1200 --record-size 1024 --throughput -1 +``` Then, after the active segment is rolled, the old segment should be moved to the remote storage and get deleted. This can be verified by checking the remote log directory configured above. For example: - - - $ ls /tmp/kafka-remote-storage/kafka-tiered-storage/tieredTopic-0-jF8s79t9SrG_PNqlwv7bAA - 00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.index - 00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.snapshot - 00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.leader_epoch_checkpoint - 00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.timeindex - 00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.log + +```bash +$ ls /tmp/kafka-remote-storage/kafka-tiered-storage/tieredTopic-0-jF8s79t9SrG_PNqlwv7bAA +00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.index +00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.snapshot +00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.leader_epoch_checkpoint +00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.timeindex +00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.log +``` Lastly, we can try to consume some data from the beginning and print offset number, to make sure it will successfully fetch offset 0 from the remote storage. - - - $ bin/kafka-console-consumer.sh --topic tieredTopic --from-beginning --max-messages 1 --bootstrap-server localhost:9092 --formatter-property print.offset=true + +```bash +$ bin/kafka-console-consumer.sh --topic tieredTopic --from-beginning --max-messages 1 --bootstrap-server localhost:9092 --formatter-property print.offset=true +``` In KRaft mode, you can disable tiered storage at the topic level, to make the remote logs as read-only logs, or completely delete all remote logs. If you want to let the remote logs become read-only and no more local logs copied to the remote storage, you can set `remote.storage.enable=true,remote.log.copy.disable=true` to the topic. Note: You also need to set `local.retention.ms` and `local.retention.bytes` to the same value as `retention.ms` and `retention.bytes`, or set to "-2". This is because after disabling remote log copy, the local retention policies will not be applied anymore, and that might confuse users and cause unexpected disk full. - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 \ - --alter --entity-type topics --entity-name tieredTopic \ - --add-config 'remote.storage.enable=true,remote.log.copy.disable=true,local.retention.ms=-2,local.retention.bytes=-2' + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 \ + --alter --entity-type topics --entity-name tieredTopic \ + --add-config 'remote.storage.enable=true,remote.log.copy.disable=true,local.retention.ms=-2,local.retention.bytes=-2' +``` If you want to completely disable tiered storage at the topic level with all remote logs deleted, you can set `remote.storage.enable=false,remote.log.delete.on.disable=true` to the topic. - - - $ bin/kafka-configs.sh --bootstrap-server localhost:9092 \ - --alter --entity-type topics --entity-name tieredTopic \ - --add-config 'remote.storage.enable=false,remote.log.delete.on.disable=true' + +```bash +$ bin/kafka-configs.sh --bootstrap-server localhost:9092 \ + --alter --entity-type topics --entity-name tieredTopic \ + --add-config 'remote.storage.enable=false,remote.log.delete.on.disable=true' +``` You can also re-enable tiered storage feature at the topic level. Please note, if you want to disable tiered storage at the cluster level, you should delete the tiered storage enabled topics explicitly. Attempting to disable tiered storage at the cluster level without deleting the topics using tiered storage will result in an exception during startup. - - - $ bin/kafka-topics.sh --delete --topic tieredTopic --bootstrap-server localhost:9092 + +```bash +$ bin/kafka-topics.sh --delete --topic tieredTopic --bootstrap-server localhost:9092 +``` After topics are deleted, you're safe to set `remote.log.storage.system.enable=false` in the broker configuration.