Improve Kafka Table Engine documentation

[morsapaes]

Something I had stashed for a while, but never finished. Needs a few more iterations, which I hope I'll finally get to over the holidays.

[vercel]

@morsapaes is attempting to deploy a commit to the ClickHouse Team on Vercel.

A member of the Team first needs to authorize it.

[randomizedcoder]

Great! Thanks for the update. - We should mention this thread started in #3666

It would be great to mention the batching that Kafka engine performs. It's now been many months since I looked at this, but from memory it batches for 15 seconds by default, so the inserts do happen in batching. This is why, although protobufList doesn't work, and doesn't have test automation, this isn't really a problem, given the batching.

It would also be good to mention that Kafka schema registry is unsupported.

[morsapaes]

@randomizedcoder, thanks for the additional feedback. Kafka Schema Registry is supported, at least for Avro:

CREATE TABLE default.test_csr
(
    `side` String COMMENT 'A simulated trade side (buy or sell or short)',
    `quantity` Int32 COMMENT 'A simulated random quantity of the trade',
    `symbol` String COMMENT 'Simulated stock symbols',
    `price` Int32 COMMENT 'A simulated random trade price in pennies',
    `account` String COMMENT 'Simulated accounts assigned to the trade',
    `userid` String COMMENT 'The simulated user who executed the trade'
)
ENGINE = Kafka
SETTINGS 
    kafka_broker_list = 'XX.eu-west-1.aws.confluent.cloud:9092',
    kafka_topic_list = 'stock-trades',
    kafka_group_name = 'ch',
    kafka_format = 'AvroConfluent',
    format_avro_schema_registry_url = 'https://XX@XX.us-east-2.aws.confluent.cloud',
    kafka_security_protocol = 'sasl_ssl',
    kafka_sasl_mechanism = 'PLAIN',
    kafka_sasl_username = 'XXX',
    kafka_sasl_password = '[HIDDEN]';

I made the edits to include this but think the changes are lost in one of my local stashes!

[randomizedcoder]

Hmmm. Are you sure? I remember digging through the Clickhouse kafka code, and it's definitely not there for protobuf. Would be great if it does work for Avro.

[morsapaes]

I tested it and we have customers using it, so yes — the sample above is working DDL to integrate with a schema registry for Avro. Support for Confluent Protobuf was implemented in ClickHouse#76623, but schema registry support is still an open task ClickHouse #81103.

[randomizedcoder]

Awesome! Kafka is getting some love! 🚀

ProtobufList still untested
ClickHouse/ClickHouse#78746 (comment)