Recommendation: Multiple Parallel Consumer instances (different group.id) in same app for workload isolation and fan-out control

[mayurjaiswal9]

Hi PC team,

We are using Confluent Parallel Consumer in a service that consumes multiple topics. We have two main workloads:

1. Command topic: each command produces ~6 events downstream
2. Event topic: consumes those events and does heavier / higher-latency processing (~300ms avg)

We currently run these through the same application and have experimented with sharing a single PC (or shared resources). We are seeing a few problems:

• When we increase concurrency / threads on the command consumer, it processes commands faster, which multiplies event production, and the event consumer falls behind even more (fan-out amplification).
• With shared execution / queues, one heavy or slow workload can starve the other (interference).
• From an ops perspective, observability is poor: lag and backpressure look “global” and it’s hard to see which workload is actually the bottleneck.

We are considering changing the architecture to:

• Run two separate Parallel Consumer instances inside the same app
• Use different group.ids
• Give each PC its own executor / concurrency / limits
• One PC dedicated to Commands, one PC dedicated to Events

This would give us:

• Kafka-level isolation (separate groups, offsets, rebalances)
• Execution isolation (separate queues / thread pools)
• Better observability (we can see which stage is lagging)
• Ability to rate-limit the command stage based on event stage capacity

Questions:

1. Is running multiple PC instances with different group IDs in the same JVM/app a recommended / supported pattern?
2. Are there any pitfalls or anti-patterns we should be aware of with this approach?
3. Is there any guidance in PC around workload isolation vs shared PCs for multi-topic, mixed-latency systems?
4. In fan-out pipelines (1 → N events), is it generally better to treat each stage as a separate consumer group / PC?

We’re not asking about partitions or just “more threads” — we’re specifically trying to avoid:

• Cross-topic interference
• Uncontrolled fan-out amplification
• And regain predictable backpressure and observability per stage.

Would appreciate guidance on whether this is the right architectural direction with Parallel Consumer.

Thanks!