Allow specifying starting offsets via the Kafka source options.

[yruslan]

Summary by CodeRabbit

• New Features
  • The Kafka Avro source now supports configurable default starting and ending offsets via configuration, allowing deployments to override the prior fixed defaults (earliest/latest) for more flexible data consumption behavior.

[coderabbitai]

Warning

Rate limit exceeded

@yruslan has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 15 minutes and 27 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8a2cbe29-5123-4156-a0e6-5f31099073b3

📥 Commits

Reviewing files that changed from the base of the PR and between f0efffa and f60ef75.

📒 Files selected for processing (1)

• pramen/extras/src/main/scala/za/co/absa/pramen/extras/source/KafkaAvroSource.scala

Walkthrough

KafkaAvroSource now reads default offset bounds from kafkaAvroConfig.extraOptions using keys "kafka.startingOffsets" and "kafka.endingOffsets" when explicit offsets are not provided, falling back to "earliest" and "latest" respectively.

Changes

Cohort / File(s)	Summary
Kafka source offset defaults pramen/extras/src/main/scala/za/co/absa/pramen/extras/source/KafkaAvroSource.scala	When offsetFromOpt/offsetToOpt are None, read "kafka.startingOffsets" and "kafka.endingOffsets" from extraOptions (defaults: "earliest" / "latest") instead of hardcoding those defaults.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 A nibble here on offset trails,
Defaults now read from tiny config tales.
Earliest or latest, softly guessed,
Extra options choose the rest. 🎋

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The pull request title focuses on starting offsets but the changeset also modifies ending offsets configuration.	Consider whether the title should mention both starting and ending offsets, or clarify if ending offsets changes are a secondary concern.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feature/allow-specifying-starging-offsets

📝 Coding Plan for PR comments

• Generate coding plan

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@pramen/extras/src/main/scala/za/co/absa/pramen/extras/source/KafkaAvroSource.scala`:
- Around line 171-172: The lookup in KafkaAvroSource.scala (case None branch)
uses the wrong key and therefore always defaults to "earliest"; change the
lookup against kafkaAvroConfig.extraOptions to use the prefixed key
"kafka.startingOffsets" (but keep the returned Map key as "startingOffsets") so
user-configured kafka.startingOffsets is respected when building the Map for
startingOffsets.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6d2bff03-6937-40e8-8dde-74457dc117cc

📥 Commits

Reviewing files that changed from the base of the PR and between 52c81d9 and c68ffb7.

📒 Files selected for processing (1)

• pramen/extras/src/main/scala/za/co/absa/pramen/extras/source/KafkaAvroSource.scala

[github-actions]

Unit Test Coverage

Overall Project	84.35% -0.07%	🍏
Files changed	0%	❌

Module	Coverage
pramen-extras Jacoco Report	66.12% -0.69%	❌

Files

Module	File	Coverage
pramen-extras Jacoco Report	KafkaAvroSource.scala	0% -7.52%	❌