🔄 Refactor: Additional Refactoring of Jobs

[MrAdder]

Refs #4550

Summary

This PR introduces consistent queue configuration, rate limiting, overlap protection, and improved observability across MSHIP and Training jobs. It also updates Horizon to ensure the new dedicated queues are actively consumed by workers.

The goal is to:

• Prevent duplicate or overlapping processing per account/check.
• Protect external services with rate limiting.
• Improve reliability with retries/backoff.
• Standardise logging and failure handling.
• Explicitly route jobs to dedicated service queues.

---

MSHIP Sync Jobs

Updated the following jobs:

• SyncToCTS
• SyncToDiscord
• SyncToHelpdesk
• SyncToMoodle

Changes

• Strongly typed Account property.
• Added:
  • $tries = 3
  • $backoff = 30
  • Dedicated $queue per service (cts, discord, forums, helpdesk, moodle)
• Added structured start/complete logging with consistent context.
• Implemented failed(Throwable $exception) logging.
• Added middleware:
  • RateLimitedWithRedis('<service>-sync')
  • WithoutOverlapping($accountId) with:
    • releaseAfter(5)
    • expireAfter(120)

This ensures:

• Per-account sync work is serialised.
• External APIs are protected.
• Failures are clearly logged.
• Jobs are isolated onto dedicated queues.

---

Training Jobs

Updated:

• ActionWaitingListRetentionCheckRemoval
• SendWaitingListRetentionCheck
• UpdateAccountWaitingListEligibility

Changes

• Added:
  • $tries = 3
  • $backoff (30 for retention jobs, 15 for eligibility)
  • Dedicated queues:
    • training-retention
    • training-eligibility
• Added structured start/complete logging.
• Added failed(Throwable $exception) logging.
• Added middleware:
  • RateLimitedWithRedis(...)
  • WithoutOverlapping(...) with scoped keys:
    • retention-check:{id}
    • waiting-list-account:{id}
    • account:{id}

Behaviour Improvements

• Retention removal job:
  • Explicitly fails if notification cannot be sent.
  • Prevents accidental removal if email delivery fails.
• Retention send job:
  • Uses DB transaction.
  • Rolls back and fails cleanly on notification error.
• Eligibility job:
  • Logs number of waiting lists processed.
  • Serialises updates per account.

---

Discord Listener Adjustment

SetupDiscordUser:

• Removed explicit ->onQueue('default').
• Now allows SyncToDiscord to self-route to the dedicated discord queue.

This aligns job routing with the new queue architecture.

---

Horizon Configuration

Updated config/horizon.php:

Queues now include:

[
  'high',
  'default',
  'discord',
  'helpdesk',
  'moodle',
  'cts',
  'training-retention',
  'training-eligibility',
]

This ensures dedicated service queues are actually consumed by workers in both production and local.

---

Tests Added

MSHIP

• SyncJobConfigurationTest
  • Verifies:
    • tries
    • backoff
    • queue name
    • middleware presence (RateLimitedWithRedis, WithoutOverlapping)

Training

• TrainingJobConfigurationTest
  • Validates queue config and middleware.
• Updated notification failure test:
  • Asserts structured logging context is present.

---

Result

This PR:

• Standardises queue configuration across services.
• Improves resilience and observability.
• Prevents duplicate execution and race conditions.
• Introduces clear separation of external integrations via dedicated queues.
• Ensures Horizon is correctly configured to process new queues.
• Adds test coverage for configuration guarantees.
• Keeps legacy integrations (Forums) aligned with current architecture.

This lays groundwork for safer horizontal scaling and improved operational visibility.

[MrAdder]

Removed the Refactor for the Forums as its now been removed

[MrAdder]

@kristiankunc or @CLC0609 These work fine locally and will not break the site, due to restrictions on my side I can not test the full functionality to Helpdesk or moodle so thats all that will need testing

[Copilot]

Pull request overview

This PR adds consistent queue configuration, rate limiting, overlap protection, structured logging, and failure handling across MSHIP sync jobs and Training jobs. It also updates Horizon worker configuration to consume the new dedicated queues.

Changes:

• Added $tries, $backoff, dedicated queues, middleware() (rate limiting + overlap protection), failed() handler, and structured logging to all MSHIP sync and Training jobs.
• Updated Horizon config to include new dedicated service queues (helpdesk, moodle, cts, training-retention, training-eligibility).
• Added configuration tests for both MSHIP and Training jobs.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
config/horizon.php	Added new dedicated queues to production and local supervisors
app/Listeners/Discord/SetupDiscordUser.php	Removed explicit ->onQueue('default') to let job self-route
app/Jobs/Mship/SyncToDiscord.php	Added logging, failed(), updated backoff, added WithoutOverlapping options
app/Jobs/Mship/SyncToCTS.php	Added queue config, middleware, logging, failed() handler
app/Jobs/Mship/SyncToHelpdesk.php	Added queue config, middleware, logging, failed() handler
app/Jobs/Mship/SyncToMoodle.php	Added queue config, middleware, logging, failed() handler
app/Jobs/Training/UpdateAccountWaitingListEligibility.php	Added queue config, middleware, logging, failed() handler
app/Jobs/Training/SendWaitingListRetentionCheck.php	Added queue config, middleware, logging, failed() handler
app/Jobs/Training/ActionWaitingListRetentionCheckRemoval.php	Added queue config, middleware, logging, failed() handler
tests/Unit/Jobs/Mship/SyncJobConfigurationTest.php	New test validating MSHIP job config and middleware
tests/Unit/Jobs/Training/TrainingJobConfigurationTest.php	New test validating Training job config and middleware
tests/Unit/Training/.../WaitingListRetentionChecksNotificationsTest.php	Updated assertion to verify structured log context

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[MrAdder]

I turned a setting on when I got my student benefits will turn of copilot reviews in morning