feat(backup): retry backup after failed attempt

[Der-Penz]

Summary

This adds the option to retry the backup job if the backup failed for some reason (e.g. The target repository was not online).
It includes the option to configure for each backup job how often it should retry the failed backup and what duration will be between backups. This can be configured in the Advanced settings of the backup job. By default it will try 3 Backups in 60 minutes intervals.
if a retry of the backup failed the maximum number of times and a new scheduled backup is also failing it wont try again so it will only exhaust the maxRetries once.

Changes

• updated backup failure states to schedule a new backup retry if needed
• updated db schema for the backup job
• integrated settings into the backup creation/edit dialog

Testing

I run a hourly backup local and it with 3 retry attempts and 25 minutes interval which worked fine. Resetting the failed attempt counter on successful backup also worked correctly.

Note

I used AI to get an initial draft on how to implement it and where i need to change something. But i looked over each change and reviewed/improved it.

Closes #481

I am open for feedback. Let me know if there is anything thats not done correctly or missing and I will try to improve it.

[CLAassistant]

All committers have signed the CLA.

[Der-Penz]

For context this is how the options dialog looks now with the retry settings:

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 152ec057-4075-42c7-89e4-5eb4a097d99b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[nicotsx]

Thank you @Der-Penz for your work, looks great! There is one problem tho that I can see it's in the following case:

1. You have a backup schedule with a hourly cron
2. It starts at 10:00
3. Fails at 10:05
4. Next backup is set to 11:05 due to failure + default value
5. Your planned run at 11:00 is "lost" or pushed later

I think we need to do a small computation before setting the next retry to see if the actual next schedule happens before the planned retry and skip retries in this case

[nicotsx]

Additionally, I think the retry mechanism should not trigger when running a backup manually from the UI

[Der-Penz]

I've injected the manual parameter to the function. I did not find another way to get information about if it was triggered manually. If there is i can revert it again

[Der-Penz]

Thanks for the great feedback. I would say it can be like this. It will try to schedule the specified backup x times but if the retried backup is after the next specified normal scheduled backup it will not do anymore retry-backups and reset the counter to 0. So when the actual next scheduled backup starts and fails the counter is at 0 again.