Model disconnected and can't resume training?

[violent508]

Search before asking

• I have searched the HUB issues and found no similar bug report.

HUB Component

Training

Bug

I was training my model and after about 25% it got disconnected. When I try to resume training it just says "something went wrong". Seems to be similar to the issue this person had here:

Is this just something wrong on the server side?

Environment

No response

Minimal Reproducible Example

No response

Additional

No response

[UltralyticsAssistant]

👋 Hello @violent508, thank you for raising an issue about Ultralytics HUB 🚀! This is an automated response to help you get started quickly—an Ultralytics engineer will follow up with you soon.

Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLO models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

Since this appears to be a 🐛 Bug Report (training disconnect and resume failure), could you please provide a Minimum Reproducible Example (MRE) to help us investigate? See our guide to reproduce. Helpful details include:

• Project ID and Model ID (from the HUB URL)
• Workspace name
• Approximate timestamp and timezone when the disconnect occurred
• Whether training was started from the HUB UI or via the Inference API
• Browser and OS (and HUB App version if using mobile)
• Screenshots of the error message and the training page
• Any logs or console output you can share
• Steps you took to attempt resume (e.g., Resume button in UI, re-launching from API) and what happened

If this is a ❓ Question instead, please include dataset details, model type, environment, and any relevant configuration so we can provide the most helpful guidance.

We try to respond to all issues as promptly as possible. Thank you for your patience! 🙏🛠️

[pderrenger]

Hi @violent508 — thanks for the report, and sorry for the disruption. Let’s get you unblocked and figure out what happened.

A couple quick clarifiers to help us triage:

• Were you training with Ultralytics HUB Cloud Training or using your own agent (local machine/Colab)?
• Could you share the Model URL or ID (from the Model page), and roughly when the disconnect occurred (with timezone)?
• If Cloud Training with “Epochs” billing, please confirm your Account Balance on the model’s Billing tab wasn’t depleted at the time of the pause. HUB will auto‑pause when balance is insufficient, and you can resume from the last checkpoint. See the Cloud Training notes on pausing/resuming and billing in the Ultralytics HUB Cloud Training docs.

Fast ways to resume, depending on your setup:

• From the Model page in HUB: If a checkpoint exists, you should be able to click Resume. This is the preferred path and works both after pauses and transient disconnects. See the HUB Models guide on resuming from the Model page.

• If you trained with your own agent and want to resume locally:

  • CLI: yolo train resume model=path/to/last.pt
  • Python:
    from ultralytics import YOLO
    model = YOLO("path/to/last.pt")
    results = model.train(resume=True)

  Details and caveats for resuming runs are in the Resuming Interrupted Trainings section of the Train docs.

Check that a resumable checkpoint exists

• Using the HUB SDK you can confirm whether HUB sees last weights and fetch their URL:
  from hub_sdk import HUBClient
  client = HUBClient({"api_key": ""})
  m = client.model("")
  print("resumable:", m.is_resumable(), "trained:", m.is_trained())
  print("last_url:", m.get_weights_url("last"))

  The is_resumable() and is_trained() helpers are documented in the HUB SDK reference, and get_weights_url("last") is shown in the HUB‑SDK Model Management guide.

If you’re using your own agent (local/Colab)

• Make sure you’re on the latest packages and logged in:
  • pip install -U ultralytics hub-sdk
  • yolo login <YOUR_API_KEY> (or ensure SETTINGS["api_key"] is set)
• Re‑run your training cell/script; the HUB integration starts heartbeats and sync automatically and will re‑attach to your model session. See how the HUB training session and heartbeats are managed in ultralytics.hub.session and the HUB callbacks.

If the UI still says “something went wrong”

• Please paste the Model URL/ID here and confirm Cloud vs BYOA. If BYOA, include:
  • ultralytics and hub-sdk versions (pip show ultralytics hub-sdk)
  • the last ~50 lines of your console log around the disconnect
• As a quick UI workaround, try Stop Training on the Model page, wait ~10 seconds, then click Resume to force a clean reinitialization. HUB will continue from the last uploaded checkpoint if available.

References

• See the Ultralytics HUB Models docs for resuming from the Model page.
• See Ultralytics HUB Cloud Training for pause/resume behavior and billing.
• See Model Training with Ultralytics YOLO for resuming interrupted trainings.
• See the HUB SDK reference (is_resumable, is_trained) and HUB‑SDK Model Management (get_weights_url).

Once we have the model ID and mode (Cloud vs BYOA), we can dig deeper on the backend if needed. Thanks for the patience!

[yogendrasinghx]

Hi @violent508,

Thanks for reporting this issue and sorry for the inconvenience.

Could you please share your Model ID so we can investigate?

You can find it at the end of your model’s URL.
For example: https://hub.ultralytics.com/models/7wzkDSKNMcwkPTs8ZVJC, the Model ID is 7wzkDSKNMcwkPTs8ZVJC.

[tlong123]

Hello! commenting here as I am having the same issue - ID 8EizNw8BqrKWGPF12hJ7 - disconnected, checkpoint saved at epoch 39 - have plenty left in balance, errors when I click resume training and when I go to the deploy tab says "Model Not Trained! Finish training your model."

[yogendrasinghx]

Hi @tlong123,

Thanks for sharing the model ID. I’ve restored your training to epoch-84. You can now resume your training.

Apologies for the inconvenience, and thank you for your patience!

[tlong123]

thanks @yogendrasinghx, seem to have hit the same problem again though and can't progress past epoch 84

[BenedictIbe]

i hit the same issue, i cant seem to resume trainig:

https://hub.ultralytics.com/models/SYXZ4d11Zn13jXlwq1Ck

[sergiuwaxmann]

@BenedictIbe Seems everything is fine with your model. You should be able to resume training normally.

[BenedictIbe]

i am still getting error and its still showing disconnected even though my credit or billing balance is still depleting,
plus i want to switch to a H200 gpu

[BenedictIbe]

i have tried to transfer the project, nothing is working from my end, it keeps throwing:

something went wrong

[BenedictIbe]

please kindly review my inability to resume and swith to a faster GPU

and the error "something went wrong"

[sergiuwaxmann]

@BenedictIbe Can you try now? If you see "Something went wrong" when you want to start training (after selecting a GPU), most likely that GPU is not available when you want to start training.