Multipart upload session returns “Not Found” during CreateMultipartUpload

[Kirubel-Fikru]

When uploading to bucket via Herald with the OpenStack backend through the openstack backend, the CreateMultipartUpload path intermittently fails. Herald logs show:

• Herald correctly routes to the handler and proxies the create-multipart request

• Immediately after, saving the multipart upload session fails with Not Found

• Regular GetObject calls succeed in the same timeframe

2025-10-17 10:33:49 INFO  [backends/worker] -- Worker started listening to tasks for bucket: expo-test-gitlab-dependency-proxy

2025-10-17 10:35:55 DEBUG [src/main] [61f426a] -- Receieved request on http://herald-expo-test-2.herald/expo-test-gitlab-dependency-proxy/tmp/uploads/1760697355-68-0009-2691-99ab2e239742b59d1ec35a825f1723a7?uploads
2025-10-17 10:35:55 DEBUG [src/main] [61f426a] -- Resolving Handler for Request...
2025-10-17 10:35:55 DEBUG [src/main] [61f426a] -- Resolving Swift Handler for Request...
2025-10-17 10:35:55 INFO  [src/main] [61f426a] -- [Swift backend] Proxying Create Multipart Upload Request...
2025-10-17 10:35:56 WARN  [src/main] [61f426a] -- Failed to save multipart upload session: Not Found upload session: Not Found

2025-10-17 10:35:57 DEBUG [src/main] [631bbbf] -- Receieved request on http://10.64.3.162:8000/health-check
2025-10-17 10:36:00 INFO  [src/main] [0ef3635] -- Get Object Successful: OK
2025-10-17 10:36:00 DEBUG [src/main] [a433f05] -- Receieved request on http://herald-expo-test-2.herald/langfuse-...

Environment

• Herald deployment: Kubernetes (svc: herald-expo-test-2.herald)
• Bucket: expo-test-gitlab-dependency-proxy
• Backend: Swift
• Herald version: (please fill in: image tag / commit SHA)
• Swift provider: OpenStack Swift

[Yohe-Am]

Hmm, interesting. Not familiar with the multipart implementation nor the swift API but the code seems fine to my eyes. When you say intermittent, I'm guessing you can't repro this? Can you try deploying an image that has printf all around this call. Specifically, when !status.ok I'd love to see what the sent url, body and the full returned response along with the body.

[Kirubel-Fikru]

By intermittent i mean normal operations work and in the middle when there is a need for a multipart upload it shows the error message and continues with other operations. it can be reproduced whenever a gitlab-ci job is run which requires a gitlab-ci dependency proxy.

[Kirubel-Fikru]

@Yohe-Am

2025-10-24 08:27:07 DEBUG [src/main] [8eabf7a] -- Receieved request on http://10.64.3.110:8000/health-check 
2025-10-24 08:27:07 INFO [src/main] [8eabf7a] -- Health Check Complete: Ok 
2025-10-24 08:27:10 INFO [backends/task_store] -- Saved locks to remote storage 
2025-10-24 08:27:10 INFO [workers/mod] -- Refreshing workers context 
2025-10-24 08:27:10 INFO [backends/worker] -- Handling message on worker of type: UpdateContext 
2025-10-24 08:27:10 INFO [backends/worker] -- Handling message on worker of type: UpdateContext 
2025-10-24 08:27:10 INFO [backends/worker] -- Updated worker's herald context 
2025-10-24 08:27:10 INFO [backends/worker] -- Updated worker's herald context 
2025-10-24 08:27:17 DEBUG [src/main] [bcfebb1] -- Receieved request on http://10.64.3.110:8000/health-check 
2025-10-24 08:27:17 INFO [src/main] [bcfebb1] -- Health Check Complete: Ok 
2025-10-24 08:27:17 DEBUG [src/main] [ace78ba] -- Receieved request on http://10.64.3.110:8000/health-check 
2025-10-24 08:27:17 INFO [src/main] [ace78ba] -- Health Check Complete: Ok 
2025-10-24 08:27:18 DEBUG [src/main] [9f1d855] -- Receieved request on http://herald-expo-test-2.herald/expo-test-gitlab-dependency-proxy/tmp/uploads/1761294438-439-0001-5891-dc2b01e13e25724ea2faff33a3e3c23a?uploads 
2025-10-24 08:27:18 DEBUG [src/main] [9f1d855] -- Resolving Handler for Request... 
2025-10-24 08:27:18 DEBUG [src/main] [9f1d855] -- Resolving Swift Handler for Request... 
2025-10-24 08:27:18 INFO [src/main] [9f1d855] -- [Swift backend] Proxying Create Multipart Upload Request...

Receieved request on http://herald-expo-test-2.herald/expo-test-gitlab-dependency-proxy/tmp/uploads/1761294438-439-0001-5891-dc2b01e13e25724ea2faff33a3e3c23a?uploads. this is the url for the multipart upload request

After this message it follows with the log from my earlier message.