feat(synapse-core): add SP-to-SP piece pull functionality by rvagg · Pull Request #544 · FilOzone/synapse-sdk

rvagg · 2026-01-12T11:24:25Z

Ref: filecoin-project/curio#864
Part of #494

cloudflare-workers-and-pages · 2026-01-12T11:27:01Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	synapse-dev	`70fa757`	Commit Preview URL Branch Preview URL	Feb 13 2026, 03:18 AM

rvagg · 2026-01-17T11:46:56Z

Added a temporary utils/example-sp-fetch-e2e.js testing file to test the sp-sp fetch flow. It stores pieces with one SP, confirms they are parked, then asks the second SP to fetch from the first and confirms they are parked. It's working currently on devnet using FilOzone/foc-devnet#34 👌.

rvagg · 2026-01-28T10:30:40Z

Rebased on top of the latest @hugomrdias stack #571

utils/example-pull-e2e.js

rvagg · 2026-02-05T10:32:39Z

latest fixup in here adds the ability to supply a pre-built extraData since we use it for auth but then use it again to addPieces and we don't want the user to have to sign twice for the same thing if we can help it

rvagg · 2026-02-05T12:15:45Z

And another fix, I realised I still had a ton of "fetch" in here. Now all "pull".

hugomrdias

im not the biggest fan of this idempotent POST endpoint, left some comment inline.

packages/synapse-core/src/warm-storage/pieces.ts

packages/synapse-core/src/warm-storage/pull.ts

hugomrdias · 2026-02-10T11:02:15Z

packages/synapse-core/src/sp.ts

+  const fetchWithFreshRequest: typeof globalThis.fetch = (input, init) => {
+    // iso-web passes the Request object as input, extract signal from it
+    const signal = input instanceof Request ? input.signal : init?.signal
+    return globalThis.fetch(url, { method: 'POST', body, headers, signal })
+  }


how does this, fixes body being consumed ?

https://github.com/hugomrdias/iso-repo/blob/7e90672039d96fae8acb6a4b174ad7a9c715e2f5/packages/iso-web/src/http.js#L221-L225

https://github.com/hugomrdias/iso-repo/blob/7e90672039d96fae8acb6a4b174ad7a9c715e2f5/packages/iso-web/src/http.js#L228

request created once, reused repeatedly within the loop, body can't be re-read, so we just pick out what we want from request and implicitly make a new one

packages/synapse-core/test/pull.test.ts

hugomrdias · 2026-02-10T11:12:39Z

packages/synapse-core/test/pull.test.ts

+      const result = await SP.waitForPullStatus({
+        ...baseOptions(),
+        minTimeout: 10,
+        onStatus: (response) => statusUpdates.push(response.status),
+      })


yeah this is why you need to presign because you need it for the waitFor function, and you poll from a http POST endpoint.

this breaks the pattern of, do mutation, get status url to http poll from.

no, it's not why we need to pre-sign. The pre-sign isn't a workaround forced by the polling pattern, it's required for the initial request itself. The extraData blob is the auth mechanism: Curio can eth_call to verify the client is setup and prepared to pay. The endpoint is idempotent by design, keyed on a hash of extraData, so re-posting the same signed request returns current status rather than creating a new fetch job.

i.e. we pre-sign because the endpoint needs auth for initiation. The idempotent POST design then gives status polling for free: same request, same response with updated status. Adding a separate GET status URL would mean either adding a second endpoint that also needs auth (more signing, more API surface), or returning a nonce/token from the POST that the GET uses (more state, same polling). We have extraData, it's a natural idempotency key already, so we just reuse it without inventing a new one to do the same job.

rvagg · 2026-02-12T13:04:01Z

Force pushed a gnarly rebase, now just one commit on top of master. 🤞

Add low-level Pull module for initiating piece pulls from external storage providers to the local SP via Curio's POST /pdp/piece/pull endpoint. Features: - pullPieces: Initiate pull requests with EIP-712 signed authorization - waitForPullStatus: Poll until completion with status callbacks - Support for both existing data sets (signAddPieces) and new data sets (signCreateDataSetAndAddPieces) - Configurable timeout and abort signal support The pull endpoint is idempotent - repeated calls with the same extraData return status updates rather than creating duplicates.

rvagg · 2026-02-13T07:28:50Z

I borked the rebase when I said this was done yesterday, didn't include the fixes I said I'd made! Should be good now though. And #593 is on top of this. They are both now on top of latest master.

github-project-automation bot added this to FOC Jan 12, 2026

github-project-automation bot moved this to 📌 Triage in FOC Jan 12, 2026

BigLep assigned rvagg Jan 15, 2026

BigLep moved this from 📌 Triage to ⌨️ In Progress in FOC Jan 15, 2026

BigLep linked an issue Jan 16, 2026 that may be closed by this pull request

GA DURABILITY: Multi-copy upload via SP-to-SP pull #494

Open

5 tasks

rvagg force-pushed the rvagg/sp-sp-fetch branch from c2011f6 to 7af205b Compare January 17, 2026 06:01

rvagg mentioned this pull request Jan 17, 2026

feat: support SP-to-SP fetch via host.docker.internal resolution FilOzone/foc-devnet#34

Merged

rvagg force-pushed the rvagg/sp-sp-fetch branch from ac3552a to 3c5cdb9 Compare January 20, 2026 09:08

rvagg changed the base branch from master to hugomrdias/move-tests January 20, 2026 09:08

hugomrdias force-pushed the hugomrdias/move-tests branch 2 times, most recently from 0a20df0 to a231d32 Compare January 26, 2026 12:20

rvagg force-pushed the rvagg/sp-sp-fetch branch from 3c5cdb9 to 429c2e3 Compare January 28, 2026 10:29

rvagg changed the base branch from hugomrdias/move-tests to hugomrdias/session-key January 28, 2026 10:30

rjan90 added this to the M4.0: mainnet staged milestone Jan 30, 2026

Base automatically changed from hugomrdias/session-key to hugomrdias/payments January 30, 2026 12:03

Base automatically changed from hugomrdias/payments to hugomrdias/move-tests February 2, 2026 19:19

rvagg force-pushed the rvagg/sp-sp-fetch branch from 429c2e3 to 8ec5942 Compare February 3, 2026 03:23

hugomrdias force-pushed the hugomrdias/move-tests branch from a5f8971 to d75fb75 Compare February 3, 2026 11:52

rvagg force-pushed the rvagg/sp-sp-fetch branch from 8ec5942 to b6d3479 Compare February 4, 2026 10:44

rvagg changed the base branch from hugomrdias/move-tests to master February 4, 2026 10:45

rvagg changed the title ~~(WIP) feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers~~ feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers Feb 4, 2026

rvagg marked this pull request as ready for review February 4, 2026 10:45

rvagg requested a review from hugomrdias as a code owner February 4, 2026 10:45

rvagg commented Feb 4, 2026

View reviewed changes

utils/example-pull-e2e.js Outdated Show resolved Hide resolved

rvagg force-pushed the rvagg/sp-sp-fetch branch 2 times, most recently from b221f97 to ff14a1d Compare February 4, 2026 11:31

rjan90 moved this from ⌨️ In Progress to 🔎 Awaiting review in FOC Feb 4, 2026

rvagg mentioned this pull request Feb 4, 2026

GA DURABILITY: Multi-copy upload via SP-to-SP pull #494

Open

5 tasks

rvagg force-pushed the rvagg/sp-sp-fetch branch from ff14a1d to 13ed974 Compare February 5, 2026 10:30

rvagg changed the title ~~feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers~~ feat(synapse-core): add SP-to-SP piece pull functionality Feb 5, 2026

rvagg force-pushed the rvagg/sp-sp-fetch branch from 13ed974 to 2d1a65e Compare February 5, 2026 10:32

rvagg force-pushed the rvagg/sp-sp-fetch branch from 2d1a65e to caebe09 Compare February 5, 2026 12:15

rvagg force-pushed the rvagg/sp-sp-fetch branch 2 times, most recently from 2f8c0ca to 59e576b Compare February 6, 2026 13:56

rjan90 mentioned this pull request Feb 6, 2026

Cut Synapse-SDK v0.38.0 #591

Open

2 tasks

rvagg mentioned this pull request Feb 6, 2026

feat(storage): multi-copy upload with store->pull->commit flow #593

Open

hugomrdias requested changes Feb 10, 2026

View reviewed changes

github-project-automation bot moved this from 🔎 Awaiting review to ⌨️ In Progress in FOC Feb 10, 2026

rvagg force-pushed the rvagg/sp-sp-fetch branch from 59e576b to 63c6170 Compare February 12, 2026 13:03

BigLep moved this from ⌨️ In Progress to 🔎 Awaiting review in FOC Feb 12, 2026

rvagg force-pushed the rvagg/sp-sp-fetch branch from 63c6170 to 2d43c4f Compare February 13, 2026 03:04

rvagg force-pushed the rvagg/sp-sp-fetch branch from 2d43c4f to 70fa757 Compare February 13, 2026 03:05

rvagg mentioned this pull request Feb 20, 2026

Add pull flow testing to validate SP piece retrieval from external sources FilOzone/dealbot#300

Open

rjan90 requested a review from juliangruber February 23, 2026 14:51

Comments

Conversation

rvagg commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages bot commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

rvagg commented Jan 17, 2026

Uh oh!

rvagg commented Jan 28, 2026

Uh oh!

Uh oh!

rvagg commented Feb 5, 2026

Uh oh!

rvagg commented Feb 5, 2026

Uh oh!

hugomrdias left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hugomrdias Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

rvagg Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hugomrdias Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

rvagg Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

rvagg commented Feb 12, 2026

Uh oh!

rvagg commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rvagg commented Jan 12, 2026 •

edited

Loading

cloudflare-workers-and-pages bot commented Jan 12, 2026 •

edited

Loading