feat(routing/http): IPIP-518: HTTP(S) URLs in Routing V1 API by lidel · Pull Request #1051 · ipfs/boxo

lidel · 2025-10-13T19:53:27Z

This PR implements IPIP-518 which extends /routing/v1 with support for URIs alongside multiaddrs.

Changes

New `Address` type (`types/address.go`)

accepts both multiaddrs (starting with /) and URIs, following IPIP-518 parsing rules
schema-agnostic: any valid URI scheme is accepted for future extensibility
protocol matching with HasProtocol handles http/https semantics (e.g. https:// matches both "http" and "https" queries)
ToMultiaddr() converts HTTP(S) URLs to multiaddrs for backward compatibility with existing software that expects /dns/host/tcp/port/https format, including /http-path/ when the URL has a non-empty path
- paths are not used (consumed) by Kubo atm, but this will make WebSeed integration easier in the future, i hope
invalid addresses are skipped per spec, not treated as errors

New `GenericRecord` type (`types/record_generic.go`)

schema-agnostic record type using Addresses (which accepts both URLs and multiaddrs) instead of []Multiaddr
preserves unknown fields in Extra map for forward compatibility with future protocols
size-limited to 10 KiB to prevent abuse in NDJSON streams

Contentrouter integration (`contentrouter/contentrouter.go`)

readProviderResponses converts GenericRecord to peer.AddrInfo for legacy routing APIs:
- records with valid PeerID: always converted (supports legacy PeerID + /https multiaddr pattern)
- records with transport-ipfs-gateway-http + HTTP(S) URLs: PeerID derived from did:key: (ed25519) or generated as a deterministic placeholder
- other non-PeerID records: skipped
PeerID interop helpers extracted to contentrouter/interop.go

Filter updates (`filters/filters.go`)

ApplyAddrFilter extended to handle Address type alongside Multiaddr
protocol filtering works across both URL schemes and multiaddr protocols

Existing types unchanged

PeerRecord and BitswapRecord continue to use []Multiaddr (multiaddrs only)
URLs belong in GenericRecord, which is the forward-looking record type

Backward compatibility

multiaddr-only implementations continue to work unchanged
HTTP(S) URLs are converted to multiaddrs via ToMultiaddr() so existing consumers (Kubo, someguy) receive addresses they can use
PeerRecord.Addrs remains []Multiaddr and rejects URLs at deserialization time

IPIP:

IPIP-518: URIs in Routing V1 API via Generic Schema specs#518

TODO

no objections in IPIP-518: URIs in Routing V1 API via Generic Schema specs#518
PR in someguy that uses this PR and tests there pass green

…ing API Implements schema-agnostic support for HTTP(S) URLs alongside multiaddrs in the Delegated Routing V1 HTTP API as specified in IPIP-518. Key changes: - Add new Address type that accepts both multiaddrs (starting with /) and URIs - Update PeerRecord and BitswapRecord to use Addresses type instead of []Multiaddr - Implement defensive programming: unsupported addresses are skipped, not errors - Add special protocol filtering for http/https/tls matching - Maintain full backward compatibility with multiaddr-only implementations The implementation is schema-agnostic, accepting any valid URI scheme for future extensibility. URLs are preserved as-is without lossy conversion to multiaddrs, avoiding conversion issues documented in multiaddr#63. Ref: ipfs/specs#518

Implements backward compatibility layer for HTTP(S) URL to multiaddr conversion during transition period. Uses /https (not /tls/http) to match IPNI convention and /dns for dual-stack support. - Add ToMultiaddr() method and String() for Addresses type - Replace TODOs in contentrouter with actual conversion - Add comprehensive tests Temporary compatibility layer until ecosystem adopts IPIP-518 URLs.

codecov · 2025-10-13T20:47:36Z

Codecov Report

❌ Patch coverage is 82.85714% with 54 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.96%. Comparing base (f0cdbf6) to head (0feb195).

Files with missing lines	Patch %	Lines
routing/http/types/address.go	85.41%	16 Missing and 5 partials ⚠️
routing/http/contentrouter/contentrouter.go	68.96%	8 Missing and 1 partial ⚠️
routing/http/filters/filters.go	87.03%	5 Missing and 2 partials ⚠️
routing/http/contentrouter/interop.go	80.64%	3 Missing and 3 partials ⚠️
routing/http/types/json/responses.go	50.00%	2 Missing and 2 partials ⚠️
routing/http/types/record_generic.go	88.88%	2 Missing and 2 partials ⚠️
routing/http/types/ndjson/records.go	70.00%	2 Missing and 1 partial ⚠️

@@            Coverage Diff             @@
##             main    #1051      +/-   ##
==========================================
+ Coverage   61.68%   61.96%   +0.27%     
==========================================
  Files         265      268       +3     
  Lines       26513    26804     +291     
==========================================
+ Hits        16355    16609     +254     
- Misses       8469     8495      +26     
- Partials     1689     1700      +11

Files with missing lines	Coverage Δ
routing/http/client/client.go	`75.60% <100.00%> (+0.06%)`	⬆️
routing/http/types/ndjson/records.go	`66.03% <70.00%> (+14.87%)`	⬆️
routing/http/types/json/responses.go	`56.66% <50.00%> (-1.03%)`	⬇️
routing/http/types/record_generic.go	`88.88% <88.88%> (ø)`
routing/http/contentrouter/interop.go	`80.64% <80.64%> (ø)`
routing/http/filters/filters.go	`83.21% <87.03%> (-1.56%)`	⬇️
routing/http/contentrouter/contentrouter.go	`56.86% <68.96%> (+2.00%)`	⬆️
routing/http/types/address.go	`85.41% <85.41%> (ø)`

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

routing/http/types/address.go

gammazero · 2025-10-31T02:37:17Z

routing/http/types/address.go

+// Returns nil if the address cannot be converted (e.g., non-HTTP schemes or invalid addresses).
+// This is a temporary compatibility layer for the transition period while existing software
+// expects multiaddrs with /http protocol to signal HTTP retrieval support.
+func (a *Address) ToMultiaddr() multiaddr.Multiaddr {


An alternative to this that avoids hardcoded values for addrProto can be found here:
https://github.com/ipni/go-libipni/blob/main/maurl/convert.go#L110-L156

Both approaches work. The go-libipni version has had years of testing and operation if you want to cut and paste that one, but that does not mean it is necessarily better. I think perhaps using megular expressions would be better still.

I am worried about unintended consequences if w blindly convert here like IPNI func does, let's keep as-is.

Mind, the only reason this func exist, is backward compatibility with existing HTTP support for trustless gateways which have "fake peerids" and "fake libp2p multiaddrs".

That is why we only do this for HTTP(S) schema URIs + have default ports.

Found interesting inconsistency:

IPNI's FromURL passes url.PathEscape(u.Path) without stripping the leading /, producing %2Fpath instead of path so you have /http-path/%2Fpath instead of `/http-path/path.

The only spec I found is example in https://github.com/multiformats/multiaddr/blob/master/protocols/http-path.md

which suggests the leading / should be stripped to avoid %2F (means IPNI has a bug, but its probably too late to fix).

gammazero · 2025-11-04T16:11:09Z

Triage: Consider updating IPIP to suggest different peer schema.
Blocked waiting for more progress in buoy to determine what is needed here.

Note: Should not assume trustless gateway when HTTP URI with no metadata

…compat introduce GenericRecord (Schema=generic) as specified in the updated IPIP-518, keeping PeerRecord (Schema=peer) unchanged with []Multiaddr. type system: - add GenericRecord with duck-typed Addresses (multiaddrs + URIs) - revert PeerRecord.Addrs and BitswapRecord.Addrs to []Multiaddr - change Address constructors to return value types instead of pointers - remove special "tls" matching from HasProtocol (redundant with default) - simplify ToMultiaddr to use URL scheme directly (http or https) contentrouter (client-side GenericRecord -> peer.AddrInfo conversion): - convert GenericRecord to peer.AddrInfo for legacy libp2p routing APIs - if ID is a valid PeerID, convert regardless of Protocols (supports the legacy pattern where PeerID + /https multiaddr hints at a Trustless IPFS HTTP Gateway even without explicit protocol declaration) - if ID is not a PeerID but record has transport-ipfs-gateway-http + HTTP(S) URLs, derive PeerID from did:key: or generate a placeholder - HTTPS URLs become /dns/host/tcp/443/https multiaddrs via ToMultiaddr - extract filterAddrs helper for nil-safe []Multiaddr extraction - add MaxGenericRecordSize check in JSON deserialization path to match the existing NDJSON check filters (server-side): - add GenericRecord support to ApplyFiltersToIter - add applyGenericFilters for protocol and address filtering on GenericRecord (operates on Addresses directly, no conversion needed) - convert PeerRecord []Multiaddr to Addresses for filtering via applyAddrFilter, then convert back server and client: - serialize/deserialize GenericRecord in JSON and NDJSON responses - add GenericRecord to FindProviders end-to-end client test tests: - add GenericRecord JSON/NDJSON round-trip tests with Extra field preservation, unparseable address handling, and size limit enforcement - add contentrouter tests for PeerID, did:key, placeholder PeerID, empty Protocols, and HTTP URL conversion paths - add filter tests for mixed multiaddr+URL addresses and mixed PeerRecord+GenericRecord iterator streams - add server tests for mixed schema responses, protocol/addr filtering, NDJSON streaming, and Extra field serialization - document that PeerRecord Addrs rejects URLs at unmarshal time

# Conflicts: # CHANGELOG.md # routing/http/contentrouter/contentrouter.go # routing/http/filters/filters.go

…terop helpers - ToMultiaddr() now appends /http-path/ when the URL has a non-empty path, matching go-multiaddr http-path protocol conventions - moved peerIDFromDIDKey, peerIDPlaceholderFromArbitraryID, hasHTTPURL to contentrouter/interop.go for better organization - added unit tests for hasHTTPURL covering http, https, non-HTTP, multiaddr, mixed, empty, and invalid address cases - fixed staticcheck lint suppression for SchemaBitswap in ndjson test

lidel mentioned this pull request Oct 13, 2025

IPIP-518: URIs in Routing V1 API via Generic Schema ipfs/specs#518

Open

8 tasks

lidel changed the title ~~feat(routing/http): implement IPIP-518 HTTP(S) URLs in Delegated Rout…~~ feat(routing/http): IPIP-518: HTTP(S) URLs in Routing V1 API Oct 13, 2025

docs(routing/http): document ToMultiaddr in CHANGELOG

c9ac579

lidel marked this pull request as ready for review October 13, 2025 21:17

lidel requested a review from a team as a code owner October 13, 2025 21:17

gammazero approved these changes Oct 31, 2025

View reviewed changes

gammazero added the status/in-progress In progress label Nov 4, 2025

hsanjuan marked this pull request as draft November 18, 2025 15:45

lidel added 2 commits February 12, 2026 03:08

Merge remote-tracking branch 'origin/main' into ipip-518-urls-in-routing

1f160d6

# Conflicts: # CHANGELOG.md # routing/http/contentrouter/contentrouter.go # routing/http/filters/filters.go

lidel force-pushed the ipip-518-urls-in-routing branch from 5cb023a to b5ccecd Compare February 12, 2026 03:08

lidel requested review from aschmahmann and gammazero February 12, 2026 03:30

lidel marked this pull request as ready for review February 12, 2026 03:30

lidel force-pushed the ipip-518-urls-in-routing branch from b5ccecd to 0feb195 Compare February 12, 2026 03:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(routing/http): IPIP-518: HTTP(S) URLs in Routing V1 API#1051

feat(routing/http): IPIP-518: HTTP(S) URLs in Routing V1 API#1051
lidel wants to merge 6 commits intomainfrom
ipip-518-urls-in-routing

lidel commented Oct 13, 2025 •

edited

Loading

Uh oh!

codecov bot commented Oct 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gammazero Oct 31, 2025

Uh oh!

lidel Feb 12, 2026

Uh oh!

lidel Feb 12, 2026

Uh oh!

gammazero commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

lidel commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

New Address type (types/address.go)

New GenericRecord type (types/record_generic.go)

Contentrouter integration (contentrouter/contentrouter.go)

Filter updates (filters/filters.go)

Existing types unchanged

Backward compatibility

TODO

Uh oh!

codecov bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gammazero Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

lidel Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

lidel Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gammazero commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

lidel commented Oct 13, 2025 •

edited

Loading

New `Address` type (`types/address.go`)

New `GenericRecord` type (`types/record_generic.go`)

Contentrouter integration (`contentrouter/contentrouter.go`)

Filter updates (`filters/filters.go`)

codecov bot commented Oct 13, 2025 •

edited

Loading