Change Concluding reader and writer to HTTP specific types

[FranzBusch]

Motivation

We moved the reader and writer types over to swift-async-algorithms that left us with the concluding variants. Those types are really HTTP specific and there is no reason to make them general purpose. Furthermore, we have some more HTTP specific requirements that we wanted to fulfill around being able to one-shot respond to a request with a header, body and trailers.

Modifications

This PR introduces new receiver and sender types for the client and server aligned with existing HTTPServerResponder type. These new types are protocols instead of a concrete type which allows implementations to customize them. Furthermore, there are new one-shot APIs on the senders.

## Result

No more concluding reader and writer APIs and instead only HTTP specific types.

[FranzBusch]

This should be removed and we need to migrate existing code off of it.

[FranzBusch]

We should indicate that this is a copying action and provide a generic extension if the reader is also a CallerAsyncReader that allows us to do this copy free

[guoye-zhang]

Not sure how useful it is to have a convenience to collect into a fixed-sized buffer. The size of the response is unpredictable, and most people would probably allocate a Content-Length sized buffer, which would break if the response has no Content-Length or if there is a Content-Encoding.

[FranzBusch]

The idea here was to replace the collect(upTo:) method from before which required us to pick the kind of buffer. I replaced it with a generic buffer where the caller passes in the buffer. This has the same semantics as the collect(upTo:) method but allows the caller now to choose the buffer type.

[guoye-zhang]

The convenience API should collect to a resizable buffer type, not forcing users to preallocate to the capacity.

[FranzBusch]

This should be gone and we need to migrate existing APIs off it

[FranzBusch]

Same comments as on the response side apply here

[FranzBusch]

Same here

[FranzBusch]

Same comment as on the request side

[guoye-zhang]

How big is the buffer?

[FranzBusch]

The caller decides that since they pass the buffer.

[guoye-zhang]

We should design fast paths around common usages. My intuition is that these are the top use cases:

1. Header + full body
2. Header + streaming body
3. Header + no body
4. Header + streaming body + trailer

It does not make sense to use a trailer without streaming the body, since the info can otherwise be sent in the header.

Yes, we can fast path 1+3 and leave 2 slow, but if we can somehow make 2 fast, there is no need for any special cases.

[FranzBusch]

What is your expectation of a "fast" 2 on the wire? The only thing that I could see us speed up here is if we take an optional buffer if the user has some data available synchronously already; however, in the general case if they are streaming my assumption is that all chunks including the first one are going to be asynchronous.

[guoye-zhang]

I want to be able to signal the end along with some data (case 1 and 2), or signal the end with the header alone (case 3). Yes, it would be nice to send the header with the beginning of the body as well, but that doesn't affect the bytes on the wire. The implementation would seem sloppy if we always sent an empty DATA frame in the end.

[guoye-zhang]

This won't work if the limit is .max

[FranzBusch]

I have worked through some of the requirements raised here and in offline conversations and tried a few different approaches. Here are my latest findings and possible options summarized using Claude.

Requirements

1. Middleware that adds writes after the user body. For example gzip CRC, OHTTP suffix, response signature.
2. Middleware that adds trailers. For example X-Trace-Id, X-Body-Checksum.
3. One-shot send of head + body + trailers for callers that have everything ready.
4. Fuse the last body chunk + optional trailers + FIN into one transport call. HTTP/2 and HTTP/3 carry trailers in the same flight as the END_STREAM flag, and the last DATA frame can ride END_STREAM. Splitting these costs an extra round trip or write syscall.
5. Statically enforce that trailers come after the body. Sender shouldn't be able to write more body after declaring trailers, receiver shouldn't be able to peek at trailers before draining the body.

Current PR solution

public protocol HTTPResponseSender<Writer>: ~Copyable, ~Escapable {
    associatedtype Writer: AsyncWriter, ~Copyable, ~Escapable
    where Writer.WriteElement == UInt8

    consuming func send<Return>(
        _ response: HTTPResponse,
        body: (consuming sending Writer) async throws -> (Return, HTTPFields?)
    ) async throws -> Return
}

public protocol HTTPResponseReceiver<Reader>: ~Copyable, ~Escapable {
    associatedtype Reader: AsyncReader, ~Copyable, ~Escapable
    where Reader.ReadElement == UInt8

    consuming func receive<Return, Failure: Error>(
        body: (consuming sending Reader) async throws(Failure) -> Return
    ) async throws(Failure) -> (Return, HTTPFields?)
}

Body closure consumes the writer/reader. Trailers come back via the (Return, HTTPFields?) tuple.

Why it doesn't meet the requirements

Requirement	Current state
Middleware post-body writes (1)	❌ The writer is consumed when the user closure returns. Both the wrapper and the underlying writer are gone. No place to do post-body work.
Middleware trailer injection (2)	✅ Wrapper appends to the returned HTTPFields.
One-shot send (3)	✅ Convenience overloads.
Fused last chunk + trailers + FIN (4)	❌ User closure returns, then the implementation writes .end(trailers) separately. The transport sees two writes when one would do.
Trailers come after body (5)	⚠️ Receiver enforces it implicitly through tuple ordering. Sender does not. The user can return ((), trailers) before flushing the body.

Recommended approach

Writer side: HTTPBodyWriter extending AsyncWriter

public protocol HTTPBodyWriter: AsyncWriter, ~Copyable, ~Escapable
where WriteElement == UInt8 {
    consuming func finish<Failure: Error>(
        trailers: HTTPFields?,
        body: (inout Buffer) async throws(Failure) -> Void
    ) async throws(EitherError<WriteFailure, Failure>)
}

public protocol HTTPResponseSender<Writer>: ~Copyable, ~Escapable {
    associatedtype Writer: HTTPBodyWriter, ~Copyable, ~Escapable

    consuming func send<Return: ~Copyable>(
        _ response: HTTPResponse,
        body: (consuming sending Writer) async throws -> Return
    ) async throws -> Return
}

Sender body closure stays (consuming sending Writer) -> Return. No trailer tuple. Trailers ride through writer.finish(trailers:body:). The body closure inside finish lets the caller emit the final chunk that fuses with trailers + FIN.

Reader side: HTTPBodyReader extending AsyncReader

public protocol HTTPBodyReader: AsyncReader, ~Copyable, ~Escapable
where ReadElement == UInt8 {
    /// Reads the next body chunk. Non-nil `trailers` marks the last chunk.
    mutating func read<Return: ~Copyable, Failure: Error>(
        body: (inout Buffer, HTTPFields?) async throws(Failure) -> Return
    ) async throws(EitherError<ReadFailure, Failure>) -> Return
}

extension HTTPBodyReader where Self: ~Copyable {
    /// AsyncReader requirement. Drops trailers for callers that don't care.
    public mutating func read<Return: ~Copyable, Failure: Error>(
        body: (inout Buffer) async throws(Failure) -> Return
    ) async throws(EitherError<ReadFailure, Failure>) -> Return {
        try await self.read { (buf: inout Buffer, _: HTTPFields?) async throws(Failure) -> Return in
            try await body(&buf)
        }
    }
}

public protocol HTTPResponseReceiver<Reader>: ~Copyable, ~Escapable {
    associatedtype Reader: HTTPBodyReader, ~Copyable, ~Escapable

    consuming func receive<Return, Failure: Error>(
        body: (consuming sending Reader) async throws(Failure) -> Return
    ) async throws(Failure) -> Return
}

Trailers ride on the read that emits the last chunk. Receiver protocols drop the tuple and return plain Return. The two read overloads disambiguate by closure arity, so generic some AsyncReader<UInt8> consumers keep working.

Requirement	Recommended state
Middleware post-body writes (1)	✅ Middleware wraps the writer. In its finish impl it does whatever post-body work it needs, then calls the underlying writer's finish.
Middleware trailer injection (2)	✅ Middleware augments trailers in finish before delegating.
One-shot send (3)	✅ Default extensions for span / buffer / trailers-only / empty.
Fused last chunk + trailers + FIN (4)	✅ The (inout Buffer) closure in finish is the last chunk. Transports can write [final bytes][trailers][FIN] in one syscall. Echo composes by calling writer.finish from inside the trailer-bearing read.
Trailers come after body (5)	✅ Statically. consuming finish is the only way to discharge the writer. Non-nil trailers on read mark end of body, and conformers must return empty + nil afterward.

Potential next step: Generalize this into swift-async-algorithms

The shape isn't HTTP-specific. Other streaming protocols have a terminator that carries metadata:

• gzip ends with a CRC and uncompressed length.
• gRPC ends with status and trailing metadata.
• HTTP/2 ends with END_STREAM, optionally with trailers.
• Length-prefixed framings end with a marker.

We should push the concept upstream:

// In AsyncStreaming
public protocol FinishableWriter: AsyncWriter, ~Copyable, ~Escapable {
    associatedtype FinishMetadata

    consuming func finish<Failure: Error>(
        metadata: FinishMetadata,
        body: (inout Buffer) async throws(Failure) -> Void
    ) async throws(EitherError<WriteFailure, Failure>)
}

public protocol FinishableReader: AsyncReader, ~Copyable, ~Escapable {
    associatedtype FinishMetadata

    mutating func read<Return: ~Copyable, Failure: Error>(
        body: (inout Buffer, FinishMetadata?) async throws(Failure) -> Return
    ) async throws(EitherError<ReadFailure, Failure>) -> Return
}

// HTTP specializations
public protocol HTTPBodyWriter: FinishableWriter, ~Copyable, ~Escapable
where WriteElement == UInt8, FinishMetadata == HTTPFields? {}

public protocol HTTPBodyReader: FinishableReader, ~Copyable, ~Escapable
where ReadElement == UInt8, FinishMetadata == HTTPFields {}

If we do that, HTTPBodyWriter and HTTPBodyReader become trivial specializations that pin the metadata type. Same for any other framing. We keep the HTTP-specific protocols in this package today because we don't want to gate the work on upstream changes, but the long-term home is AsyncStreaming.

Alternatives considered

Writer side

A. Current: (consuming sending Writer) -> (Return, HTTPFields?)

public protocol HTTPResponseSender<Writer>: ~Copyable, ~Escapable {
    associatedtype Writer: AsyncWriter, ~Copyable, ~Escapable
    where Writer.WriteElement == UInt8

    consuming func send<Return>(
        _ response: HTTPResponse,
        body: (consuming sending Writer) async throws -> (Return, HTTPFields?)
    ) async throws -> Return
}

Trailers via tuple, writer consumed by closure return. Fails requirements 1 (middleware writes), 4 (fusion), 5 (sender ordering).

B. Plain finish on the writer without a fused body chunk

public protocol HTTPBodyWriter: AsyncWriter, ~Copyable, ~Escapable
where WriteElement == UInt8 {
    consuming func finish(trailers: HTTPFields?) async throws(WriteFailure)
}

public protocol HTTPResponseSender<Writer>: ~Copyable, ~Escapable {
    associatedtype Writer: HTTPBodyWriter, ~Copyable, ~Escapable

    consuming func send<Return>(
        _ response: HTTPResponse,
        body: (consuming sending Writer) async throws -> Return
    ) async throws -> Return
}

Solves middleware writes and trailers, but the user's last write and finish are still two transport calls. No fusion.

C. finish on the sender, not the writer

public protocol HTTPResponseSender<Writer>: ~Copyable, ~Escapable {
    associatedtype Writer: AsyncWriter, ~Copyable, ~Escapable
    where Writer.WriteElement == UInt8

    func sendHead(_ response: HTTPResponse) async throws

    mutating func write<Return>(
        body: (consuming sending Writer) async throws -> Return
    ) async throws -> Return

    consuming func finish(trailers: HTTPFields?) async throws
}

State machine. Doesn't solve middleware writes. A wrapper still has to wrap the writer to observe writes, and we get the same lifetime problem we have today.

D. Frame-stream model

public enum HTTPFrame<Element: ~Copyable>: ~Copyable {
    case data(UniqueArray<Element>)
    case trailers(HTTPFields)
}

public protocol HTTPResponseSender<Writer>: ~Copyable, ~Escapable {
    associatedtype Writer: AsyncWriter, ~Copyable, ~Escapable
    where Writer.WriteElement == HTTPFrame<UInt8>

    consuming func send<Return>(
        _ response: HTTPResponse,
        body: (consuming sending Writer) async throws -> Return
    ) async throws -> Return
}

Reshapes the byte-stream protocol from AsyncWriter<UInt8> to AsyncWriter<HTTPFrame<UInt8>>. This losses the ordering guarantees of head, body, trailer.

E. (inout sending Writer) -> (Return, HTTPFields?)

public protocol HTTPResponseSender<Writer>: ~Copyable, ~Escapable {
    associatedtype Writer: AsyncWriter, ~Copyable, ~Escapable
    where Writer.WriteElement == UInt8

    consuming func send<Return>(
        _ response: HTTPResponse,
        body: (inout sending Writer) async throws -> (Return, HTTPFields?)
    ) async throws -> Return
}

Implementation owns the writer in its send method, lends mutable access to the closure, regains it after. Solves middleware writes. Doesn't solve fusion: the closure declares "body is done" by returning, but the writer's last write already flushed.

H. HTTPResponseSender directly inherits AsyncWriter

public protocol HTTPResponseSender: AsyncWriter, ~Copyable, ~Escapable
where WriteElement == UInt8 {
    mutating func sendHead(_ response: HTTPResponse) async throws
    consuming func sendTrailers(_ trailers: HTTPFields?) async throws
}

Sender is the writer. Conflates protocol layers. The sender has a head/body/trailers lifecycle that doesn't fit the open-ended write/write/... shape of AsyncWriter. Composing middleware that wraps only the body gets messier.

Reader side

A. Current: receiver returns (Return, HTTPFields?)

public protocol HTTPResponseReceiver<Reader>: ~Copyable, ~Escapable {
    associatedtype Reader: AsyncReader, ~Copyable, ~Escapable
    where Reader.ReadElement == UInt8

    consuming func receive<Return, Failure: Error>(
        body: (consuming sending Reader) async throws(Failure) -> Return
    ) async throws(Failure) -> (Return, HTTPFields?)
}

Trailers come back from receive after the reader is consumed. Fails fusion. Forces an intermediate copy in echo and proxy.

B. HTTPBodyReader with consuming finish

public protocol HTTPBodyReader: AsyncReader, ~Copyable, ~Escapable
where ReadElement == UInt8 {
    consuming func finish<Return, Failure: Error>(
        body: (inout Buffer, HTTPFields?) async throws(Failure) -> Return
    ) async throws(EitherError<ReadFailure, Failure>) -> Return
}

public protocol HTTPResponseReceiver<Reader>: ~Copyable, ~Escapable {
    associatedtype Reader: HTTPBodyReader, ~Copyable, ~Escapable

    consuming func receive<Return, Failure: Error>(
        body: (consuming sending Reader) async throws(Failure) -> Return
    ) async throws(Failure) -> Return
}

Mirror of HTTPBodyWriter.finish. Solves fusion, but conformers need an empty-buffer-from-read handshake to signal "now call finish", and lookahead in HTTP/1.1 chunked. Pushes mess onto every conformer.

C. Standalone HTTPBodyReader, read always delivers (Buffer, HTTPFields?)

public protocol HTTPBodyReader: ~Copyable, ~Escapable {
    associatedtype Buffer: RangeReplaceableContainer<UInt8> & ~Copyable
    associatedtype ReadFailure: Error

    mutating func read<Return: ~Copyable, Failure: Error>(
        body: (inout Buffer, HTTPFields?) async throws(Failure) -> Return
    ) async throws(EitherError<ReadFailure, Failure>) -> Return
}

Drops AsyncReader inheritance, single trailers-aware read. Loses compatibility with generic some AsyncReader<UInt8> consumers.

D. Frame-stream model

public protocol HTTPResponseReceiver<Reader>: ~Copyable, ~Escapable {
    associatedtype Reader: AsyncReader, ~Copyable, ~Escapable
    where Reader.ReadElement == HTTPFrame<UInt8>

    consuming func receive<Return, Failure: Error>(
        body: (consuming sending Reader) async throws(Failure) -> Return
    ) async throws(Failure) -> Return
}

Same as writer-side D.