Problem
server-initiated-flac and server-initiated-opus pass if and only if server_audio.sent_encoded_sha256 == client_audio.received_encoded_sha256 (see src/conformance/runner.py:626-637, _compare_encoded_audio_summaries).
That proves the wire bytes arrived intact. It does not prove:
- The server's FLAC/OPUS encoder produced valid, decodable output for the source PCM.
- The client's decoder can correctly turn the received bytes back into PCM that matches the source.
For OPUS this is especially misleading — OPUS is lossy, so the comparison can never assert perceptual closeness of the reconstructed PCM, it can only assert "whatever bytes came out of the encoder came into the client unmodified." A broken encoder that produced garbage would still pass.
The aiosendspin client actually already decodes FLAC into a PCM hasher (src/conformance/adapters/aiosendspin_client.py:267-271) but the resulting received_pcm_sha256 is never compared for audio-encoded-bytes scenarios — the value is dropped.
Proposed fix
- For
server-initiated-flac: require the client to decode received FLAC into PCM and compare received_pcm_sha256 against the server's source_pcm_sha256. The lossless round-trip must hold.
- For
server-initiated-opus: compare an acceptance band (e.g. PSNR / spectral distance / sample-count-based tolerance) against the source PCM, not just byte-identity. At minimum, assert that the client can decode the received bytes without error and produces PCM of the expected sample count.
- Keep the byte-identity check as an additional assertion, not the only one.
Spec references
- "Player messages" — codec list includes OPUS (lossy) and FLAC (lossless)
- "PCM Encoding Convention"
Problem
server-initiated-flacandserver-initiated-opuspass if and only ifserver_audio.sent_encoded_sha256 == client_audio.received_encoded_sha256(seesrc/conformance/runner.py:626-637,_compare_encoded_audio_summaries).That proves the wire bytes arrived intact. It does not prove:
For OPUS this is especially misleading — OPUS is lossy, so the comparison can never assert perceptual closeness of the reconstructed PCM, it can only assert "whatever bytes came out of the encoder came into the client unmodified." A broken encoder that produced garbage would still pass.
The aiosendspin client actually already decodes FLAC into a PCM hasher (
src/conformance/adapters/aiosendspin_client.py:267-271) but the resultingreceived_pcm_sha256is never compared foraudio-encoded-bytesscenarios — the value is dropped.Proposed fix
server-initiated-flac: require the client to decode received FLAC into PCM and comparereceived_pcm_sha256against the server'ssource_pcm_sha256. The lossless round-trip must hold.server-initiated-opus: compare an acceptance band (e.g. PSNR / spectral distance / sample-count-based tolerance) against the source PCM, not just byte-identity. At minimum, assert that the client can decode the received bytes without error and produces PCM of the expected sample count.Spec references