Skip to content

[Test logic] FLAC and OPUS scenarios only verify byte-for-byte retransmission, not codec correctness #41

@balloob

Description

@balloob

Problem

server-initiated-flac and server-initiated-opus pass if and only if server_audio.sent_encoded_sha256 == client_audio.received_encoded_sha256 (see src/conformance/runner.py:626-637, _compare_encoded_audio_summaries).

That proves the wire bytes arrived intact. It does not prove:

  • The server's FLAC/OPUS encoder produced valid, decodable output for the source PCM.
  • The client's decoder can correctly turn the received bytes back into PCM that matches the source.

For OPUS this is especially misleading — OPUS is lossy, so the comparison can never assert perceptual closeness of the reconstructed PCM, it can only assert "whatever bytes came out of the encoder came into the client unmodified." A broken encoder that produced garbage would still pass.

The aiosendspin client actually already decodes FLAC into a PCM hasher (src/conformance/adapters/aiosendspin_client.py:267-271) but the resulting received_pcm_sha256 is never compared for audio-encoded-bytes scenarios — the value is dropped.

Proposed fix

  1. For server-initiated-flac: require the client to decode received FLAC into PCM and compare received_pcm_sha256 against the server's source_pcm_sha256. The lossless round-trip must hold.
  2. For server-initiated-opus: compare an acceptance band (e.g. PSNR / spectral distance / sample-count-based tolerance) against the source PCM, not just byte-identity. At minimum, assert that the client can decode the received bytes without error and produces PCM of the expected sample count.
  3. Keep the byte-identity check as an additional assertion, not the only one.

Spec references

  • "Player messages" — codec list includes OPUS (lossy) and FLAC (lossless)
  • "PCM Encoding Convention"

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions