Non-determinism: leading space characters in diff occasionally turn into non-breaking space

[mejedi]

go 1.22.5
github.com/google/go-cmp v0.6.0

I noticed that cmp.Diff alternates between regular and non-breaking space in +- column. One is way more likely to get regular space.

This behavior creates issues when output is compared verbatim, e.g. in testable examples:

func ExampleCmpRepr() {
    type foo struct{ B, A int }
    fmt.Printf("%#v\n", cmp.Diff(foo{A: 1}, foo{A: 2}))
    // Output: "  testnet.foo{\n  \tB: 0,\n- \tA: 1,\n+ \tA: 2,\n  }\n"
}

It fails occasionally with the following output. Using go test -bench=. somehow makes it more likely to manifest.

--- FAIL: ExampleCmpRepr (0.00s)
got:
"\u00a0\u00a0testnet.foo{\n\u00a0\u00a0\tB: 0,\n-\u00a0\tA: 1,\n+\u00a0\tA: 2,\n\u00a0\u00a0}\n"
want:
"  testnet.foo{\n  \tB: 0,\n- \tA: 1,\n+ \tA: 2,\n  }\n"

[dsnet]

Hi @mejedi, sorry to hear about the issue, but this is working as intended. The documentation on Diff says:

Do not depend on this output being stable. If you need the ability to programmatically interpret the difference, consider using a custom Reporter.

The output of Diff has in the past and will continue to change because it is intended to evolve to be more and more humanly readable. Codifying it's output in something like an example test would mean that the test is brittle and breaks whenever Diff's output changes again. The instability of the output was done to deliberately make it harder to accidentally assume that the output was stable.

[mejedi]

Hi @dsnet, thank you for the prompt reply.

At the moment, I use go-cmp for testing custom low-level network code. Go-cmp turned out great to verify packets. However, for an outsider, it is not immediately apparent what does it mean to diff network packets and how the output looks like (object representation is being diffed, not raw bytes, so that individual mismatching fields or entire layers show up).

What is the best way to showcase cmp.Diff output in docs?

Re intentional non-determinism: I agree with the idea. The current implementation however is rather hostile. Go test results are cached, AND the probabilities of different outcomes are very skewed, AND the difference in the output is hard to tell with a naked eye.

[dsnet]

What is the best way to showcase cmp.Diff output in docs?

Unfortunately there is no answer at the moment, but the ability to evolve the output is currently deemed a greater benefit than stability.

I've long thought about forking the current reporter implementation and codifying it as "stable". This way we can continue to evolve the default output, and also provide a stable output for use cases that need it. Output in example docs is a legitimate reason to want a stable format.

[saser]

I'm experimenting with writing an expect test library for Go (which are actually pretty similar to the testable examples), and printing diffs is a very useful thing you can do with expect tests. If the cmp.Diff output format changes the library has an easy way to automatically update all expected values, so I'm less concerned about instability, but determinism is necessary to prevent diff-printing tests from becoming flaky.

Is it an option to provide deterministic but not stable output? I'm thinking along the lines of what proto.MarshalOptions.Deterministic provides. (I see that an internal variable flags.Deterministic exists and is referenced to decide between regular and non-breaking spaces, so I guess I'm asking if you're willing to somehow expose that flag to users of the package.)