Fix flaky transformers_wav2vec2 test

[3manifold]

Revive cpu_threads in test_transformers_wav2vec2 similar to test_transformers_wav2vec2bert.

resolves #1997

[jordimas]

Do we know why this fixes the issue and why it started at some point?
I am trying to understand the issue myself, thanks

cc: @sssshhhhhh

[Purfview]

My guess, its behavior becomes non-deterministic when the thread count isn't explicitly set and different hardware triggered it.

[sssshhhhhh]

wav2vec with set threads already is still flaky tho. I guess it's some hardware thing
Here's the relevant logits; 96 has T 0.3 behind which isn't unreasonable to flip in int8

93: S 10.6604
[  5.8699, -17.1565, -17.1737, -16.9353,   0.9821,  -0.7297,   0.2576,
  -0.5730,  -1.1717,  -0.6090,  -0.6322,  -2.1852,  10.6604,  -2.4010,
  -2.3614,  -3.1571,  -1.1926,  -1.7402,  -4.0642,  -2.9550,   1.4997,
  -2.1394,  -0.7178,   0.5809,  -2.4853,  -2.5506,  -2.5225,  -4.3844,
  -4.1468,  -1.3011,  -5.5800,  -1.2218],
94: S 9.3430
[  7.3162, -19.6428, -19.5482, -19.5689,   1.0854,  -0.6690,   4.9634,
  -1.7996,  -1.8668,  -0.6434,  -1.2932,  -1.3266,   9.3430,  -3.9079,
  -2.8894,  -3.8125,  -1.7758,  -3.0734,  -4.0288,  -3.6915,   2.6494,
  -3.0250,  -1.6254,   0.3615,  -2.7883,  -1.4328,  -3.1949,  -4.1023,
  -4.3149,  -2.1690,  -5.8798,  -1.8926],
95: <pad> 8.3382, T 7.9935
[  8.3382, -21.5072, -21.3548, -21.4830,   1.8911,  -0.2359,   7.9935,
  -2.3772,  -2.1847,  -0.0386,  -1.7056,  -0.0380,   4.5368,  -3.5758,
  -3.2549,  -3.4112,  -1.0526,  -3.0031,  -3.8084,  -3.9261,   1.6391,
  -3.7269,  -3.0971,  -1.0801,  -3.5790,  -1.2856,  -3.0240,  -4.2499,
  -3.6436,  -3.5346,  -6.0521,  -2.7278],
96: <pad> 8.7053, T 8.4424
[  8.7053, -22.7372, -22.5332, -22.6827,   1.6886,   0.7675,   8.4424,
  -1.3929,  -1.6288,   0.0388,  -1.7864,   1.6052,   0.9392,  -2.0094,
  -1.9775,  -2.3847,  -1.8057,  -2.7417,  -2.7039,  -4.5188,  -0.4568,
  -4.8336,  -3.1781,  -2.6786,  -4.4156,  -1.7055,  -3.0976,  -4.2438,
  -4.8969,  -5.8820,  -7.7593,  -4.8945],
97: E 6.8019
[  4.1432, -17.8858, -17.4800, -17.5025,  -1.0175,   6.8019,   2.8335,
    4.8259,   4.8419,   3.4145,  -3.6240,  -0.1039,  -1.4034,  -3.1419,
    2.0505,  -2.5247,   1.3074,  -0.0567,  -2.5088,  -4.2908,  -3.2960,
  -4.8016,  -3.5715,  -2.3134,  -4.3975,  -3.1872,  -1.2774,  -3.5743,
  -7.0010,  -6.4344,  -6.9160,  -5.6466],
98: L 8.5390, <pad> 8.4392
[  8.4392, -19.6191, -19.3853, -19.5242,  -0.6890,   3.4183,   1.0579,
    1.8278,   1.3909,  -0.0637,  -2.5247,  -2.3066,  -2.3664,  -0.8577,
    8.5390,  -2.5191,  -0.7955,  -3.0545,  -2.1300,  -2.7006,  -3.8802,
  -4.5860,  -2.9400,  -2.0070,  -3.2020,  -4.4432,  -2.2106,  -1.4179,
  -8.0574,  -5.6030,  -8.6484,  -5.2863],

[sssshhhhhh]

Maybe it could change to jfk, I checked every diff <0.5 can swap and it won't change the output

[3manifold]

Do we know why this fixes the issue and why it started at some point? I am trying to understand the issue myself, thanks

cc: @sssshhhhhh

@jordimas
As mentioned in #1997, there were flaky tests detected (again). I attempted to fix test_transformers_wav2vec2 (there was also an attempt to fix that in #1939) but did not work. Nevertheless, after discussing with @Purfview (#1997 (comment)), I revoked test_transformers_wav2vec2 and attempted to modify test_transformers_wav2vec2bert; this way we know that, at least, intra_threads>1 will not cause flakiness in the future.

[Purfview]

Maybe let's check against both variations, "APOSSTEL" and "APOSSEL"?

@3manifold did it passed the check when you reverted intra_threads=1 in the first commit?

[3manifold]

Maybe let's check against both variations, "APOSSTEL" and "APOSSEL"?

@3manifold did it passed the check when you reverted intra_threads=1 in the first commit?

All tests pass now. Although intra_threads=1 is a good starting point to alleviate flaky tests, there may be more changes needed.

I can add a commit to check for both "APOSSTEL" and "APOSSEL". Then, it is up to the reviewers to approve the change.

[3manifold]

@sssshhhhhh @jordimas I added a check for both "APOSSTEL" and "APOSSEL" in test_transformers_wav2vec2.