GNU version of FATES SMS test resulting in differences to baseline with `ctsm5.1.dev049`

[glemieux]

Brief summary of bug

Differences are being seen for SMS_Lm13.1x1_brazil.I2000Clm50FatesCru.cheyenne_gnu.clm-FatesColdDef on Cheyenne between ctsm5.1.dev048 and ctsm5.1.dev049. The intel version of the same test passes B4B.

General bug information

CTSM version you are using: ctsm5.1.dev049

Does this bug cause significantly incorrect results in the model's science? [Yes / No] Yes?

Configurations affected: Fates single-point tests with gnu compiler.

Details of bug

The differences appear to occur around the 9th time step of the run and are affecting a number of variables. In particular the AREA_ prefixed history outputs, which suggests that the calculation of patch%area may be the primary affected variable which then is bleeding into downstream calculations.

Important details of your setup / configuration so we can reproduce the bug

Note that this tag also introduced MOSART issue 45. Adding frivinp_rtm = '/dev/null' to the user_nl_mosart list and rebuilding the test is necessary to get it to run. I don't think this is (direct) cause of the issue as other 1x1_brazil fates grid tests pass b4b that have this same modification (although those are also intel based tests).

The comparison between the tags uses fates tag sci.1.46.2_api.16.1.0 which is the default when using checkout_externals.

[ekluzek]

I've verified that I see this change and now have a baseline version of this test for both ctsm5.1.dev048 and ctsm5.1.dev049.

In comparing the namelists I only see the frivinp_rtm change that @glemieux mentions above and an explicit setting ofpio_netcdf_format = "64bit_offset". There are changes in module versions and a change in Macros.make.

[ekluzek]

OK, I ran this test in ctsm5.1.dev049 with the same env_mach_specific.xml, Macros.cmake, and Macros.make as ctsm5.1.dev048 and I get identical answers again. So this change is just because of a compiler version update between the two tags. As such I don't think we need to worry about this, especially since gnu isn't our main compiler. As long as the difference is only because of roundoff differences in optimization it really doesn't matter. The only thing that would matter is if those optimization differences lead to substantially different answers. Because this is only showing up for this one test it's reasonable that isn't the case here.

[ekluzek]

Looks like the compiler update came in with cime6.0.0, and this PR

ESMCI/cime#3985

[glemieux]

I ran a new ctsm5.1.dev046 baseline version of the same test mod with a different grid and then compared it to the same test setup using ctsm5.1.dev049. Doing so resulted in b4b comparison, so this seems like it might be specific to the 1x1_brazil grid for gnu only. I also ran a non-mosart compset dev046 baseline and dev049 case comparison for the 1x1_brazil grid to make sure it didn't have anything to do with the issue 45 workaround. This also failed the comparison with the same DIFF results.

I think that we can probably close this and label it wontfix given your comments, Erik and the fact that gridded gnu cases don't seem to be suffering the same issue.