Failing prebeta tests.

[fischer-ncar]

What happened?

SMS_D_Ln9_P1280x1.ne0ARCTICGRISne30x8_ne0ARCTICGRISne30x8_mt12.FHIST.derecho_intel.cam-outfrq9s is failing with "floating invalid" error. I expect the test to run without errors.

What are the steps to reproduce the bug?

git clone https://github.com/ESCOMP/cesm cesm3
cd cesm3
git checkout cesm3_0_alpha08o
./bin/git-fleximod update -o
cd cime/scripts
qcmd -- ./create_test SMS_D_Ln9_P1280x1.ne0ARCTICGRISne30x8_ne0ARCTICGRISne30x8_mt12.FHIST.derecho_intel.cam-outfrq9s --walltime 00:30:00

What CAM tag were you using?

cesm3_0_alpha08o

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

/glade/derecho/scratch/fischer/t/cesm3_0_beta08_i/SMS_D_Ln9_P1280x1.ne0ARCTICGRISne30x8_ne0ARCTICGRISne30x8_mt12.FHIST.derecho_intel.cam-outfrq9s.GC.c3_0_b08o_int

Will you be addressing this bug yourself?

No

Extra info

Increasing the number of processors to 1920 will get the test to pass.

[oehmke]

Looking at the line that failed ( *element += factor * *value;), it looks like either factor or value is the problem. It looks like "factor" is a regridding weight and "value" is from the Field input to ESMF_FieldRegridStore(). Obviously, there are things that can be going wrong with either of those. However, Brian and I talked about it a bit and one possibility is that the input Field is uninitialized and that it's occasionally getting a NaN in it from the random bit patterns that are in the memory it gets. When this code runs the NaN then triggers the error. If someone who knows the code has a chance, could they check to see if the Field going into the ESMF_FieldRegridStore() call at line 510 in lnd_set_decomp_and_domain.F90 is getting initialized? (Or let me know and I can dig around.) Thanks!

[adamrher]

Just adding here an anecdote from the MCT days running SE-VR grids, that could possibly be related. We used to have to relax the tolerance EPS_AAREA to 1e-4 for VR grids. I always suspected it was due to problems with the coordinates in the SCRIP file or exodus grid file, but I don't think we ever tracked down the source of the problem. Once we moved to CMEPS, there was no EPS_AAREA variable and didn't have any issues that I'm aware of.