Document how to customize the build options

[ekluzek]

I couldn't find a lot of documentation on how the CIME build works in the CIME User's Guide nor in README's or comments in the code.

From what I see the options for customization are:

1. Modify: Macros.cmake, Macros.make, or env_mach_specific.xml for your case (use --keep on case.setup to preserve your changes if you have to rerun case.setup)
2. Explicitly set any of these environment variables for the build:
   USER_SLIBS
   USER_CPPDEFS
   USER_INCLDIR
   USER_CMAKE_MODULE_PATH
   USER_CMAKE_OPTS
   USER_MKDEPENDS_OPTS
3. Customize the files in your ccs_config in your checkout for the given machine
4. Customize specfic file compilation by adding files to the Depends.$COMPILER file in ccs_config
5. Use your own .cime directory to customize your settings
   For "1" I just filed bugs: Add error checking to the --keep option to case.setup #4971 case.setup "--keep" option NOT working when it gets to the case.build step #4972

For "2" I showed that setting $USER_SLIBS does work correctly. But, haven't tried the others.

For "3" I've of course shown this works

I haven't done anything with "4", but obviously this is working.

I haven't tried "5" for a machine that ccs_config already has a definition for.

[ekluzek]

I'd like to hear from people:

• if I'm correct in above.?
• That I didn't miss another option that exists?
• I suggest documenting this in the wiki at first -- let me know if that's the right place (could be a discussion or in the CIME User's Guide)?
• I also think there should be some documentation on this in your case in the Makefile and/or maybe the Macros files or in the README.case -- how does that sound?
• Any other thoughts on the CIME build that people have?

Pinging a few folks: @billsacks @fischer-ncar @jgfouca @rljacob

[ekluzek]

Oh, another customization that's archived in the ccs_config is the Depends file for each compiler. Which allows you to add specific files to lists like these, which then have specific rules on compilation:

PERFOJBS
REDUCED_OPT_OBJS
REDUCED_ERRORCHKS_IN_DEBUG
REDUCED_PRECISION_OBJS
SHR_RANDNUM_FORT_OBJS
SHR_RANDNUM_C_OBJS
PUMAS_OBJS
RRTMGP_OBJS
CLUBB_OBJS

[jgfouca]

@ekluzek , when E3SM transitioned to a cmake build system, the behavior significantly diverged from what was there before and what CESM is using now. I've tried to keep this page up to date for us:
https://e3sm.atlassian.net/wiki/spaces/DOC/pages/1011942171/E3SM+Build+System+and+CMake

... but I don't know if any of that is useful to you.

[billsacks]

Thanks a lot for starting this discussion, @ekluzek , and for your comprehensive list of ways to customize the build. I hadn't been aware of some of these options myself, like the various USER_* env vars.

It feels like this belongs in the CIME User's Guide, but as @jgfouca points out, it's tricky that CESM and E3SM work differently.

What do people think about having some sections of the CIME User's Guide that only apply to certain models – either E3SM or CESM/NorESM? (Also including @mvertens in this discussion as we start to plan towards updating the CIME documentation for the upcoming CESM and NorESM releases.) In addition to this issue, this would also apply to other things like #1964. Based on discussion in #1964 , we were thinking at one point of having appendices for different models, but then later stopped considering that. Personally I feel like it could be better to put documentation like this in the main body rather than relegating it to an appendix, since it seems likely to get lost in an appendix, but would need a way to differentiate documentation that only applies to certain models. What do others think? We can also discuss on an upcoming CIME call.

[billsacks]

Discussion from today's CIME call: People are generally happy with having some model-specific documentation in the CIME documentation. We can have a note on a page saying if it just applies to certain models. Feeling is that it's better to put it within the main organizational structure of CIME documentation rather than as an appendix.

[rljacob]

Its useful to compare how these dirs appear in CESM vs. E3SM.

CESM

ChangeLog		bin/			components/		manage_externals/
ChangeLog_template	ccs_config/		describe_version*	share/
LICENSE.txt		cime/			doc/			tools/
README.rst		cime_config/		libraries/

CESM/ccs_config

README.md			config_grids_nuopc.xml		maps_nuopc.xml
component_grids_nuopc.xml	config_inputdata.xml		modelgrid_aliases_nuopc.xml
config_archive.xml		machines/

CESM/cime_config

README.md		config_pes.xml		testlist_allactive.xml
SystemTests/		config_tests.xml	testmods_dirs/
config_compsets.xml	testfiles/		usermods_dirs/

E3SM

AGENTS.md		cesm_mod_Ccase		driver-mct/		share/
AUTHORS			cime/			driver-moab/		site/
CITATION.cff		cime_config/		externals/		tools/
CONTRIBUTING.md		codemeta.json		mbcesm_mod_Ccase
LICENSE			components/		mkdocs.yaml
README.md		docs/			run_e3sm.template.sh*

E3SM/cime_config

__pycache__/		config_files.xml	customize/		testmods_dirs/
allactive/		config_grids.xml*	machines/		tests.py
config_archive.xml	config_inputdata.xml	scripts/		usermods