Department of Earth System Science zender@uci.edu
University of California Voice: (949) 824-2987
Irvine, CA  92697-3100 Fax: (949) 824-3256

Contents

List of Tables

1 Introduction

The Earth System Modeling Facility (ESMF) is designed to run large climate simulation codes. This document describes some general usage pointers for the ESMF, then demonstrates how to install and run certain codes (Section  3).

2 Environment

Users must have and maintain standard environment files. Essentially these files provide path information and helpful shortcuts to the users and any batch scripts that invoke them. These files need not be overly complex.

2.1 Bash

The default login shell on the ESMF is Bash. Add these lines to your default standard shell file .bashrc, or .profile:

Many ESMF users will want to run models, such as CCSM, which use C Shell scripts. For these scripts to work in batch mode, users must have a valid .cshrc file as well.

Users with a .login file in their home directory should ensure that it executes error free.

Alternatively, users can change their login shell to csh or tcsh vis the standard ESMF procedure

3 Models

This section describes how to install and run some of the various climate, chemical, and biogeochemical models for which the ESMF is designed.

3.1 CCSM Component Models

This section documents how to install and run the component models which compose the fully coupled NCAR CCSM (Community Climate System Model). Running the fully coupled CCSM itself is described in Section  3.2. First we describe the preliminary system modifications common to all CCSM component models.

CCSM component models may run in Symmetric Multi-Processor (SMP), Single Program Multiple Data (SPMD), and hybrid (i.e., combined SMP and SPMD) modes. To take avantage of the SPMD capabilities, make sure you have the Message Passing Interface (MPI) installed. MPI is already installed on the ESMF. To install it on a Debian GNU/Linux machine use:

CCSM build scripts may require the GNU Make executable to be named gmake rather than make. gmake is installed in /usr/local/bin on the ESMF. To point the GNU make executable on a Debian GNU/Linux machine to gmake use, e.g.,

All CCSM input-output (I/O) is performed with the netCDF data model. netCDF is already installed on the ESMF. To install netCDF on a Debian GNU/Linux machine use:

3.1.1 CAM: Community Atmosphere Model

No documentation yet. Please contribute!

3.1.2 CLM: Community Land Model

The Community Land Model (CLM) is described by Dai et al. [2003]. Other important references are Bonan [2002], Bonan [1996], and Bonan [1998].

The most frequently used WWW sites pertinent to CLM are 1. ESMF Homepage 2. CLM Homepage 3. LSM (CLM’s predecessor) Homepage 4. CLM 2.1 Source Code 5. CLM 2.1 User’s Guide (CLMUG) 6. NCL CCSM Graphics 7. NCO netCDF Operators

Make ESMF account capable of running CLM offline:

3.2 CCSM: Community Climate System Model

In modifying the CCSM to run on the ESMF, it is helpful to know that the ESMF and the NCAR-supported machine bluesky have nearly identical hardware and software configurations. First, esmf and bluesky are both clusters of p655 (8-way) and p690 (32-way) nodes connected by Federation switches. Each 8-way esmf node in the compute queues has 16 GB RAM (the interactive 8-way node esmf04m, which is on the internet as esmf.ess.uci.edu, has twice as much RAM, 32 GB). The 32-way p690 node has 64 GB RAM. Hence, each computational CPU on the ESMF has 2 GB RAM available. bluesky has about twenty times as many nodes as esmf. Finally, esmf has faster CPUs than bluesky (1.5 GHz rather than 1.3 GHz).

3.2.1 ESMF-specific CCSM modifications

With this knowledge, we created machine files which describe the esmf cluster and queueing architecture to the CCSM. These files are based on the corresponding bluesky machine files distributed with the CCSM. Four machine-specific files are required to make the esmf a valid CCSM machine-type. These files are 1. Macros.AIX (ccsm3_0/models/bld/Macros.AIX) 2. batch.ibm.esmf (ccsm3_0/scripts/ccsm_utils/Machines/batch.ibm.esmf) 3. env.ibm.esmf (ccsm3_0/scripts/ccsm_utils/Machines/env.ibm.esmf) 4. run.ibm.esmf (ccsm3_0/scripts/ccsm_utils/Machines/run.ibm.esmf)

First, read the purpose of the ESMF customizations. This will be helpful to those attempting to port the CCSM to other machines (e.g., Linux clusters) at UCI. Then, copy the ESMF-customized files to your local CCSM source tree.

The file Macros.AIX controls the options the compiler uses to build the CCSM source code (both C and Fortran). The first change replaces the default NCAR netCDF library path, /usr/local/lib64/r4i4, with its ESMF equivalent, /usr/local/lib:

All ESMF system libraries use the 64-bit ABI where possible. Hence NCAR’s lib64 is simply the ESMF’s lib. Second, the default size for Fortran variables of type real is four bytes, i.e., single precision. The default size for Fortran variables of type integer is also four bytes. Newer scientific computing codes, such as CCSM, explicitly specify the sizes of integers and reals so the default sizes are never used. However, many old codes rely on compiler-defaults to specify the sizes of integer and real. Some applications expect other defaults, e.g., eight-byte real and four-byte integer or r8i4 in NCAR parlance. The ESMF uses the r4i4 standard for all libraries in /usr/local. This obsoletes the need to explicitly specify an equivalent to NCAR’s r4i4 subdirectory.

The file batch.ibm.esmf contains most of the default header that will be used by the CCSM run script when the script is run in batch mode. Here is where the default LoadLeveler commands are set. The ESMF uses the default CCSM settings for the IBM AIX machine bluesky except for the following changes:

1. Default class (i.e., queue) is com_rg8 rather than csl_rg${max_tasks_per_node}
2. Default wall_clock_limit is 21600 s (6 hr), rather than 7200 s.
3. Job accounting is disabled by commenting out the line containing jareport.

The file env.ibm.esmf contains environment variables which determine how the CCSM is built and where it looks for input data and stores output data. env.ibm.esmf is the same as env.ibm.bluesky with the following exceptions

1. GMAKE_J determines how many jobs (in effect, processors) the parallelization of the compilation process will request. This is set to eight but should be set to four if you are often compiling interactively (rather than in the compute queues).
2. DIN_LOC_ROOT is /datashare/inputdata/csm instead of /fs/cgd/csm/inputdata
3. DIN_LOC_ROOT_USER is /datashare/inputdata_user instead of /fs/cgd/csm/inputdata_user
4. DIN_LOC_MSROOT is /datashare/inputdata/csm instead of /CCSM/inputdata

Finally, the CCSM versions of env.ibm.* contain a subtle bug in the task geometry clauses. Essentially, C Shell syntax splits the two set statements that look like the body of many if-endif clauses into two. The first statement is inside the condition and the second is outside and thus always executed. This bug causes geometry statements to overwrite eachother. We re-coded env.ibm.esmf to avoid this confusion.

The file run.ibm.esmf is the template run script. The fully resolved run script will be created from this template. This template handles all the details of building, executing the model, as well as storing the output data and possibly continually re-submitting the job until completion. run.ibm.esmf is identical to run.ibm.bluesky.

Copy these three files to your local source tree:

Normally, no further customization is necessary. These esmf-customized defaults are sufficient to get started running the CCSM. However, advanced CCSM users will often need to further modify these esmf-customized defaults.

3.2.2 Creating a new CCSM case

We are now ready to create the new CCSM case, i.e., experiment. First, store the case name the $CASEID environment variable. This name should be clueful about the experiments’ purpose. Next, use the create_newcase script which will create environment variable files and generate the configure script for this experiment.

This will create the $CASEROOT directory. Before the model may be run, however, it is necessary to populate the $CASEROOT directory with the resolved scripts.

3.2.3 Resolved Scripts

Each $CASEID may have multiple resolved-scripts generated for it. The resolved scripts are customized for particular initialization sequences (e.g., restart runs), resolution, machine, and component sets. The configure command generates the resolved scripts:

3.2.4 Run Geometry

3.2.5 Testing CCSM

CCSM supports a sohphisticated test suite. For many purposes

3.2.6 Submitting Run Scripts

Submit the job to the ESMF LoadLeveler queues with llsubmit:

Check the status of the job in the queue with llq.

All CCSM component models create log files during execution. Examine these log files to ensure your job is proceeding smoothly. The coupler log is most informative about the overall progress of the simulation.

Log files are stamped with the job-submission time. The time-stamp has the format YYMMDD-HHMMSS. Hence each simulation has a unique identifer, such as 041229-190833 above. The time-stamp is consistent among all the model components in the same simulation. After the simulation completes, the model compresses the log files and stores them, along with the history tapes (i.e., output data) in /ptmp/${USER}/archive/${CASEID}.

3.2.7 Archiving Data

CCSM, by default, stores short term output files in

CCSM also has automated long term archiving capability. This can be customized to copy data and logs to any computer on the Internet. Once the ESMF disk farm is working, we will activate automatic long-term archiving to

References

Index