Model Experiments for PalMod II: AWI-ESM Phase Space Analysis

This project contains model source code, various run configurations, boundary conditions, and analysis software for examining the behaviour of AWI-ESM 2.1 during various times throughout the last glacial cycle. The goal of the project is to determine a phase space for several key climate indicators.

Contents

• Getting a copy
• Updating
• Directory structure
• Included Boundary Conditions
• Setting up an experiment
• Orography Bootstrapping
• Branching off from an existing experiment
• Freshwater Forcing (Hosing) Experiments

Getting a copy

The project can be downloaded via git. Several preliminary steps may be needed.

1. On ollie, you need:

module load git

On mistral, the git version is already new enough.

2. Since there are several files included which use the git "large file storage" (git lfs), you might need to initialize that first. Run the following to do that:

git lfs install

3. It is probably a good idea to store your password, as otherwise you will need to enter it over and over until everything finishes downloading. You have two options:

A. Use a .netrc file. In your home directory, create a file with permissions 600:

# Creates an empty file if not already there:
touch ${HOME}/.netrc
# Gives only you read/write permission, everyone else has nothing
chmod 600 ${HOME}/.netrc

In this file, you can place your login credentials for both gitlab.awi.de and gitlab.dkrz.de:

machine gitlab.awi.de login pgierz password <PUT YOUR PASSWORD>
machine gitlab.dkrz.de login a270077 password <PUT YOUR PASSWORD>

The example above has Paul's account details, but of course you should use your own.

Otherwise, you could also use option B below:

B. Cache your git credentials. In this example, we save the password for 1 hour for each account you use:

git config --global credential.helper 'cache --timeout=3600'

4. Now you can download everything. We need to extend the standard clone command to ensure you download actual binary files rather than just checksums and pointers. Furthermore, some parts are within git submodules, particularly the climate model code and boundary conditions, we need to add options to get that as well:

git lfs clone --depth 1 --recurse-submodules https://gitlab.awi.de/paleodyn/Projects/PalMod/phase-space-analysis.git

Note that the lfs portion of git lfs clone could also just be git clone. Including lfs allows the program to download the large files in parallel, thus significantly decreasing the time you need to get your first copy. Newer versions of git do that by default.

If you run into any problems, there are likely permissions errors. Please let Paul know and he will help to fix it.

Note that the download is fairly large, and gives you about ~38 Gb

5. Install the esm-tools. This will be used to compile your model and control your runs. A copy is given to you in the ./software folder.

6. Compile your model. To do that, go into the ./model_codes/ folder and run esm_master comp-awiesm-2.1

cd ./model_codes
esm_master comp-awiesm-2.1

7. Install the direnv program and connect it to your shell. That will allow you to set "per project" environmental variables, and do some checks as soon as you go to the model folder. All of the run configurations are designed to be locally contained within in the project folder, as is identified with the environmental variable ${PROJECT_BASE}. This can be automatically set for you when you change into the folder by using the direnv program. Source code of this program is delivered with the remaining source code under the software/github.com/direnv/direnv/ folder. A pre-compiled, ready-to-use version for DKRZ's mistral is under software/bin. Please follow the directions here: https://direnv.net

Updating

You can update the project by running the following in the top level:

git pull

This should pull both the main files, as well as any boundary conditions that may have been updated.

Directory structure

• boundary_conditions: Contains various boundary conditions made with the standard boundary condition creation software. These correspond to the GLAC-1D 38ka BP, LGM, and 15ka BP reconstruction. Additionally, boundary conditions with "white blanket" ice sheets are provided.
• model_codes: Model binaries and source code
• experiment_run_configs: YAML files to be used with the esm-tools
• test_run_configs: YAML files to be used with the esm-tools, specifically for test_experiments
• preliminary_run_configs: YAML files to be used with the esm-tools, specifically for preliminary_experiments
• experiments: Experiments that can be usefully evaluated
• test_experiments: Experiments performed purely for technical testing purposes. No scientific value. Often, test runs will be used to get the actual YAML files to look correct.
• preliminary_experiments: Experiments performed so that scientifically useful ones can follow, e.g. stepwise adjustment to a new orogaphy.
• software: Helper software used for this project, e.g. the boundary condition creator. This is organized in such a way that the URL you would use to run git clone has the exact same form as the actual helper software. For examle, esm-tools is officially hosted under https://github.com/esm-tools/esm_tools. In the software folder, it is therefore under ./software/github.com/esm-tools/esm_tools.
• nonstandard_pool_files: Files that might be needed in FESOM, but are not in the standard Mistral pool.
• utils: Various utility scripts that may come in handy. Read the headers, they tell you what these do. Some have silly dinosaurs in the comments.

Please note that everything, excluding the experiment directories, is checked into git version control here.

Included Boundary Conditions

Boundary conditions are hosted here. Currently, the following are included in this project:

• GLAC1D 15ka
• GLAC1D 15ka White Blanket
• GLAC1D 38ka
• GLAC1D 38ka White Blanket
• PISM 38ka
• GLAC1D LGM
• GLAC1D LGM White Blanket
• PISM LGM

If we need more, let Uta know so she can prepare them.

Setting up an experiment

All experiments are controlled via the esm-tools, version 5. The table below shows the current versions of each component:

Experiment run configurations are created in two or three phases. To avoid cluttering your main experiments folder with technical tests, use instead the test_experiments folder for short, 1 or 2 year test simulations to ensure the configuration you are using does what you want. Once that is the case, you can then begin the experiment and change out the base_dir setting under the general section of your yaml file.

A special case is needed when you change the orography and land-sea mask. Here, we must first perform a "bootstrap" simulation, which starts from a Pre-Industrial orography and iterates towards the desired orography over one simulation year. This must be done to avoid numerical instability, as directly starting the model with a new set of boundary conditions for the geopotential height (variable geosp) and the orography (and corresponding gravity-wave drag parameterization variables, OROMEA and friends) leads to a lookup table overflow and the model crashes. This is described in detail in the next section.

Orography Bootstrapping

To perform an orography bootstrap experiment, please set the following in your YAML file for the echam section:

echam:
    add_input_sources:
            jansurf: /work/ollie/pgierz/PalModII/boundary_conditions/finished_files/echam/T63CORE2_jan_surf_with_PI_GEOSP.nc
            target_orography: /work/ollie/pgierz/PalModII/boundary_conditions/finished_files/echam/T63CORE2_jan_surf_target_orog.nc
    add_input_files:
            target_orography: target_orography
    add_input_in_work:
            target_orography: target_orography_echam6.nc

    add_namelist_changes:
        namelist.echam:
            submodelctl:
                lupdate_orog: True

You may use the utility script utils/create_orography_bootstrap.py to set everything up. Please read that first.. The next table summarizes what variables should come from which file. The "Timeslice" file, is, for example, boundary_conditions/GLAC1D_15ka/echam/T63CORE2_jan_surf.nc

For the file listed under echam.add_input_sources.jansurf

For the file listed under echam.add_input_sources.target_orography, include only the variables for timeslice for GEOSP and ORO*:

• GEOSP
• OROGAM
• OROMEA
• OROPIC
• OROSIG
• OROSTD
• OROTHE
• OROVAL

You can use the script in utils/ to prepare that. If instead you would like to do it via CDO on your own, this may be a starting point (also in utils/mk_orog_bootstrap_files.sh):

PROJECT_ROOT=/work/ba0989/a270077/PalModII/phase_space_analysis
target_unit24=${PROJECT_ROOT}/boundary_conditions/GLAC1D_15ka/echam/T63CORE2_jan_surf.nc
pi_unit24=${PROJECT_ROOT}/nonstandard_pool_files/fesom/mesh_CORE2_finaltopo_mean/tarfilesT63/input/echam6/T63CORE2_jan_surf.nc
mkdir create_bootstrap_file
cd create_bootstrap_file
mkdir target_vars pi_vars merged
cdo splitname ${target_unit24} target_vars/target_
cdo splitname ${pi_unit24} pi_vars/pi_
rm target_vars/target_ORO*.nc
rm target_vars/target_GEOSP.nc
mv target_vars/* merged
mv pi_vars/pi_GEOSP.nc merged
mv pi_vars/pi_ORO*nc merged
cdo merge merged/*nc initial_unit24.nc
cdo splitname ${target_unit24} target_vars/target_
rm target_vars/target_ALAKE.nc
rm target_vars/target_ALB.nc
rm target_vars/target_AZ0.nc
rm target_vars/target_FAO.nc
rm target_vars/target_FOREST.nc
rm target_vars/target_GLAC.nc
rm target_vars/target_SLF.nc
rm target_vars/target_SLM.nc
rm target_vars/target_SN.nc
rm target_vars/target_WSMX.nc
rm target_vars/target_WS.nc
cdo merge target_vars/target_* target_uni24.nc

A bootstrapping experiment could be run as a "preliminary experiment", so as not to clutter your main experiment folder. If you want to run a test experiment first to ensure everything is technically correct, that's fine too...

Branching off from an existing experiment

To branch off from another run (or from the pre-orography bootstrapping run), you need to include the following in your YAML files:

echam:
    lresume: 1
    ini_parent_exp_id: Pre_LGM_orog_boot_022
    fake_initial_date: "2001-01-01"
    ini_parent_date: "$(( ${fake_initial_date} - ${echam.time_step}seconds ))"
    ini_parent_dir: "/work/ba0989/a270077/PalModII/phase_space_analysis/restart_files_from_ollie/Pre_LGM_orog_boot_022/restart/echam/"

fesom:
    lresume: 1
    ini_parent_exp_id: Pre_LGM_orog_boot_022
    ini_parent_dir: "/work/ba0989/a270077/PalModII/phase_space_analysis/restart_files_from_ollie/Pre_LGM_orog_boot_022/restart/fesom/"

Note that the JSBACH and HDMODEL components do not need explicit sections, they follow suite from ECHAM6.

Freshwater Forcing (Hosing) Experiments

The newest version of AWI-ESM allows you to perform freshwater perturbation (hosing) experiments without modifying with your binary. To get that to work, you first need to install the esm-tools plugin gfw_creator. Install it like this:

$ pip install gfw_creator

You might need to include the --user flag after pip install if you don't have root permissions.

To actually do a hosing run, you need to add the following in your run configuration:

fesom:
    do_hosing: True
    hosing_type: "homogeneous"
    hosing_lat0: <lower_corner_lat>
    hosing_lon0: <lower_corner_lon>
    hosing_lat1: <upper_corner_lat>
    hosing_lon1: <upper_corner_lon>
    hosing_strength: 0.2  # Should be in Sv!

The run will generate a hosing file for you, and FESOM will read that. To examine the hosing file, you can look in your current work directory for gfw_atmo.nc. Please look at the gfw_atmo.nc file before starting your run to make sure it makes sense!

The name gfw_atmo.nc is chosen as the hosing, but technically speaking, is applied to the atmosphere model, and simply given to the oceanic component as "extra rainfall".