Mississippi Basin¶

Welcome to the real-world application tutorial! In this guide, we walk through a complete, end-to-end data pipeline using the Mississippi River Basin. You will learn how to obtain external land surface model data, couple it to the river network, and run a full RAPID2 simulation.

This tutorial is provided as an interactive python notebook (ipynb), and can be opend and run in your browser by clicking the Binder badge:

All commands used in this tutorial are written in bash and can also be run by copying them into your terminal. Note that the %%bash at the start of most code cells simply converts the cell from python to bash.

Within a cell, commands are chained together with &&, which means "only run the next command if the previous one succeeded". Each cell ends by printing a short [OK] message, so that message appears only when every step before it worked. If a step fails, the chain stops there, the cell reports an error, and no success message is printed. This behaves the same way when you copy the commands into your terminal.

Before starting, esnure that RAPID2 has been installed in your vistual environment (see the Quickstart Guide). If running this tutorial in Binder, this setup is done automatically. We can check that rapid2 is installed and verify the version:

%%bash
rapid2 --version && echo "[OK] - rapid2 is installed."

%%bash rapid2 --version && echo "[OK] - rapid2 is installed."

1. Get Static Parameter Files for the Mississippi River¶

First, let's create working directories where the inputs and outputs of RAPID2 will be saved.

%%bash
mkdir -p input/Tutorial \
    && mkdir -p output/Tutorial \
    && echo "[OK] - created input/Tutorial and output/Tutorial."

%%bash mkdir -p input/Tutorial \ && mkdir -p output/Tutorial \ && echo "[OK] - created input/Tutorial and output/Tutorial."

Next, we need to get six static parameter files for the Mississippi Basin. We'll be using prepared files from a Zenodo repository:

RAPID2 static files for global implementation with MERIT Basins v1.0 and GLDAS v2.0

This repository includes 61 zipped folders corresponding to the global Pfafstetter Level 2 hydrologic regions. For this tutorial, we'll use the pfaf_74 files, corresponding to the Mississippi River Basin. We can download MERIT_Basins_v1.0_pfaf_74_GLDAS_v2.0_RAPID_v2.0.zip using wget, and then extract the files using unzip.

%%bash
ZIP=MERIT_Basins_v1.0_pfaf_74_GLDAS_v2.0_RAPID_v2.0.zip
URL=https://zenodo.org/records/20672740/files/$ZIP

# --continue resumes a partially downloaded archive instead of starting over,
# and --tries/--waitretry ride out transient network errors rather than
# failing the cell. -o on unzip overwrites quietly so that re-running this
# cell never blocks on an interactive "replace?" prompt.
wget --continue \
     --tries=5 \
     --waitretry=10 \
     --retry-connrefused \
     --timeout=30 \
     -P input/Tutorial/ "$URL" \
    && unzip -o input/Tutorial/$ZIP -d input/Tutorial/ \
    && echo "[OK] - extracted $(ls input/Tutorial/*.parquet | wc -l) parquet files to input/Tutorial."

%%bash ZIP=MERIT_Basins_v1.0_pfaf_74_GLDAS_v2.0_RAPID_v2.0.zip URL=https://zenodo.org/records/20672740/files/$ZIP # --continue resumes a partially downloaded archive instead of starting over, # and --tries/--waitretry ride out transient network errors rather than # failing the cell. -o on unzip overwrites quietly so that re-running this # cell never blocks on an interactive "replace?" prompt. wget --continue \ --tries=5 \ --waitretry=10 \ --retry-connrefused \ --timeout=30 \ -P input/Tutorial/ "$URL" \ && unzip -o input/Tutorial/$ZIP -d input/Tutorial/ \ && echo "[OK] - extracted $(ls input/Tutorial/*.parquet | wc -l) parquet files to input/Tutorial."

After opening the folder, it contains six parquet files needed to run RAPID2. Here is a quick overview of the input files:

1. bas_pfaf_74_topo.parquet: List of reaches within the basin.
2. con_pfaf_74.parquet: Network topology of the reaches, representing upstream/downstream connectivity.
3. cpl_pfaf_74_GLDAS.parquet: Spatial linkage between the land surface model outputs and hydrography network.
4. crd_pfaf_74.parquet: Coordinates of each reach polyline.
5. kpr_pfaf_74_nrm.parquet: Parameter k that represents the flow wave travel time.
6. xpr_pfaf_74_nrm.parquet: Dimensionless parameter x that relates to flow wave attenuation.

Files 1-4 correspond to the hydrography of the MERIT-Hydro v0.7 Basins v1.0. dataset. Meanwhile, files 5-6 include reach-specific values of k and x, which are Muskingum routing parameters.

Note: It is possible to adjust the residence times related to the k and x parameters. Three different residence times are supported: low (low), medium (nrm), and high (high). These files can also be found on Zenodo within zipped folders corresponding to the Pfafstetter Level 2 basins (e.g., k_pfaf_ii_low.zip, x_pfaf_ii_hig.zip). For more information, please see Collins et al., 2024. Nature Geoscience.

2. Download Raw Runoff Data with dgldas2¶

The RAPID2 routing model requires runoff inputs from a land surface model. We'll use runoff data downloaded from the Global Land Data Assimilation System (GLDAS) using the dgldas2 command that is part of the rapid2 package.

A NASA Earthdata account is required to download the GLDAS data. If you have not yet done so, you may also need to accept an end-user license agreement from the NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC).

For this tutorial, we'll download GLDAS phase 2.1, model VIC, for 2010-01, using the following command.

⚠️ If running this tutorial in Binder, you first need to authenticate with NASA Earthdata using an Earthdata token. Generate one at urs.earthdata.nasa.gov under "Generate Token", then run the cell below and paste it at the prompt. If running everything in your terminal, the dgldas2 command will handle your login credentials and the cell below can be skipped.

⚠️ Because Binder runs whatever code is in the repository it points at, launch this tutorial only from the badge on the official rapid-hub repository.

# Skip this cell if running in terminal
import getpass
import os

import earthaccess

# Held in an environment variable, which the %%bash cells below inherit so that
# dgldas2 can authenticate.
os.environ["EARTHDATA_TOKEN"] = getpass.getpass("Earthdata token: ").strip()

if not os.environ["EARTHDATA_TOKEN"]:
    raise RuntimeError(
        "[ERROR] No token entered. Generate one at https://urs.earthdata.nasa.gov "
        "under 'Generate Token', then re-run this cell."
    )

auth = earthaccess.login(strategy="environment")

if not auth.authenticated:
    raise RuntimeError("[ERROR] Earthdata login failed - re-run this cell.")

print("[OK] - Earthdata token registered for this session.")

# Skip this cell if running in terminal import getpass import os import earthaccess # Held in an environment variable, which the %%bash cells below inherit so that # dgldas2 can authenticate. os.environ["EARTHDATA_TOKEN"] = getpass.getpass("Earthdata token: ").strip() if not os.environ["EARTHDATA_TOKEN"]: raise RuntimeError( "[ERROR] No token entered. Generate one at https://urs.earthdata.nasa.gov " "under 'Generate Token', then re-run this cell." ) auth = earthaccess.login(strategy="environment") if not auth.authenticated: raise RuntimeError("[ERROR] Earthdata login failed - re-run this cell.") print("[OK] - Earthdata token registered for this session.")

The following dgldas2 command will download runoff from GLDAS phase 2.1, using the VIC model, for January, 2010:

%%bash
# `dgldas2` retrieves hundreds of GLDAS granules from NASA GES DISC. Across
# that many network transfers an occasional transient failure is expected, and
# it can interrupt the command before every granule has been retrieved.
#
# Retrying is safe and cheap: granules already on disk are skipped, and
# dgldas2 returns immediately once the final file exists. So retry here with
# exponential backoff rather than asking the user to re-run the cell.

MAX_ATTEMPTS=5
DELAY=10
LSM=input/Tutorial/GLDAS_2.1_VIC_2010-01.nc4
DONE=no

echo "[INFO] - downloading GLDAS granules from NASA GES DISC. This can take"
echo "         several minutes; wait for the [OK] line below before"
echo "         running the next cell."

for ATTEMPT in $(seq 1 $MAX_ATTEMPTS); do
    echo "[INFO] - dgldas2 attempt $ATTEMPT of $MAX_ATTEMPTS ..."

    if dgldas2 \
        --phase 2.1 \
        --model VIC \
        --time 2010-01 \
        --land_surface_model \
        "$LSM"
    then
        DONE=yes
        break
    fi

    if [ "$ATTEMPT" -lt "$MAX_ATTEMPTS" ]; then
        echo "[WARNING] - attempt $ATTEMPT failed; retrying in ${DELAY}s"
        echo "            (granules already downloaded will be skipped)."
        sleep "$DELAY"
        DELAY=$((DELAY * 2))
    fi
done

if [ "$DONE" = yes ]; then
    echo "[OK] - downloaded $LSM ($(du -h "$LSM" | cut -f1))."
else
    echo "[ERROR] - dgldas2 failed after $MAX_ATTEMPTS attempts."
    echo "          If the errors above mention authorization, your Earthdata"
    echo "          token may be mistyped or expired - generate a new one and"
    echo "          re-run the login cell. Otherwise, confirm that you have"
    echo "          accepted the NASA GES DISC end-user license agreement."
    false
fi

%%bash # `dgldas2` retrieves hundreds of GLDAS granules from NASA GES DISC. Across # that many network transfers an occasional transient failure is expected, and # it can interrupt the command before every granule has been retrieved. # # Retrying is safe and cheap: granules already on disk are skipped, and # dgldas2 returns immediately once the final file exists. So retry here with # exponential backoff rather than asking the user to re-run the cell. MAX_ATTEMPTS=5 DELAY=10 LSM=input/Tutorial/GLDAS_2.1_VIC_2010-01.nc4 DONE=no echo "[INFO] - downloading GLDAS granules from NASA GES DISC. This can take" echo " several minutes; wait for the [OK] line below before" echo " running the next cell." for ATTEMPT in $(seq 1 $MAX_ATTEMPTS); do echo "[INFO] - dgldas2 attempt $ATTEMPT of $MAX_ATTEMPTS ..." if dgldas2 \ --phase 2.1 \ --model VIC \ --time 2010-01 \ --land_surface_model \ "$LSM" then DONE=yes break fi if [ "$ATTEMPT" -lt "$MAX_ATTEMPTS" ]; then echo "[WARNING] - attempt $ATTEMPT failed; retrying in ${DELAY}s" echo " (granules already downloaded will be skipped)." sleep "$DELAY" DELAY=$((DELAY * 2)) fi done if [ "$DONE" = yes ]; then echo "[OK] - downloaded $LSM ($(du -h "$LSM" | cut -f1))." else echo "[ERROR] - dgldas2 failed after $MAX_ATTEMPTS attempts." echo " If the errors above mention authorization, your Earthdata" echo " token may be mistyped or expired - generate a new one and" echo " re-run the login cell. Otherwise, confirm that you have" echo " accepted the NASA GES DISC end-user license agreement." false fi

The new file is placed in input/Tutorial. Other files are also automatically downloaded in that directory and removed after use.

Note: GLDAS 2.0 is also supported, as are the CLSM and NOAH models.

3. Coupling the Data with cpllsm¶

Once you have the raw land surface model data and the static river basin parameters, the two must be linked by mapping the runoff to the river network.. The cpllsm tool transforms the gridded runoff data into RAPID2's reach-based external inflow format. By providing the land surface model data, connectivity, coordinates, and coupling files, this tool generates the external inflow NetCDF file required for routing.

%%bash
cpllsm \
    --land_surface_model \
    input/Tutorial/GLDAS_2.1_VIC_2010-01.nc4 \
    --connectivity \
    input/Tutorial/con_pfaf_74.parquet \
    --coordinates \
    input/Tutorial/crd_pfaf_74.parquet \
    --coupling \
    input/Tutorial/cpl_pfaf_74_GLDAS.parquet \
    --external_inflow \
    input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4 \
    && echo "[OK] - wrote input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4."

%%bash cpllsm \ --land_surface_model \ input/Tutorial/GLDAS_2.1_VIC_2010-01.nc4 \ --connectivity \ input/Tutorial/con_pfaf_74.parquet \ --coordinates \ input/Tutorial/crd_pfaf_74.parquet \ --coupling \ input/Tutorial/cpl_pfaf_74_GLDAS.parquet \ --external_inflow \ input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4 \ && echo "[OK] - wrote input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4."

4. Create Cold Start File with zeroqinit¶

At the start of our simulation, we need to provide RAPID2 with initial values of discharge. We'll use the next command to create a cold start file with all zeroes.

%%bash
zeroqinit \
    --external_inflow \
    input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4 \
    --initial_outflow \
    input/Tutorial/Qinit_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4 \
    && echo "[OK] - wrote input/Tutorial/Qinit_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4."

%%bash zeroqinit \ --external_inflow \ input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4 \ --initial_outflow \ input/Tutorial/Qinit_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4 \ && echo "[OK] - wrote input/Tutorial/Qinit_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4."

5. Run the Routing Model with rapid2¶

RAPID2 uses a YAML configuration file (namelist) to specify input files, parameters, and output locations. IS_dtR is the routing time step (in seconds).

Before running the model, we need to generate the namelist file for our tutorial run. We can do this using the cat command to concatenate lines of text into a single .yml file.

%%bash
cat << 'EOF' > input/Tutorial/namelist_Tutorial.yml \
    && echo "[OK] - wrote input/Tutorial/namelist_Tutorial.yml."
# -----------------------------------------------------------------------------
# Mandatory input files
# -----------------------------------------------------------------------------
Qex_ncf: './input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4'
Q00_ncf: './input/Tutorial/Qinit_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4'

con_pqt: './input/Tutorial/con_pfaf_74.parquet'
kpr_pqt: './input/Tutorial/kpr_pfaf_74_nrm.parquet'
xpr_pqt: './input/Tutorial/xpr_pfaf_74_nrm.parquet'

bas_pqt: './input/Tutorial/bas_pfaf_74_topo.parquet'

# -----------------------------------------------------------------------------
# Mandatory values
# -----------------------------------------------------------------------------
IS_dtR: 900

# -----------------------------------------------------------------------------
# Mandatory output files
# -----------------------------------------------------------------------------
Qou_ncf: './output/Tutorial/Qout_pfaf_74_GLDAS_2.1_VIC_2010-01_tst.nc4'
Qfi_ncf: './output/Tutorial/Qfinal_pfaf_74_GLDAS_2.1_VIC_2010-01_tst.nc4'
EOF

%%bash cat << 'EOF' > input/Tutorial/namelist_Tutorial.yml \ && echo "[OK] - wrote input/Tutorial/namelist_Tutorial.yml." # ----------------------------------------------------------------------------- # Mandatory input files # ----------------------------------------------------------------------------- Qex_ncf: './input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4' Q00_ncf: './input/Tutorial/Qinit_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4' con_pqt: './input/Tutorial/con_pfaf_74.parquet' kpr_pqt: './input/Tutorial/kpr_pfaf_74_nrm.parquet' xpr_pqt: './input/Tutorial/xpr_pfaf_74_nrm.parquet' bas_pqt: './input/Tutorial/bas_pfaf_74_topo.parquet' # ----------------------------------------------------------------------------- # Mandatory values # ----------------------------------------------------------------------------- IS_dtR: 900 # ----------------------------------------------------------------------------- # Mandatory output files # ----------------------------------------------------------------------------- Qou_ncf: './output/Tutorial/Qout_pfaf_74_GLDAS_2.1_VIC_2010-01_tst.nc4' Qfi_ncf: './output/Tutorial/Qfinal_pfaf_74_GLDAS_2.1_VIC_2010-01_tst.nc4' EOF

The output namelist file should look like this:

# -----------------------------------------------------------------------------
# Mandatory input files
# -----------------------------------------------------------------------------
Qex_ncf: './input/Tutorial/Qext_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4'
Q00_ncf: './input/Tutorial/Qinit_pfaf_74_GLDAS_2.1_VIC_2010-01.nc4'

con_pqt: './input/Tutorial/con_pfaf_74.parquet'
kpr_pqt: './input/Tutorial/kpr_pfaf_74_nrm.parquet'
xpr_pqt: './input/Tutorial/xpr_pfaf_74_nrm.parquet'

bas_pqt: './input/Tutorial/bas_pfaf_74_topo.parquet'

# -----------------------------------------------------------------------------
# Mandatory values
# -----------------------------------------------------------------------------
IS_dtR: 900

# -----------------------------------------------------------------------------
# Mandatory output files
# -----------------------------------------------------------------------------
Qou_ncf: './output/Tutorial/Qout_pfaf_74_GLDAS_2.1_VIC_2010-01_tst.nc4'
Qfi_ncf: './output/Tutorial/Qfinal_pfaf_74_GLDAS_2.1_VIC_2010-01_tst.nc4'

Once the namelist file is created, it's time to run RAPID2! The basic command is:

%%bash
rapid2 --namelist input/Tutorial/namelist_Tutorial.yml \
    && echo "[OK] - simulation complete. Output files:" \
    && ls -lh output/Tutorial/

%%bash rapid2 --namelist input/Tutorial/namelist_Tutorial.yml \ && echo "[OK] - simulation complete. Output files:" \ && ls -lh output/Tutorial/

That's it!

Our output files were written to the output/Tutorial folder. Qout_pfaf_74_GLDAS_2.1_VIC_2010-01_tst.nc4 represents the discharge time series for the Mississippi River Basin in January 2010. Meanwhile, Qfinal_pfaf_74_GLDAS_2.1_VIC_2010-01_tst.nc4 gives us the final state for a model restart.

Note: To run RAPID2 for longer than one month, the simulation can be extended by using the Qfinal file from the previous time step (Qfi_ncf) as the Qinit file for the next time step (Q00_ncf). It is currently possible in RAPID2 to download up to 300 time steps of runoff data from GLDAS at once, which is about 37.5 days of data.