The DKRZ CMIP Data Pool#
This is a beginners-level demonstration notebook and introduces you to the Data Pool at DKRZ. Based on the example of the recent phase 6 of the Coupled Model Intercomparison Project (CMIP6), you will learn
how you benefit from the CMIP Data Pool (CDP)
how to approach CMIP data
how to use the python packages
intake-esm,xarrayandpandasto investigate the CMIP Data Pool
This notebook can be executed on DKRZ’s jupyterhub platform. For a detailled introduction into jupyterhub and intake, we recommend the DKRZ tech talks
jupyterhub by Sofiane Bendoukha
intake by Aaron Spring
Customizing the code inside, however, only requires basic python knowledge.
Introduction#
The Scientific Steering Commitee has thankfully granted a disk space on lustre file system of 5PB for the CMIP Data Pool for 2023. Started in 2016, DKRZ runs and maintains this common storage place.
📢 The DKRZ CMIP data pool contains often needed flagship collections of climate model data, is hosted as part of the DKRZ data infrastructure and supports scientists in high volume climate data collection, access and processing.
The notebook sources for the doc pages are available in this gitlab-repo
Important news and updates will be announced
on the DKRZ user portal
via a mailing list. Subscribe for ✉ cmip-data-poolATlists.dkrz.de
⭐ Highlight CDP climate model data collections are:
CMIP6: In May 2021, DKRZ provides Europe’s largest data pool with an amount of 4 PB for the recent phase of the Coupled Model Intercomparison Project
CORDEX: The size of data for the Coordinated Regional Downscaling Experiment is about 600TB over different projects.
CMIP5: The fifth phase of CMIP.
An example of a project which is also in the data pool, but not included in the term CMIP6⁺:
ERA5: Weather data from the European Centre for Medium-Range Weather Forecasts by re-analysed and homogenised observation data.
from IPython.display import HTML, display, Markdown, IFrame
display(Markdown("Time series of three different data pool disk space measures. DKRZ has published about 1.5 PB, 2.5 PB are replicated data from other data nodes. An average CMIP6 dataset contains about 5 files and covers 4GB."))
IFrame(src="https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/Pool-Statistics/pool-timeseries-hvplot.html",width="900",height="550",frameborder="0")
Time series of three different data pool disk space measures. DKRZ has published about 1.5 PB, 2.5 PB are replicated data from other data nodes. An average CMIP6 dataset contains about 5 files and covers 4GB.
display(Markdown("We develop, prepare and provide [jupyter notebook demonstrations](https://gitlab.dkrz.de/data-infrastructure-services/tutorials-and-use-cases) <br> "
"- as tutorials for software packages and applications *starting from scratch* </br>"
"- for more frequent use cases like the plot of `tas` of one member of two experiments and simulated by the German ESMs."))
IFrame(src="https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/plots/globalmean-yearlymean-tas.html",width="1000",height="650",frameborder="0")
We develop, prepare and provide jupyter notebook demonstrations
- as tutorials for software packages and applications starting from scratch - for more frequent use cases like the plot of tas of one member of two experiments and simulated by the German ESMs.
Why do we host the CDP? 🤔#
👉 The key benefit of the data pool is that the data is available on lustre (/work) so that All DKRZ users with a current account have access. There is less need for local copies or data downloads. 👈
Where can I find the data pool? 🐕#
The Data pool can be accessed from different portals.
Server-side on the file system e.g. under
/pool/data/CMIP6/dataAll levante users with a current account have permission to do that.
Fastest way to work with the data
#Browsing with linux commands
ls /pool/data/CMIP6/data/ -x
echo ""
#For which MIPs did MPI-ESM1-2-XR produce data for?
find /pool/data/CMIP6/data/ -maxdepth 3 -name MPI-ESM1-2-XR -type d
#Using the FreVA CMIP-ESMVal tool
module load cmip6-dicad/1.0
freva --databrowser --help
Web-based from remote: 🕸
Published data via the fail-safe Earth System Grid Federation data portal
Partly available in the Copernicus Climate Data Store
Regularly updated Intake-esm catalogs made publically available in the cloud or in
/pool/data/Catalogs📒
Understanding CMIP6 data#
🧑🏫 The goal of CMIP6
In order to evaluate and compare climate models, a globally organized intercomparison project is periodically conducted. CMIP6 tackles three major questions:
How does the Earth system respond to forcing? 🚂
What are the origins and consequences of systematic model biases? 🐞
How can we assess future climate changes given internal climate variability, predictability, and uncertainties and scenarios? 🌡
Metadata: Required Attributes and Controlled Vocabularies#
CDP data is self-descriptive as it contains extensive and controlled metadata. This metadata is prepared in the search facets of the data portals and catalogs.
📜
Besides the technical requirements, the CMIP data standard defines required attributes in so called Controlled Vocabularies (CV). While some values are predefined, models and institutions have to be registered to become a valid value of corresponding attributes. For many attributes, both a short form with _id and a longer description exist.
Important required attributes:
activity_id: A CMIP6-endorsed MIP that investigates a specific research question. It definesexperiments and requests data for it.source_id: An ID for the Earth System Model used to produce the data.experiment_id: The experiment which was conducted by thesource_id.member_id: The ensemble simulation member of theexperiment_id. All members should be statistically equal.
Investigating the CMIP6 data pool with intake-esm ⛵#
Features
display catalogs as clearly structured tables inside jupyter notebooks for easy investigation
import intake
cloudpath=["https://www.dkrz.de/s/intake"]
poolpath="/pool/data/Catalogs/dkrz_cmip6_disk.json"
#cdp = intake.open_catalog(pagespath)
#col = cdp.dkrz_cmip6_disk
col = intake.open_esm_datastore("/work/ik1017/Catalogs/dkrz_cmip6_disk.json")
col.df.head()
/mnt/root_disk3/gitlab-runner/.conda/envs/mambaenv/lib/python3.11/site-packages/intake_esm/cat.py:264: DtypeWarning: Columns (21,22,23) have mixed types. Specify dtype option on import or set low_memory=False.
df = pd.read_csv(
| activity_id | institution_id | source_id | experiment_id | member_id | table_id | variable_id | grid_label | dcpp_init_year | version | ... | frequency | time_reduction | long_name | units | realm | level_type | time_min | time_max | format | uri | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | AerChemMIP | BCC | BCC-ESM1 | hist-piAer | r1i1p1f1 | AERmon | c2h6 | gn | NaN | v20200511 | ... | mon | mean | C2H6 Volume Mixing Ratio | mol mol-1 | aerosol | alevel | 185001.0 | 201412 | netcdf | /work/ik1017/CMIP6/data/CMIP6/AerChemMIP/BCC/B... |
| 1 | AerChemMIP | BCC | BCC-ESM1 | hist-piAer | r1i1p1f1 | AERmon | c2h6 | gn | NaN | v20200511 | ... | mon | mean | C2H6 Volume Mixing Ratio | mol mol-1 | aerosol | alevel | 185001.0 | 201412.nc.modified | netcdf | /work/ik1017/CMIP6/data/CMIP6/AerChemMIP/BCC/B... |
| 2 | AerChemMIP | BCC | BCC-ESM1 | hist-piAer | r1i1p1f1 | AERmon | c3h6 | gn | NaN | v20200511 | ... | mon | mean | C3H6 volume mixing ratio | mol mol-1 | aerosol | alevel | 185001.0 | 201412 | netcdf | /work/ik1017/CMIP6/data/CMIP6/AerChemMIP/BCC/B... |
| 3 | AerChemMIP | BCC | BCC-ESM1 | hist-piAer | r1i1p1f1 | AERmon | c3h8 | gn | NaN | v20200511 | ... | mon | mean | C3H8 volume mixing ratio | mol mol-1 | aerosol | alevel | 185001.0 | 201412 | netcdf | /work/ik1017/CMIP6/data/CMIP6/AerChemMIP/BCC/B... |
| 4 | AerChemMIP | BCC | BCC-ESM1 | hist-piAer | r1i1p1f1 | AERmon | cdnc | gn | NaN | v20200522 | ... | mon | mean | Cloud Liquid Droplet Number Concentration | m-3 | aerosol | alevel | 185001.0 | 201412 | netcdf | /work/ik1017/CMIP6/data/CMIP6/AerChemMIP/BCC/B... |
5 rows × 26 columns
Features
browse through the catalog and select your data without being on the pool file system
⇨ A pythonic reproducable alternative compared to complex find commands or GUI searches. No need for Filesystems and filenames.
tas = col.search(experiment_id="historical", source_id="MPI-ESM1-2-HR", variable_id="tas", table_id="Amon", member_id="r1i1p1f1")
tas
/work/ik1017/Catalogs/dkrz_cmip6_disk catalog with 1 dataset(s) from 33 asset(s):
| unique | |
|---|---|
| activity_id | 1 |
| institution_id | 1 |
| source_id | 1 |
| experiment_id | 1 |
| member_id | 1 |
| table_id | 1 |
| variable_id | 1 |
| grid_label | 1 |
| dcpp_init_year | 0 |
| version | 1 |
| time_range | 33 |
| path | 33 |
| opendap_url | 33 |
| project | 1 |
| simulation_id | 1 |
| grid_id | 1 |
| frequency | 1 |
| time_reduction | 1 |
| long_name | 1 |
| units | 1 |
| realm | 1 |
| level_type | 0 |
| time_min | 33 |
| time_max | 33 |
| format | 1 |
| uri | 33 |
| derived_variable_id | 0 |
Features
open climate data in an analysis ready dictionary of
xarraydatasets
Forget about temporary merging and reformatting steps!
tas.to_dataset_dict()
--> The keys in the returned dictionary of datasets are constructed as follows:
'activity_id.source_id.experiment_id.table_id.grid_label'
{'CMIP.MPI-ESM1-2-HR.historical.Amon.gn': <xarray.Dataset>
Dimensions: (time: 1980, bnds: 2, lat: 192, lon: 384, member_id: 1,
dcpp_init_year: 1)
Coordinates:
* time (time) datetime64[ns] 1850-01-16T12:00:00 ... 2014-12-16T...
time_bnds (time, bnds) datetime64[ns] dask.array<chunksize=(60, 2), meta=np.ndarray>
* lat (lat) float64 -89.28 -88.36 -87.42 ... 87.42 88.36 89.28
lat_bnds (lat, bnds) float64 dask.array<chunksize=(192, 2), meta=np.ndarray>
* lon (lon) float64 0.0 0.9375 1.875 2.812 ... 357.2 358.1 359.1
lon_bnds (lon, bnds) float64 dask.array<chunksize=(384, 2), meta=np.ndarray>
height float64 2.0
* member_id (member_id) object 'r1i1p1f1'
* dcpp_init_year (dcpp_init_year) float64 nan
Dimensions without coordinates: bnds
Data variables:
tas (member_id, dcpp_init_year, time, lat, lon) float32 dask.array<chunksize=(1, 1, 60, 192, 384), meta=np.ndarray>
Attributes: (12/64)
Conventions: CF-1.7 CMIP-6.2
activity_id: CMIP
branch_method: standard
branch_time_in_child: 0.0
branch_time_in_parent: 0.0
contact: cmip6-mpi-esm@dkrz.de
... ...
intake_esm_attrs:time_reduction: mean
intake_esm_attrs:long_name: Near-Surface Air Temperature
intake_esm_attrs:units: K
intake_esm_attrs:realm: atmos
intake_esm_attrs:_data_format_: netcdf
intake_esm_dataset_key: CMIP.MPI-ESM1-2-HR.historical.Amon.gn}
Intake best practises:#
Intakecan make your scripts reusable.Instead of working with local copy or editions of files, always start from a globally defined catalog which everyone can access
Save the subset of the catalog which you work on as a new catalog instead of a subset of files
Check for new ingests by just repeating your script - it will open the most recent catalog.
Only load datasets with
to_dataset_dictinto xarrray which do not exceed your memory limits
Let’s get an overview over the CMIP6 Data pool by
finding the number of unique values of attributes
group and plot the names and sizes of different entries
The resulting statistics is about the percentage of File numbers.
unique_activites=col.unique("activity_id")
print(list(unique_activites["activity_id"].values()))
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[6], line 1
----> 1 unique_activites=col.unique("activity_id")
2 print(list(unique_activites["activity_id"].values()))
TypeError: esm_datastore.unique() takes 1 positional argument but 2 were given
def pieplot(gbyelem) :
#groupby, sort and select the ten largest
size = col.df.groupby([gbyelem]).size().sort_values(ascending=False)
size10 = size.nlargest(10)
#Sum all others as 10th entry
size10[9] = sum(size[9:])
size10.rename(index={size10.index.values[9]:'all other'},inplace=True)
#return a pie plot
return size10.plot.pie(figsize=(18,8),ylabel='',autopct='%.2f', fontsize=16)
pieplot("activity_id")
<Axes: >
unique_sources=col.unique("source_id")
print("Number of unique earth system models in the cmip6 data pool: "+str(list(unique_sources["source_id"].values())[0]))
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[9], line 1
----> 1 unique_sources=col.unique("source_id")
2 print("Number of unique earth system models in the cmip6 data pool: "+str(list(unique_sources["source_id"].values())[0]))
TypeError: esm_datastore.unique() takes 1 positional argument but 2 were given
pieplot("source_id")
<Axes: >
unique_members=col.unique("member_id")
list(unique_members["member_id"].values())[1][0:3]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[11], line 1
----> 1 unique_members=col.unique("member_id")
2 list(unique_members["member_id"].values())[1][0:3]
TypeError: esm_datastore.unique() takes 1 positional argument but 2 were given
Data Reference Syntax#
An atomic Dataset contains all files which cover the entire time span of a single variable of a single simulation. This can be multiple files in one.
The Data Reference Syntax (DRS) is a set of required attributes which uniquely identify and describe a dataset. The DRS usually includes all attributes used in the path templates so that both words are used synonymously. The DRS elements are arranged to a hierarchical path template for CMIP6:
CMIP6: mip_era/activity_id/institution_id/source_id/experiment_id/member_id/table_id/variable_id/grid_label/version
Be careful when browsing through the CMIP6 data tree!
Unique in CMIP6 data hierarchy:
experiment_id(only in oneactivity_id)variable_idintable_id: Both combined represent the CMIP VariableOnly one
versionfor one dataset should be published
# Searching for the MIP which defines the experiment 'historical':
cat = col.search(experiment_id="historical")
cat.unique("activity_id")
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[12], line 4
1 # Searching for the MIP which defines the experiment 'historical':
3 cat = col.search(experiment_id="historical")
----> 4 cat.unique("activity_id")
TypeError: esm_datastore.unique() takes 1 positional argument but 2 were given
# Searching for all tables which contain the variable 'tas':
cat = col.search(variable_id="tas")
cat.unique("table_id")
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[13], line 4
1 # Searching for all tables which contain the variable 'tas':
3 cat = col.search(variable_id="tas")
----> 4 cat.unique("table_id")
TypeError: esm_datastore.unique() takes 1 positional argument but 2 were given
Not Unique in CMIP6 data hierarchy:
institution_idfor bothsource_id+experiment_id( +member_id)
No requirements for member_id
# Searching for all institution_ids which uses the model 'MPI-ESM1-2-HR' to produce 'ssp585' results:
cat = col.search(source_id="MPI-ESM1-2-HR", experiment_id="ssp585")
cat.unique("institution_id")
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[14], line 4
1 # Searching for all institution_ids which uses the model 'MPI-ESM1-2-HR' to produce 'ssp585' results:
3 cat = col.search(source_id="MPI-ESM1-2-HR", experiment_id="ssp585")
----> 4 cat.unique("institution_id")
TypeError: esm_datastore.unique() takes 1 positional argument but 2 were given
# Searching for all experiment_ids produced with ESM 'EC-Earth3' and as ensemble member 'r1i1p1f1':
cat = col.search(source_id="EC-Earth3", member_id="r1i1p1f1")
cat.unique("experiment_id")
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[15], line 4
1 # Searching for all experiment_ids produced with ESM 'EC-Earth3' and as ensemble member 'r1i1p1f1':
3 cat = col.search(source_id="EC-Earth3", member_id="r1i1p1f1")
----> 4 cat.unique("experiment_id")
TypeError: esm_datastore.unique() takes 1 positional argument but 2 were given
# Searching for all valid ensemble member_ids produced with ESM 'EC-Earth3' for experiment 'abrupt-4xCO2'
cat = col.search(source_id="EC-Earth3", experiment_id="abrupt-4xCO2")
cat.unique("member_id")
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[16], line 4
1 # Searching for all valid ensemble member_ids produced with ESM 'EC-Earth3' for experiment 'abrupt-4xCO2'
3 cat = col.search(source_id="EC-Earth3", experiment_id="abrupt-4xCO2")
----> 4 cat.unique("member_id")
TypeError: esm_datastore.unique() takes 1 positional argument but 2 were given
⇨ Do not search for institution_id, table_id and member_id unless you are sure about what you are doing.
Instead, begin to search for
experiment_id, source_id, variable_id.
How can I find the variables I need? 🔎#
Search for the matching
standard_name
Most of the data in the data pool is compliant to the Climate and Forecast Convention. This defines standard_names, which need to be assigned to variables as a variable attribute inside the data. As a reliable description of the variable, the standard_name is a bridge to the shorter variable identifier name in the data, the so-called short_name. This short name is saved in the data catalogs which can be searched.
Search for corresponding
short_names in the CMIP6 data request
E.g., you get many results for air_temperature. Multiple definitions for one ‘physical’ variable like air_temperature exist in CMIP which are mostly specific diagnostics of that variable like tasmin and tasmax. Sometimes, there is output for a specific level given as a variable, e.g. ta500. This can be the case if not all levels are requested for a specific frequency.
Best practice in ESGF
Search for the fitting
mip_table
Each mip_table is a combination of requirements for an output variable_id including
frequency
time cell methods (average or instantaneous)
vertical level (e.g. interpolated on pressure levels)
grid
realm (e.g. atmosphere model output or ocean model output)
This requirements are set according to the interest of the MIPs. Variables with the similar requirements are collected in one MIP-table which can be identified by table_id.
The data infrastructure for the DKRZ CDP#
In order to tackle the challenges of data provision and dissemination for a 4 PB repository, a state-of-the-art data infrastructure has been developed around that pool. In the following, we highlight three aspects of the data workflow.
You benefit from the DKRZ CDP because
its data is standardized and quality controlled 🛂
it is a curated, updated, published and catalogized data repository 👩🏭
it prevents data duplication and downloading into local workspaces which is inefficient, expensive and just a waste of storage resources 🗑
Data quality#
CMIP6 data is only available in a common and reliable Data format
No adaptions needed for output of specific models
Makes data interoperable 📠 enabling evaluation software products as, for example, ESMValTool
🏅 CMIP6 data was quality controlled before published with PrePARE
CMIP6 data is transparent about occuring errors
Search the errata data base for origins of suspicious analysis results ⚠
If you find an error, please inform the modeling group. Either via the contact in the citation or, if available, via the contact attriubte in the file.
Data publication#
Exentended documentation for simulation conducts provided in the ES-Doc data base
Persistent Identfier (PIDs) ensure long-term webaccess to dataset information
Citation information and DOIs for all published datasets easily retrievable
One method to retrieve a citation from the data is via the attribute further_info_url
import xarray
random_file=xarray.open_dataset(cat.df["uri"][0])
random_file.attrs["further_info_url"]
'https://furtherinfo.es-doc.org/CMIP6.EC-Earth-Consortium.EC-Earth3.abrupt-4xCO2.none.r3i1p1f1'
When using data provided in the framework of the DKRZ CMIP Data Pool as basis for a publication, we ask you to add the following text to the Acknowledgements-Section:
“We acknowledge the CMIP6 community for providing the climate model data, retained and globally distributed in the framework of the ESGF. The CMIP6 data of this study were replicated and made available for this study by the DKRZ.”
Contacts#
Requests for replication and retraction: data-poolATdkrz.de 🎫
News and updates will be announced
on the new DKRZ User Portal
via the mailing list ✉ cmip-data-poolATlists.dkrz.de
This notebook is a collaboration effort by the DM Data Infrastructure team.
🙂 Thank you for your attention!