Intake catalogs#

In order to make the DKRZ CMIP data pool more FAIR, we support the python package intake-esm which allows you to use collections of climate data easily and fast.

We provide a tutorial here: https://tutorials.dkrz.de/intake.html

The offical intake-esm page: https://intake-esm.readthedocs.io/

Features

display catalogs as clearly structured tables inside jupyter notebooks for easy investigation

[1]:

import intake
col = intake.open_esm_datastore("/work/ik1017/Catalogs/dkrz_cmip6_disk.json")
col.df.head()

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 2
      1 import intake
----> 2 col = intake.open_esm_datastore("/work/ik1017/Catalogs/dkrz_cmip6_disk.json")
      3 col.df.head()

File /builds/data-infrastructure-services/cmip-data-pool/.cache/mamba/envs/datapoolservices/lib/python3.13/site-packages/intake/__init__.py:66, in __getattr__(attr)
     64 if attr[:5] == "open_":
     65     if attr[5:] in registry.drivers.enabled_plugins():
---> 66         driver = registry[attr[5:]]  # "open_..."
     67         return driver
     68     else:

File /builds/data-infrastructure-services/cmip-data-pool/.cache/mamba/envs/datapoolservices/lib/python3.13/site-packages/intake/source/__init__.py:32, in DriverRegistry.__getitem__(self, item)
     30 it = self.drivers.enabled_plugins()[item]
     31 if isinstance(it, entrypoints.EntryPoint):
---> 32     return it.load()
     33 elif isinstance(it, str):
     34     return import_name(it)

File /builds/data-infrastructure-services/cmip-data-pool/.cache/mamba/envs/datapoolservices/lib/python3.13/site-packages/entrypoints.py:79, in EntryPoint.load(self)
     76 def load(self):
     77     """Load the object to which this entry point refers.
     78     """
---> 79     mod = import_module(self.module_name)
     80     obj = mod
     81     if self.object_name:

File /builds/data-infrastructure-services/cmip-data-pool/.cache/mamba/envs/datapoolservices/lib/python3.13/importlib/__init__.py:88, in import_module(name, package)
     86             break
     87         level += 1
---> 88 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1387, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1360, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1310, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:488, in _call_with_frames_removed(f, *args, **kwds)

File <frozen importlib._bootstrap>:1387, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1360, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1331, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:935, in _load_unlocked(spec)

File <frozen importlib._bootstrap_external>:1023, in exec_module(self, module)

File <frozen importlib._bootstrap>:488, in _call_with_frames_removed(f, *args, **kwds)

File /builds/data-infrastructure-services/cmip-data-pool/.cache/mamba/envs/datapoolservices/lib/python3.13/site-packages/intake_esm/__init__.py:6
      4 # Import intake first to avoid circular imports during discovery.
      5 import intake
----> 6 from pkg_resources import DistributionNotFound, get_distribution
      8 from . import tutorial
      9 from .core import esm_datastore

ModuleNotFoundError: No module named 'pkg_resources'

[2]:

col.esmcat.description

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 col.esmcat.description

NameError: name 'col' is not defined

Features

browse through the catalog and select your data without being on the pool file system

⇨ A pythonic reproducable alternative compared to complex find commands or GUI searches. No need for Filesystems and filenames.

[3]:

tas = col.search(experiment_id="historical", source_id="MPI-ESM1-2-HR", variable_id="tas", table_id="Amon", member_id="r1i1p1f1")
tas

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 tas = col.search(experiment_id="historical", source_id="MPI-ESM1-2-HR", variable_id="tas", table_id="Amon", member_id="r1i1p1f1")
      2 tas

NameError: name 'col' is not defined

Features

open climate data in an analysis ready dictionary of xarray datasets

Forget about annoying temporary merging and reformatting steps!

[4]:

tas.to_dataset_dict(cdf_kwargs={"chunks":{"time":1}})

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 tas.to_dataset_dict(cdf_kwargs={"chunks":{"time":1}})

NameError: name 'tas' is not defined

Features

display catalogs as clearly structured tables inside jupyter notebooks for easy investigation
browse through the catalog and select your data without being on the pool file system
open climate data in an analysis ready dictionary of xarray datasets

⇨ intake-esm reduces the data access and data preparation tasks on analysists side

Catalog content#

The catalog is a combination of

a list of files (at dkrz compressed as .csv.gz) where each line contains a filepath as an index and column values to describe that file
- The columns of the catalog should be selected such that a dataset in the project’s data repository can be uniquely identified. I.e., all elements of the project’s Data Reference Syntax should be covered (See the project’s documentation for more information about the DRS) .
a .json formatted descriptor file for the list which contains additional settings which tell intake how to interprete the data.

According to our policy, both files have the same name and are available in the same directory.

[5]:

print("What is this catalog about? \n" + col.esmcat.description)
#
print("The path to the list of files: "+ col.esmcat.catalog_file)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 print("What is this catalog about? \n" + col.esmcat.description)
      2 #
      3 print("The path to the list of files: "+ col.esmcat.catalog_file)

NameError: name 'col' is not defined

Creation of the ``.csv.gz`` list :

A file list is created based on a find shell command on the project directory in the data pool.
For the column values, filenames and Pathes are parsed according to the project’s path_template and filename_template. These templates need to be constructed with attribute values requested and required by the project.
- Filenames that cannot be parsed are sorted out
Depending on the project, additional columns can be created by adding project’s specifications.
- E.g., for CMIP6, we added a OpenDAP column which allows users to access data from everywhere via http

Configuration of the ``.json`` descriptor:

Makes the catalog self-descriptive by defining all necessary information to understand the .csv.gz file

Specifications for the headers of the columns - in case of CMIP6, each column is linked to a Controlled Vocabulary.

[6]:

col.esmcat.attributes[0]

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[6], line 1
----> 1 col.esmcat.attributes[0]

NameError: name 'col' is not defined

Defines how to open the data as analysis ready as possible with the underlaying xarray tool:

which column of the .csv.gz file contains the path or link to the files
what is the data format
how to aggregate files to a dataset
- set a column to be used as a new dimension for the xarray by merge
- when opened a file, what is concat dimension?
- additional options for the open function

Jobs we do for you#

We make all catalogs available under /pool/data/Catalogs/ and in the cloud
We create and update the content of project’s catalogs regularly by running scripts which are automatically executed and called cronjobs. We set the creation frequency so that the data of the project is updated sufficently quickly.
- The updated catalog replaces the outdated one.
- The updated catalog is uploaded to the DKRZ swift cloud
- We plan to provide a catalog that tracks data which is removed by the update.

[7]:

!ls /work/ik1017/Catalogs/dkrz_*.json

ls: cannot access '/work/ik1017/Catalogs/dkrz_*.json': No such file or directory

[8]:

import pandas as pd
#pd.options.display.max_colwidth = 100
services = pd.DataFrame.from_dict({"CMIP6" : {
    "Update Frequency" : "Daily",
    "On cloud" : "Yes", #"https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/mistral-cmip6.json",
    "Path to catalog" : "/pool/data/Catalogs/dkrz_cmip6_disk.json",
    "OpenDAP" : "Yes",
    "Retraction Tracking" : "Yes",
    "Minimum required Memory" : "10GB",
}, "CMIP5": {
    "Update Frequency" : "On demand",
    "On cloud" : "Yes", #"https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/mistral-cmip5.json",
    "Path to catalog" : "/pool/data/Catalogs/dkrz_cmip5_disk.json",
    "OpenDAP" : "Yes",
    "Retraction Tracking" : "",
    "Minimum required Memory" : "5GB",
}, "CORDEX": {
    "Update Frequency" : "Monthly",
    "On cloud" : "Yes", #"https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/mistral-cordex.json",
    "Path to catalog" : "/pool/data/Catalogs/dkrz_cordex_disk.json",
    "OpenDAP" : "No",
    "Retraction Tracking" : "",
    "Minimum required Memory" : "5GB",
}, "ERA5": {
    "Update Frequency" : "On demand",
    "On cloud" : "Yes",
    "Path to catalog" : "/pool/data/Catalogs/dkrz_era5_disk.json",
    "OpenDAP" : "No",
    "Retraction Tracking" : "--",
    "Minimum required Memory" : "5GB",
}, "MPI-GE": {
    "Update Frequency" : "On demand",
    "On cloud" : "Yes",# "https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/mistral-MPI-GE.json
    "Path to catalog" : "/pool/data/Catalogs/dkrz_mpige_disk.json",
    "OpenDAP" : "",
    "Retraction Tracking" : "--",
    "Minimum required Memory" : "No minimum",
}}, orient  = "index")
servicestb=services.style.set_properties(**{
    'font-size': '14pt',
})

servicestb

[8]:

	Update Frequency	On cloud	Path to catalog	OpenDAP	Retraction Tracking	Minimum required Memory
CMIP6	Daily	Yes	/pool/data/Catalogs/dkrz_cmip6_disk.json	Yes	Yes	10GB
CMIP5	On demand	Yes	/pool/data/Catalogs/dkrz_cmip5_disk.json	Yes		5GB
CORDEX	Monthly	Yes	/pool/data/Catalogs/dkrz_cordex_disk.json	No		5GB
ERA5	On demand	Yes	/pool/data/Catalogs/dkrz_era5_disk.json	No	--	5GB
MPI-GE	On demand	Yes	/pool/data/Catalogs/dkrz_mpige_disk.json		--	No minimum

Best practises and recommendations:#

Intake can make your scripts reusable.
- Instead of working with local copy or editions of files, always start from a globally defined catalog which everyone can access.
- Save the subset of the catalog which you work on as a new catalog instead of a subset of files. It can be hard to find out why data is not included anymore in recent catalog versions, especially if retraction tracking is not enabled.
Intake helps you to avoid downloading data by reducing necessary temporary steps which can cause temporary output.
Check for new ingests by just repeating your script - it will open the most recent catalog.
Only load datasets with to_dataset_dict into xarrray with the argument cdf_kwargs={"chunks":{"time":1}}. Otherwise, the chunnk will let your memory exceed limits.

Technical requirements for usage#

Memory:
- Depending on the project’s volume, the catalogs can be big. If you need to work with the total catalog, you require at least 10GB memory.
- On jupyterhub.dkrz.de, start the notebook server with matching ressources.
Software:
- Intake works on the basis of xarray and pandas.
- On jupyterhub.dkrz.de , use one of the recent kernels:
  - unstable
  - bleeding edge

Load the catalog#

[9]:

#import intake
#collection = intake.open_esm_datastore(services["Path to catalog"][0])

Next step:#

https://tutorials.dkrz.de/intake.html

[ ]:

Intake catalogs

Contents

Intake catalogs#

Catalog content#

Jobs we do for you#

Best practises and recommendations:#

Technical requirements for usage#

Load the catalog#

Next step:#