Generating a STAC Collection for the PRR

Introduction¶

This notebook has been created to show the core steps required of EarthCODE users to upload their research outcomes to the ESA Project Results Repository (PRR). It focuses on generating metadata for a project with a single netcdf file.

PRR provides access to data, workflows, experiments and documentation from ESA Projects organised across Collections, accessible via the STAC API. Each Collection contains STAC Items, with their related Assets stored within the PRR storage. Scientists/commercial companies can access the PRR via the EarthCODE and APEx projects.

The STAC Specification, provides detailed explanation and more information on this metadata format.

In order to upload data to the ESA Project Results Repository (PRR) you have to generate a STAC Collection that is associated to your files. The STAC Collection provides metadata about your files and makes them searchable and machine readable. The metadata generation process is organised in four steps process:

1. Generate a root STAC Collection
2. Group your dataset files into STAC Items and STAC Assets
3. Add the Items to the Collection
4. Save the normalised Collection

The easiest way to generate all the required files is to use a STAC library, such as pystac or riostac. This library will take care of creating the links and formating the files in the correct way. In the examples below we are using pystac.

Have a look at the steps below and learn how to prepare your dataset to generate a valid STAC Collection. You will find all the steps descibed in the markdown cell, together with the example code (executable) to make this process easier. Please adjust the information in the fields required to describe your Collection and Items according to the comments, starting with : “#”

NOTE: Depending on the information that you put in the Assets or Items the code, you may get an error about an object not being json-serialisable. If this happens, you have to transform the problem field into an object that can be described using standard JSON. For example, transforming a numpy array into a list.

🌊 Example: 4DATLANTIC-OHC Project¶

The code below demonstrates how to perform the necessary steps using real data from the ESA Regional Initiative Project 4DATLANTIC-OHC. The project focuses on ocean heat content and provides monthly gridded Atlantic Ocean heat content change as well as OHC trends and their uncertainties.

🔗 Learn more about the project here: 4DATLANTIC-OHC – EO4Society
🔗 Check the project website: 4DATLANTIC-OHC – Website

This example is intended to help you understand the workflow and apply similar steps to your own Earth observation data analysis.

Import necessary Python libraries¶

You can create an example conda/miniconda enviroment to run the below code using:

conda create -n prr_stack_example pystac xarray shapely
conda activate prr_stack_example

1. Generate a root STAC collection¶

The root STAC Collection provides a general description of the enitre dataset, that you would like to store in ESA PRR. In the STAC Specification a Collection is defined as an extension of the STAC Catalog with additional information such as the extents, license, keywords, providers, etc that describe STAC Items that fall within the Collection.

In short: it behaves as the container to store the various Items that build up your dataset.

STAC Collection has some required fields that you need to provide in order to build its valid description. Most of these metadata fields should be extracted from your data. Please have a look at the example below.

{
  "type": "Collection", # Do not change
  "id": "", # add a unique variation of project name + dataset name 
  "stac_version": "1.1.0", # Do not change
  "title": "", # Meaningful title of your dataset
  "description": "", # General description of your dataset
  "extent": {
    "spatial": {
      "bbox": [
        [
          -180.0,
          -90.0,
          180.0,
          90.0
        ]
      ]
    }, # Spatial extent of your dataset. If you have multiple data files take the minimum bounding box that covers all.
    "temporal": {
      "interval": [
        [
          "1982-01-01T00:00:00Z",
          "2022-12-31T23:59:59Z"
        ] # Temporal extent of your dataset. If you have multiple data files take the minimum temporal range that covers all.
      ]
    }
  },
"license": "", # the license that applies to entire dataset
"links": [] # do not change

}

Example | Create Collection¶

2. Group your dataset files into STAC Items and STAC Assets¶

The second step is to describe the different files as Items and Assets. This is the most time-consuming step. There are multiple strategies for doing this and it is up to you to decide how to do it. The main consideration should be usability of the data.

For example:

• Microsoft Planatery Computer groups its Sentinel-2 data into Items which represent individual regions, and each Item has 13 Assets each representing a band - https://stacindex.org/catalogs/microsoft-pc#/43bjKKcJQfxYaT1ir3Ep6uENfjEoQrjkzhd2?cp=1&t=5 .

• The California Forest Observatory (on Google Earth Engine) groups its data into Items, where each Item represents a specific year, data type and resolution for the whole study area. Each Item has only one Asset ( dataset ) associated with it - https://stacindex.org/catalogs/forest-observatory#/4dGsSbK8F5jjmhRZYE6kjUMmgWCUKe6J2qqw?t=2.

• A More complex example from real-data from ESA-funded project: ESA Projects Results Repository, gives the researchers flexibility in terms on how their datasets will be grouped into Items and Assets. You may need to consider that the more Items you have in your Collection, the slower the browsing would be if the user would like to browse through the publicly open STAC Browser. Please have a look at one example, that provides one Sentinel-3 AMPLI Ice Sheet Elevation Collection with around 400 Items complemented by around 360 Assets each. https://eoresults.esa.int/browser/#/external/eoresults.esa.int/stac/collections/sentinel3-ampli-ice-sheet-elevation

• More general examples about creating STAC catalogs are available here - https://github.com/stac-utils/pystac/tree/main/docs/tutorials.

The easiest way to generate the required STAC Items is to copy over the metadata directly from your files.

Example | Open Dataset¶

Example | Create valid STAC Item from your product (nc)¶

3. Add the STAC Item to the STAC Collection¶

Adding the Items to the Collection is a single function call when using a library such as pystac.

4. Save the Collection¶

Again this step is a single function call.

Congratulations, you have created your first STAC Collection.
¶

Now, you have your results ready to be ingested into ESA PRR. To request data storage in ESA PRR, contact EarthCODE team at: earth-code@esa.int and provide following information:

• your project name
• total size of your dataset
• link to STAC Collection created together with associated Items (e.g. entire example_4datlantic folder) - can be provided as a .zip or link to online repository / GitHub public repository
• link to the datasets (access link to final outcomes of the project or assets)
• specify any restrictions related to the access of your dataset.
• in the email, do not forget to CC your ESA TO to acknowledge that the dataset will be imported into PRR.

Once the email is received, the EarthCODE team will make a request to publish your product into PRR on your behalf (in the future the self-ingestion system will be supported).

Once the collection is imported you will receive a dedicated URL to your products, which you can use to create the record on Open Science Data Catalogue to make your data discoverable or/and request a DOI for your dataset (at the moment this has to be done by external service of your choice).