Prerequisites - local

Prerequisites - local#

Short introduction to EarthCODE#

ESA has developed EarthCODE, a Collaborative Open Development Environment for Earth System Science. The scope of EarthCODE is to enable adoption of FAIR and Open Science Principles throughout ESA-funded Earth Science activities and supporting the ESA Science Clusters, to deliver long-term persistence of data, code and documentation, aiding reproducibility, reuse and consumption of research outputs by a wider community. EarthCODE integrates 1) the Open Science Catalogue, making it effortless to discover novel research outcomes from ESA Science Clusters activities; 2) access to commercial platform services; and 3) a wealth of community resources and tools that help maintain data and research quality.

In this guide we provide steps on how to expoit resources provided in this first compontent related to data cataloguing and sharing.

The Open Science Catalogue is built on the Spatio Temporal Asset Catalog (STAC), which is a standard for describing geospatial data. Therefore new entries must conform to its specification. We’ve broken the process into six steps, contained within six notebooks. You can run the individual notebooks and commit the results via pull request to the open science catalog metadata repository.
Below you can find instructions about setting up the environment, running the notebooks and creating a pull request.

0.1 Install the earthcode library#

Option 1. - Existing enviroment#

If you want to install the earthcode library in an existing enviroment simply run: pip install earthcode

Option 2. - New environment#

If you want to create a new environment to run the library in you can run:

  1. git clone https://github.com/ESA-EarthCODE/earthcode-library.git

  2. Install pixi - https://pixi.sh/dev/installation/

  3. cd earthcode-library

  4. pixi install

  5. pixi run jupyter lab

Refer to Documentation published on https://esa-earthcode.github.io/earthcode-library/README.html

0.2 Create local copy of Open Science Catalog metadata#

You can add new content to the OSC via GitHub Pull Request. To do this, you need a to fork the OSC repository, embeded the new information into the existing catalog and merge. The steps below describe the process.

⚠️ Important

All steps below must be executed in a terminal (command-line shell).
Do not run these commands inside a Jupyter notebook or any other notebook environment.

  1. (if needed) Install git & create a GitHub account

  2. Fork the open science catalog repository on github - ESA-EarthCODE/open-science-catalog-metadata

  3. Clone your forked repository
    git clone https://github.com/your-gh-username/open-science-catalog-metadata.git

  4. Create a new branch in the local clone

    • Set the current workspace to your local clone of the open science catalog metadata repository: cd open-science-catalog-metadata/

    • Create a new branch: git checkout -b project_branch

✅ You are now on a new branch and ready to start adding or editing OSC metadata.

0.3 Sequence of actions#

To process your data and publish it on Open Science Data Catalogue, we need six things from you to get started.
After completing Step 0.2 (above), run notebooks in suggested order and follow the instructions inside.

  1. Information about your ESA (EO Programme) funded Project >>> guide/1.Project.ipynb,

  2. Information about your Product (metadata and description) >>> guide/2.0.Product.ipynb

  3. Generate STAC Items for your dataset >>> guide/2.2.Product_files_self_hosted.ipynb

  4. Request permanent storage of data on ESA cloud storage >>> guide/2.1.Product_files_PRR.ipynb.

  5. Information about the Workflow you used to generate the data >>> guide/3.Workflow.ipynb

  6. Information about the Experiment that generated the dataset >>> guide/4.Experiment.ipynb

  7. Open a pull request against the main OSC repository (section 0.4 below)

0.4 Opening a GitHub PR#

Each step which refer to metadata generation and contribution to Open Science Catalogue metadata repsitory (notebooks 1-2 and 4-5) end with validation of the entries against the entire repository. This might take few minutes, depending on the number of changes you make to the catalogue. Once the validation is successfull, you are ready to request changes into existing open-science-catalog-metadata meaning being able to contribute with your project, products and workflows to the catalogue.

⚠️ Important

All steps below must be executed in a terminal (command-line shell).
Do not run these commands inside a Jupyter notebook or any other notebook environment.

By using the terminal:

  1. Check if the new entry has been added to the repository by typing: git status

  2. Add all changes to the commit: git add .

  3. Create commit to the newly created branch on your local copy of repository:
    git commit -m "Adding new product\_v2.0"

  4. Push the changes to your fork:
    git push origin \<branch-name\>

  5. Open a pull request against the main open science catalog repository

    • now open your browser and go to the github website (link appears automatically in the terminal command line)

    • Automatically you will be redirected to New Pull Request website. Compare changes across two branches between your branche in forked repository and main open-science-catalog-metadata repository.

    • update the title of the Pull Request and give it meaningful description

    • Click on “Create Pull Request”. ✅ You now syncronised Pull Request from your local branch to the main branch and your inputs are ready to review.

⚠️ Extra: you can use gh pr create -f to create a Pull Request using Github Client: https://cli.github.com/

Check the status of your PR directly in GitHub#

After creation of Pull Request you should see it on the list: ESA-EarthCODE/open-science-catalog-metadata

Check the status of your PR under: ESA-EarthCODE/open-science-catalog-metadata

Changes to the OSC content will be reviewed by the EarthCODE Data Steward team. In case any changes are needed to your inputs, you will be contacted by the team.