Skip to content

Data integration steps

Jump to bottom
Lubomír Doležal edited this page Sep 20, 2024 · 5 revisions

In this page we try to cover potential data sources and what options for integration can be explored independently. Following sections cover different ways that data is provided.

Data is provided as service

If the service is supported as described in the Resource section a branch can be created in the catalog repository adding a yaml file as described in the wiki. Assets linked in the yaml file need to be added to the assets repository.

As quick non extensive reference here is a list with examples yaml files:

  • Dataset available through SentinelHub - example
  • Dataset available through WMS (with time dimension) - example
  • Dataset available through VEDA (NASA) - example for tiles; example for cogs
  • Data available through GeoDB in eodash format - example
  • Data available through XCube Server - example
  • Data available through Marine Copernicus Data Store - example

Data is provided raw

If the data is not provided through a service but is accessible publicly and is small or in cloud optimized format and allows cross origin requests, integration can potentially be done by EOX and can be discussed. If one of these points does not apply the data needs to be copied (potentially reprocessed) and hosted through a supported service. Afterwards it can be integrated as any other supported endpoint.

Integration testing

In order to independently test data integration following steps can be done.

  1. Create a branch in the catalog repository where the new collection yaml has been added
  2. Add the collection file name (without extension) to the corresponding catalog definition
  3. Make sure any referenced assets have been added to the assets repository
  4. Create a pull request for the branch where the pull request text is (a space separated list) of the filename(s) (without extension) for the collections to be tested. For example for ESA SO2 and VEDA cogs it would be N1_SO2 N2_CO2_diff
  5. Check the automatic action was triggered and runs through successfully (it is referenced in the pull request but can also be found in the Actions page.
  6. Go to the staging or testing dashboard instance where you want to test the integration and add ?catalog=<branch_name> to the url. <branch_name> being the name you gave to the branch. For example https://testing-instance.eox.at/?catalog=integrationtest.
  7. If all looks as expected request EOX to merge the pull request
Clone this wiki locally