Test adding metopa and metopb as placeholder drafts to stac database #426

shmh40 · 2025-07-01T17:52:33Z

Description

Adding MHS on Metop-A and B to the STAC database, and Fengyun 3 MWHS, and ABI-GOES.

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Issue Number

Fixes #427

Code Compatibility

I have performed a self-review of my code

Code Performance and Testing

I ran the uv run train and (if necessary) uv run evaluate on a least one GPU node and it works
If the new feature introduces modifications at the config level, I have made sure to have notified the other software developers through Mattermost and updated the paths in the $WEATHER_GENERATOR_PRIVATE directory

Dependencies

I have ensured that the code is still pip-installable after the changes and runs
I have tested that new dependencies themselves are pip-installable.
I have not introduced new dependencies in the inference portion of the pipeline

Documentation

My code follows the style guidelines of this project
I have updated the documentation and docstrings to reflect the changes
I have added comments to my code, particularly in hard-to-understand areas

Additional Notes

shmh40 · 2025-07-01T17:59:16Z

Hey @iluise - this is very rough, I will fill in all the actual details/fix titles etc. - but looks ok to you?

iluise · 2025-07-03T08:00:59Z

stac/jsons/catalogue.json

+      {
+         "href": "https://raw.githubusercontent.com/ecmwf/WeatherGenerator/refs/heads/shmh40/dev/stac_test/stac/jsons/metopa.json",
+         "rel": "child",
+         "title": "metopa",


can we have the title capital as all the others?

Let's follow OSCAR's naming: https://space.oscar.wmo.int/satellites/view/metop_a, i.e. METOP-A

iluise · 2025-07-03T08:01:06Z

stac/jsons/catalogue.json

+      {
+         "href": "https://raw.githubusercontent.com/ecmwf/WeatherGenerator/refs/heads/shmh40/dev/stac_test/stac/jsons/metopb.json",
+         "rel": "child",
+         "title": "metopb",


iluise · 2025-07-03T08:03:56Z

stac/metopa.jsonnet

+    type: 'application/vnd+netcdf',
+    description: 'Observation dataset',
+    locations: [common.hpc.hpc2020, common.hpc.jsc],
+    size: '0.5 TB',


can we have it in GB? is this compressed?

Yes currently compressed - shall I unzip them on ATOS? I guess we are ok for inodes now?

Yes, we have plenty of space/inodes you can untar them there, thanks!

iluise · 2025-07-03T08:04:28Z

stac/metopa.jsonnet

+    description: 'Observation dataset',
+    locations: [common.hpc.hpc2020, common.hpc.jsc],
+    size: '0.5 TB',
+    inodes: '10',


this is tarred for the moment I assume, we should update it once we untar them

I think we should only report in the form ready for training, i.e. leave empty otherwise.

stac/metopa.jsonnet

clessig · 2025-07-07T11:22:45Z

stac/jsons/metopa.json

@@ -48,7 +48,7 @@
      ],
      "type": "Polygon"
   },
-   "id": "weathergen.atmo.metopa",
+   "id": "weathergen.atmo.METOP-A",


I think this is METOP-A, AMSU-A. The METOP-A platform (satellite) has quite a number of instruments: https://space.oscar.wmo.int/satellites/view/metop_b

Ok, it's from the MHS instrument

clessig · 2025-07-07T11:23:03Z

stac/jsons/metopb.json

@@ -48,7 +48,7 @@
      ],
      "type": "Polygon"
   },
-   "id": "weathergen.atmo.metopb",
+   "id": "weathergen.atmo.METOP-B",


Same as above, distinguish platform and instrument

Shall we just have the main name/title as the instrument, and then the datasets from the 2 different satellites are given as different assets of the same instrument?

Let's keep it separate. We can merge at the stream level (this is why a stream can have multiple sources).

…fengyun satellites

iluise · 2025-07-09T12:35:34Z

Hi, it looks good to me! I added ABI-GOES as well. @shmh40 if it looks good for you then can you prepare the final develop version so you get to try the whole pipeline? :)

shmh40 added 2 commits July 1, 2025 17:51

testing adding metopa and metopb as placeholder drafts to stac database

59e101e

added the actual json files because I think we have to

2492e0f

shmh40 requested a review from iluise July 1, 2025 17:58

shmh40 self-assigned this Jul 2, 2025

shmh40 added the datasets Anything related to the datasets used in the project label Jul 2, 2025

shmh40 added this to WeatherGen-dev Jul 2, 2025

Merge branch 'develop' into shmh40/dev/stac_test

82b001c

iluise reviewed Jul 3, 2025

View reviewed changes

shmh40 added 4 commits July 7, 2025 09:26

updated metopa metopb jsons and ets

1566193

add fy3 and update metops

0c81cc7

updated names of metops

248cc80

updated metopb untarred size inodes and end date

2a7f56c

clessig reviewed Jul 7, 2025

View reviewed changes

shmh40 added 11 commits July 7, 2025 11:41

update names to instrument, satellite

841fbdb

add untarred data size and inodes for metopa

3eee5bb

updated to oscar naming, with format platform, instrument, and added …

07600df

…fengyun satellites

update size and inodes of fy3c mwhs

bbc357b

add fengyun jsons, missing before, and update unique ids of metopa and b

e65d7e5

add processing_level field to metopa as a test

6935a67

adding processing level field

8c29a15

fix up processing level

a11c1e6

updated jsons and jsonnets for provenance

b9bdb69

actually include provenance

56522d3

Merge remote-tracking branch 'origin/develop' into shmh40/dev/stac_test

7471f54

shmh40 marked this pull request as ready for review July 8, 2025 10:09

shmh40 and others added 4 commits July 8, 2025 10:19

updated to include processor and provider, remove provenance

89a1b3c

add abi-goes

de93b1a

fix abi goes geometry

80c9112

fix latitude and longitude

1ae4456

fix typo

d9a2d08

iluise and others added 5 commits July 9, 2025 16:46

hopefully this time lat is right..

a8586c8

Merge remote-tracking branch 'origin/develop' into shmh40/dev/stac_test

7c80a42

update catalogue json for develop

681e1b5

check catalogue on this branch

1695a9f

jsonneted for develop

9b1c202

shmh40 requested a review from iluise July 11, 2025 09:18

iluise approved these changes Jul 11, 2025

View reviewed changes

iluise merged commit ee6e757 into develop Jul 11, 2025
3 checks passed

github-project-automation bot moved this to Done in WeatherGen-dev Jul 11, 2025

iluise deleted the shmh40/dev/stac_test branch July 11, 2025 09:31

Test adding metopa and metopb as placeholder drafts to stac database #426

Test adding metopa and metopb as placeholder drafts to stac database #426

Uh oh!

Conversation

shmh40 commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Issue Number

Code Compatibility

Code Performance and Testing

Dependencies

Documentation

Additional Notes

Uh oh!

shmh40 commented Jul 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iluise commented Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

shmh40 commented Jul 1, 2025 •

edited

Loading