Skip to content

Commit 97638f5

Browse files
TschounTschoun
authored
v1.2.2 (#102)
## Release Notes v1.2.2 ### Bug Fixes - Resolved issue #100 in the VHI calculation process. This fix has also been applied to the production environment. ### Enhancements - Fixed NDSI (Normalized Difference Snow Index) calculation: - NDSI is now added to all Sentinel-2 assets within the specified timeframe. - Masking is applied based on the NDSI of each specific date, enhancing the accuracy of snow coverage detection. ### Documentation - Updated README with latest information. - Added VHI Data Flow documentation to provide a clearer understanding of the process. ### New Features - Introduced a batch script (`batch_VHI.bat`) to facilitate month-wise VHI processing. ### Other Changes - Made preparations for merging the updated code. ### Contributors - @Tschoun - @davidoesch
1 parent 0304b21 commit 97638f5

File tree

7 files changed

+402
-109
lines changed

7 files changed

+402
-109
lines changed

‎README.md

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
[![Run processor](https://github.com/swisstopo/topo-satromo/actions/workflows/run-processor.yml/badge.svg)](https://github.com/swisstopo/topo-satromo/actions/workflows/run-processor.yml) [![Run publisher](https://github.com/swisstopo/topo-satromo/actions/workflows/run-publisher.yml/badge.svg)](https://github.com/swisstopo/topo-satromo/actions/workflows/run-publisher.yml)
22
[![GitHub commit](https://img.shields.io/github/last-commit/swisstopo/topo-satromo)](https://github.com/swisstopo/topo-satromo/commits/main)
3+
4+
![Python Versions](https://img.shields.io/badge/python-3.9%20%7C%203.10-blue.svg)
35
# SATROMO Processing chain
46

57
The "Erdbeobachtungs-SAtellitendaten fürs TRockenheitsMOnitoring" (SATROMO) consists of python code with ETL functionalities to operationally generate and provide AnalysisReadyData and indices from satellite sensors using GoogleEarthEngine, GithubAction and AWS S3 / Cloudfront
@@ -8,8 +10,11 @@ The "Erdbeobachtungs-SAtellitendaten fürs TRockenheitsMOnitoring" (SATROMO) con
810

911
**Note: This project is currently in the commissioning phase and is limited for operational use.**
1012

11-
- Data description: [Products](https://www.swisstopo.admin.ch/en/satellite-images-swisseo)
12-
- Access to data: [STAC BROWSER](https://data.geo.admin.ch/browser/index.html#/collections/ch.swisstopo.swisseo_s2-sr_v100?.language=en)
13+
14+
| | swissEO S2-SR | swissEO VHI |
15+
|------------------|---------------|--------------|
16+
| Data description | [Product site](https://www.swisstopo.admin.ch/en/satelliteimage-swisseo-s2-sr) | [Product site](https://www.swisstopo.admin.ch/en/satelliteimage-swisseo-vhi) |
17+
| Access to data | [STAC](https://data.geo.admin.ch/browser/index.html#/collections/ch.swisstopo.swisseo_s2-sr_v100) | [STAC](https://data.geo.admin.ch/browser/index.html#/collections/ch.swisstopo.swisseo_vhi_v100) |
1318

1419

1520
## Introduction and Project Description
@@ -85,6 +90,11 @@ Adapt your GoogleEarthEngine and rclone credentials
8590
8691
![Diagram](satromo_processor.drawio.svg)
8792
93+
94+
### Program flows VHI (`step1_processor_vhi.py` )
95+
See [VHI flowchart](VHI_flowchart.md)
96+
97+
8898
## Functionality in detail
8999
90100
For starters, we build a GithubAction & GEE python based ARD and Indices extractor for Switzerland. The SATROMO operational module is started on a pre-defined schedule using [GitHub Actions](https://github.com/features/actions).The "processor" run is using by default the [dev_config.py](configuration/dev_config.py]). A specific configuration can be started with `python satromo_processor.py my_config.py`. During a "processor" run, the code:
@@ -225,7 +235,7 @@ Update the step0_empty_assets.csv directly on GitHub PROD: https://github.com/sw
225235
226236
## Roadmap
227237
228-
- [ ] Add Changelog
238+
- [X] Add Changelog
229239
- [X] Implement STAC Cataloge description
230240
- [X] Implement Co-Registration correction
231241
- [X] Implement Topografic shadow information

‎VHI_flowchart.md

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
```mermaid
2+
graph TD
3+
A[Start VHI Processing] --> AA[Set Parameters]
4+
AA --> AB[Initialize Masks]
5+
AB --> AC[Set Processing Switches]
6+
AC --> AD[Define AOI]
7+
AD --> AE[Set Time Parameters]
8+
AE --> AF[Load S2 SR Data]
9+
AF --> B[Load NDVI Data]
10+
B --> C[Calculate VCI]
11+
AF --> D[Load LST Data]
12+
D --> E[Calculate TCI]
13+
C --> F[Combine VCI and TCI]
14+
E --> F
15+
F --> G[Calculate VHI]
16+
G --> H[Apply Vegetation/Forest Mask]
17+
H --> I[Export VHI]
18+
I --> J[End VHI Processing]
19+
20+
subgraph NDVI Processing
21+
B --> B1[Load Reference NDVI]
22+
B1 --> B1a[Load NDVI Stats for DOY]
23+
B1a --> B1b[Adjust for Offset and Scale]
24+
B --> B2[Load Current NDVI]
25+
B2 --> B3[Apply Cloud/Snow/Shadow Mask]
26+
B3 --> B3a[Calculate NDSI]
27+
B3a --> B3b[Apply NDSI Mask]
28+
B3b --> B3c[Apply Terrain Shadow Mask]
29+
B3c --> B3d[Apply Cloud/Cloud Shadow Mask]
30+
B3d --> B4[Create NDVI Mosaic]
31+
B4 --> B4a[Sort by Time]
32+
B4a --> B4b[Create Latest Pixel Mosaic]
33+
B4b --> B4c[Calculate NDVI]
34+
B1b --> C
35+
B4c --> C
36+
end
37+
38+
subgraph LST Processing
39+
D --> D1[Load Reference LST]
40+
D1 --> D1a[Load LST Stats for DOY]
41+
D1a --> D1b[Adjust for Scale]
42+
D --> D2[Load Current LST]
43+
D2 --> D3[Create LST Mosaic]
44+
D3 --> D3a[Filter LST Collection]
45+
D3a --> D3b[Sort by Time]
46+
D3b --> D3c[Create Latest Pixel Mosaic]
47+
D3c --> D3d[Select LST Band]
48+
D3d --> D3e[Adjust for Scale]
49+
D1b --> E
50+
D3e --> E
51+
end
52+
53+
subgraph Data Checks
54+
K[Check LST Coverage]
55+
K --> K1{LST Data Available?}
56+
K1 -->|No| K2[Process LST from Raw Data]
57+
K2 --> K2a[Generate MSG LST Mosaic]
58+
K1 -->|Yes| L
59+
L[Check Existing VHI Asset]
60+
L --> L1{VHI Asset Exists?}
61+
L1 -->|Yes| J
62+
L1 -->|No| M
63+
M[Check Empty Asset List]
64+
M --> M1{In Empty Asset List?}
65+
M1 -->|Yes| J
66+
M1 -->|No| N
67+
N[Check S2 SR Data Availability]
68+
N --> N1{S2 SR Data Available?}
69+
N1 -->|No| N2[Mark as Empty Asset]
70+
N1 -->|Yes| AA
71+
end
72+
73+
subgraph Parameter Setting
74+
AA --> AA1[Set AOI]
75+
AA --> AA2[Set Date Range]
76+
AA --> AA3[Set Alpha Value]
77+
AA --> AA4[Set CI Method]
78+
end
79+
80+
subgraph Masking
81+
AB --> AB1[Load Vegetation Mask]
82+
AB --> AB2[Load Forest Mask]
83+
end
84+
85+
subgraph Switches
86+
AC --> AC1[Set Export Options]
87+
AC --> AC2[Set Percentile Usage]
88+
end
89+
90+
subgraph VCI Calculation
91+
C --> C1{Using Percentiles?}
92+
C1 -->|Yes| C2[Use 5th and 95th Percentiles]
93+
C1 -->|No| C3[Use Min and Max]
94+
C2 --> C4[Calculate VCI]
95+
C3 --> C4
96+
end
97+
98+
subgraph TCI Calculation
99+
E --> E1{Using Percentiles?}
100+
E1 -->|Yes| E2[Use 5th and 95th Percentiles]
101+
E1 -->|No| E3[Use Min and Max]
102+
E2 --> E4[Calculate TCI]
103+
E3 --> E4
104+
end
105+
106+
subgraph VHI Calculation
107+
G --> G1[Combine VCI and TCI]
108+
G1 --> G2[Apply Alpha Weight]
109+
end
110+
111+
subgraph Export
112+
I --> I1[Export to GEE Asset]
113+
I --> I2[Export to Google Drive]
114+
end

‎batch_VHI.bat

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
@echo off
2+
setlocal EnableDelayedExpansion
3+
4+
set "config_file=%~1"
5+
set "year=%~2"
6+
set "month=%~3"
7+
8+
echo *********************************************
9+
echo ACHTUNG: Sicherstellen dass nur VHI Produkt im config aktiviert ist!
10+
echo *********************************************
11+
echo Bitte Enter druecken, um fortzufahren...
12+
pause >nul
13+
14+
if "!config_file!"=="" (
15+
echo Bitte Konfigurationsfile, Jahr und Monat als Parameter eingeben.
16+
echo Beispiel: batch_vhi.bat oed_prod_config.py 2024 04
17+
goto :eof
18+
)
19+
20+
if "!year!"=="" (
21+
echo Bitte Jahr als Parameter eingeben.
22+
goto :eof
23+
)
24+
25+
if "!month!"=="" (
26+
echo Bitte Monat als Parameter eingeben.
27+
goto :eof
28+
)
29+
30+
set "days_in_month=31"
31+
if !month!==04 set "days_in_month=30"
32+
if !month!==06 set "days_in_month=30"
33+
if !month!==09 set "days_in_month=30"
34+
if !month!==11 set "days_in_month=30"
35+
if !month!==02 (
36+
set "days_in_month=28"
37+
set /a "is_leap_year=(!year! %% 4)==0 && (!year! %% 100)!=0 || (!year! %% 400)==0"
38+
if !is_leap_year!==1 set "days_in_month=29"
39+
)
40+
41+
for /L %%D in (1,1,!days_in_month!) do (
42+
set "day=%%D"
43+
if %%D LSS 10 set "day=0%%D"
44+
python satromo_processor.py !config_file! !year!-!month!-!day!
45+
)
46+
47+
:publish_loop
48+
ping -n 3600 localhost >nul
49+
python satromo_publish.py !config_file!
50+
51+
for /f %%A in ('find /c /v "" ^< processing/running_task.csv') do set "lines=%%A"
52+
if !lines! GTR 1 goto publish_loop
53+
54+
echo *********************************************
55+
echo VHI Prozess fuer !year! !month! abgeschlossen.
56+
echo *********************************************
57+
echo *********************************************
58+
pause
59+
60+
endlocal

‎configuration/oed_prod_config.py

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# -*- coding: utf-8 -*-
2+
import os
3+
4+
# General variables
5+
# --------------------------
6+
7+
# GitHub repository
8+
GITHUB_OWNER = "swisstopo"
9+
GITHUB_REPO = "topo-satromo"
10+
11+
# Secrets
12+
GDRIVE_SECRETS = os.path.join("secrets", "geetest-credentials-int.secret")
13+
RCLONE_SECRETS = os.path.join("secrets", "rclone.conf")
14+
FSDI_SECRETS = os.path.join("secrets", "stac_fsdi-prod.json")
15+
16+
# File and directory paths
17+
GEE_RUNNING_TASKS = os.path.join("processing", "running_tasks.csv")
18+
GEE_COMPLETED_TASKS = os.path.join("tools", "completed_tasks.csv")
19+
EMPTY_ASSET_LIST = os.path.join("tools", "step0_empty_assets.csv")
20+
PROCESSING_DIR = "processing"
21+
LAST_PRODUCT_UPDATES = os.path.join("tools", "last_updates.csv")
22+
# Set GDRIVE Type: GCS for Google Cloud Storage and DRIVE for Google Drive
23+
GDRIVE_TYPE = "GCS"
24+
# Set GCS Bucket name of Google Cloud Storage
25+
GCLOUD_BUCKET = "satromo_export"
26+
# Local Machine
27+
GDRIVE_SOURCE_DEV = "geedriveINT:"
28+
# under Windows, add \\ to escape the backslash like r'Y:\\'
29+
# GDRIVE_MOUNT_DEV = r'G:\\'
30+
GDRIVE_MOUNT_DEV = r'M:\\satromo_export' # for GCS
31+
# under Windows, add \\ to escape the backslash like r'X:\\'
32+
S3_DESTINATION_DEV = r'X:\\'
33+
34+
# GITHUB
35+
GDRIVE_SOURCE_INT = "geedrivePROD:"
36+
GDRIVE_MOUNT_INT = "localgdrive"
37+
S3_DESTINATION_INT = os.path.join("s3INT:satromoint", "data")
38+
39+
40+
# General GEE parameters
41+
42+
# TODO: check if needed
43+
SHARD_SIZE = 256
44+
45+
# Development environment parameters
46+
RESULTS = os.path.join("results") # Local path for results
47+
48+
# General product parameters
49+
# ---------------------------
50+
51+
# Coordinate Reference System (EPSG:4326 for WGS84, EPSG:2056 for CH1903+, see epsg.io)
52+
OUTPUT_CRS = "EPSG:2056"
53+
54+
# Desired buffer in m width around ROI, e.g., 25000, this defines the final extent
55+
# TODO: check if needed in context with step0
56+
BUFFER = os.path.join("assets", "ch_buffer_5000m.shp")
57+
OVERVIEW_LAKES = os.path.join("assets", "overview_lakes_2056.shp")
58+
OVERVIEW_RIVERS = os.path.join("assets", "overview_rivers_2056.shp")
59+
WARNREGIONS = os.path.join("assets", "warnregionen_vhi_2056.shp")
60+
61+
# Switzerland border with 10km buffer: [5.78, 45.70, 10.69, 47.89] , Schönbühl [ 7.471940, 47.011335, 7.497431, 47.027602] Martigny [ 7.075402, 46.107098, 7.100894, 46.123639]
62+
# Defines the initial extent to search for image tiles This is not the final extent is defined by BUFFER
63+
# TODO: check if needed in context with step0
64+
ROI_RECTANGLE = [5.78, 45.70, 10.69, 47.89]
65+
ROI_BORDER_BUFFER = 5000 # Buffer around Switzerland
66+
67+
# No data value
68+
NODATA = 9999
69+
70+
71+
## PRODUCTS, INDICES and custom COLLECTIONS ###
72+
# ---------------------------
73+
# See https://github.com/swisstopo/topo-satromo/tree/main?tab=readme-ov-file#configuration-in-_configpy for details
74+
# TL;DR : First define in A) PRODUCTS, INDICES: for step0 (cloud, shadow, co-register, mosaic) the TOA SR data custom "step0_collection" to be generated / used
75+
# then
76+
77+
# A) PRODUCTS, INDICES
78+
# ********************
79+
80+
# ch.swisstopo.swisseo_s2-sr
81+
PRODUCT_S2_LEVEL_2A = {
82+
# "prefix": "S2_L2A_SR",
83+
# TODO: check if needed in context with step0
84+
"image_collection": "COPERNICUS/S2_SR_HARMONIZED",
85+
"geocat_id": "7ae5cd5b-e872-4719-92c0-dc2f86c4d471",
86+
"temporal_coverage": 31, # Days
87+
"spatial_scale_export": 10, # Meters # TODO: check if needed in context with step0
88+
"asset_size": 5,
89+
"spatial_scale_export_mask": 10,
90+
"product_name": "ch.swisstopo.swisseo_s2-sr_v100",
91+
"no_data": 9999,
92+
"step0_collection": "projects/satromo-prod/assets/col/S2_SR_HARMONIZED_SWISS"
93+
}
94+
95+
# VHI – Trockenstress ch.swisstopo.swisseo_vhi_v100
96+
PRODUCT_VHI = {
97+
# TODO: check if needed in context with step0
98+
"image_collection": "COPERNICUS/S2_SR_HARMONIZED",
99+
"geocat_id": "bc4d0e6b-e92e-4f28-a7d2-f41bf61e98bc",
100+
"temporal_coverage": 7, # Days
101+
"spatial_scale_export": 10, # Meters
102+
"product_name": "ch.swisstopo.swisseo_vhi_v100",
103+
"no_data": 255,
104+
"missing_data": 110,
105+
"asset_size": 2,
106+
'NDVI_reference_data': 'projects/satromo-prod/assets/col/1991-2020_NDVI_SWISS',
107+
'LST_reference_data': 'projects/satromo-prod/assets/col/1991-2020_LST_SWISS',
108+
'LST_current_data': 'projects/satromo-prod/assets/col/LST_SWISS',
109+
"step1_collection": 'projects/satromo-prod/assets/col/VHI_SWISS',
110+
# "step0_collection": "projects/satromo-prod/assets/col/S2_SR_HARMONIZED_SWISS"
111+
}
112+
113+
# MSG – MeteoSchweiz: only used for repreocessing
114+
PRODUCT_MSG_CLIMA = {
115+
#
116+
# this is placeholder, needed for the step0 function,
117+
"image_collection": "METEOSCHWEIZ/MSG",
118+
"temporal_coverage": 1, # Days
119+
"product_name": "ch.meteoschweiz.landoberflaechentemperatur",
120+
"no_data": 0,
121+
# 'step0_collection': 'projects/satromo-prod/assets/col/LST_SWISS'
122+
}
123+
124+
125+
# B custom COLLECTION
126+
# ********************
127+
# Contains dictionary used to manage custom collection (asset) in GEE,
128+
# for example to clear old images not used anymore.
129+
130+
# Configure the dict containing
131+
# - the name of the custom collection (asset) in GEE, (eg: projects/satromo-int/assets/COL_S2_SR_HARMONIZED_SWISS )
132+
# - the function to process the raw data for teh collection (eg:step0_processor_s2_sr.generate_s2_sr_mosaic_for_single_date )
133+
134+
# Make sure that the products above use the corresponding custom collection (assets)
135+
136+
step0 = {
137+
# 'projects/satromo-exolabs/assets/col_s2_toa': {
138+
# 'step0_function': 'step0_processor_s2_toa.generate_s2_toa_mosaic_for_single_date',
139+
# # cleaning_older_than: 2 # entry used to clean assets
140+
# },
141+
# 'projects/satromo-int/assets/LST_SWISS': {
142+
# 'step0_function': 'step0_processor_msg_lst.generate_msg_lst_mosaic_for_single_date'
143+
# # cleaning_older_than: 2 # entry used to clean assets
144+
# },
145+
'projects/satromo-prod/assets/col/S2_SR_HARMONIZED_SWISS': {
146+
'step0_function': 'step0_processor_s2_sr.generate_s2_sr_mosaic_for_single_date'
147+
# cleaning_older_than: 2 # entry used to clean assets
148+
}
149+
}
150+
151+
152+
# STAC Integration
153+
# ---------------
154+
155+
STAC_FOLDER = "stac-collection"
156+
# Use the AWS Cloudfront distribution instead of "https://satromoint.s3.eu-central-2.amazonaws.com/"
157+
STAC_BASE_URL = "https://d29cp2gnktw6by.cloudfront.net/"
158+
STAC_PRODUCT = ["S2_LEVEL_2A", "NDVI-MAX"]
159+
160+
# under Windows, add \\ to escape the backslash like r'X:\\'
161+
STAC_DESTINATION_DEV = r'X:\\'
162+
163+
GDRIVE_SOURCE_INT = "geedrivePROD:"
164+
GDRIVE_MOUNT_INT = "localgdrive"
165+
STAC_DESTINATION_INT = "s3INT:satromoint"
166+
167+
# STAC FSDI Production
168+
# ---------------
169+
170+
STAC_FSDI_SCHEME = 'https'
171+
STAC_FSDI_HOSTNAME = 'data.geo.admin.ch'
172+
STAC_FSDI_API = '/api/stac/v0.9/'

0 commit comments

Comments
 (0)