Skip to content

Commit 292a23d

Browse files
m-mohrm-mohr
authored
Merge branch 'draft' into job-timestamps
2 parents 2c3815e + e4f607a commit 292a23d

File tree

7 files changed

+219
-5
lines changed

7 files changed

+219
-5
lines changed

‎CHANGELOG.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1010

1111
- **New extensions:**
1212
- [Remote Process Definition Extension](./extensions/remote-process-definition/README.md)
13+
- [Processing Parameters Extension](./extensions/processing-parameters/README.md)
1314
- `POST /result`: Added response header "OpenEO-Identifier" to expose an identifier associated with a synchronous processing request.
1415
- Added `version` property to `GET /processes` [#517](https://github.com/Open-EO/openeo-api/issues/517)
1516
- Added `queued`, `started` and `unpublished` to the batch job metadata and the corresponding STAC results [#542](https://github.com/Open-EO/openeo-api/issues/542)
@@ -44,9 +45,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
4445
- Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397)
4546
- Added metadata field `openeo:status` to indicate the job status (and whether the result is complete or not).
4647
- Added parameter `partial` to allow retrieving incomplete results, which must also add the new property `openeo:status` to the metadata. [#430](https://github.com/Open-EO/openeo-api/issues/430)
47-
- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs`: Added `level` parameter to requests to set the minimum log level returned by the response. [#485](https://github.com/Open-EO/openeo-api/issues/485)
48+
- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Added `level` parameter to requests to set the minimum log level returned by the response. [#485](https://github.com/Open-EO/openeo-api/issues/485)
4849
- Added property `log_level` to secondary web service, batch job and synchronous processing endpoints to indicate the minimum severity level that should be stored for logs. [#329](https://github.com/Open-EO/openeo-api/issues/329)
49-
- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Added `level` property in responses to reflect the minimum log level that may appear in the response. [#329](https://github.com/Open-EO/openeo-api/issues/329)
50+
- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs`: Added `level` property in responses to reflect the minimum log level that may appear in the response. [#329](https://github.com/Open-EO/openeo-api/issues/329)
5051
- Recommendation to add media types and titles to links for a better user experience.
5152
- Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405)
5253
- Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409)

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/O
3333
| ------------------------------------------------------------------ | ------- | ------------ | ----------- |
3434
| [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. |
3535
| [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. |
36+
| [Processing Parameters](extensions/processing-parameters/) | 0.1.0 | experimental | Explore and handle additional processing parameters that a back-end can offer for the processing modes (sync. processing, batch jobs, web services). |
3637
| [Remote Process Definition](extensions/remote-process-definition/) | 0.1.0 | experimental | Load user-defined processes that are hosted externally through the process namespace into process graphs. |
3738

3839
## Repository

‎extensions/processing-parameters/.spectral.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
extends: "spectral:oas"
2+
rules:
3+
contact-properties: true
4+
tag-description: true
5+
oas3-parameter-description: true
6+
oas3-unused-component: true
7+
operation-id-kebab-case:
8+
given: "$..operationId"
9+
then:
10+
function: pattern
11+
functionOptions:
12+
match: ^[a-z][a-z0-9\-]*$

‎extensions/processing-parameters/README.md

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Processing Parameters Extension
2+
3+
The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing parameters that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services).
4+
5+
- Version: **0.1.0**
6+
- Stability: **experimental**
7+
- [OpenAPI document](openapi.yaml)
8+
- Conformance class: `https://api.openeo.org/extensions/processing-parameters/0.1.0`
9+
10+
**Note:** This document only documents the additions to the specification.
11+
Extensions can not change or break existing behavior of the openEO API.
12+
13+
This extension adds a new endpoint (`GET /processing_parameters`, see [OpenAPI document](openapi.yaml))
14+
to discover the additional processing parameters that a back-end offers.
15+
16+
Additionally, this extension allows to provide specific default values for user-defined processes (UDPs),
17+
which includes:
18+
19+
- UDPs submitted directly for synchronous processing, as batch jobs, or as secondary web services
20+
- UDPs stored through the `/process_graphs` endpoints
21+
- UDPs stored external to the API and retrieved through the [Remote Process Definition Extension](../remote-process-definition/README.md)
22+
23+
The parameters and its values are provided separately for each processing mode.
24+
25+
## Embedding default processing options in UDPs
26+
27+
UDPs can provide default values for specific processing parameters.
28+
29+
The values for each parameter (so called 'options') are provided separately for each processing mode.
30+
The following properties are added to the top-level of a UDP (e.g. as sibling nodes to the "process_graph" property) for the respective processing modes:
31+
32+
- `default_synchronous_options` for synchronous processing
33+
- `default_job_options` for batch jobs
34+
- `default_service_options` for secondary web services
35+
36+
The schema for each of these properties is:
37+
38+
```yaml
39+
type: object
40+
additionalProperties:
41+
description: Any type
42+
```
43+
44+
The keys of the object are the respective parameter names.
45+
The values of the object are the default values for the parameters.
46+
Schematic restrictions are not defined for the object, but the schemas for the parameters as defined in `GET /processing_parameters` apply to the given values.
47+
These values provide the defaults unless a user overrides them in the actual data processing request (e.g. `POST /jobs`, see below).
48+
49+
Example UDP including defaults for the processing parameters `memory` and `block-sizes` of a batch job:
50+
51+
```json
52+
{
53+
"id": "my_evi",
54+
"parameters": [...],
55+
"process_graph": {...},
56+
"default_job_options": {
57+
"memory": "2GB",
58+
"block-sizes": [128, 32]
59+
}
60+
}
61+
```
62+
63+
## Resolving parameters
64+
65+
Due to the variety of places where processing parameters can be provided, the following
66+
list defines how the parameters must be resolved. The prioritization is as follows:
67+
68+
1. If present, use the parameter specified in the processing request directly (e.g. in `POST /jobs` as a top-level property)
69+
2. If present, use the default parameter specified in the UDP
70+
3. Otherwise, use the default value for the parameter as specified in `GET /processing_parameters`
71+
72+
"Present" means that the property is present in the JSON representation regardless of the value given, i.e.
73+
properties are present if an empty string, an empty array, an empty object, `false`, `0`, or `null` are provided.
74+
75+
Unrecognized/unknown parameters that are provided through UDPs must be ignored by backends.

‎extensions/processing-parameters/openapi.yaml

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
openapi: 3.0.2
2+
info:
3+
title: openEO API - Processing Parameters Extension
4+
version: 0.1.0
5+
description: >-
6+
The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing parameters that a back-end can offer for the three processing modes (synchronous processing, batch jobs, and secondary web services).
7+
The openEO API specification allows back-ends to accept "additional back-end specific properties" for each of the processing modes.
8+
As the openEO API specification does not define the name and schemas of these properties, this extension provides a standardized way to define and document these properties.
9+
contact:
10+
name: openEO Consortium
11+
url: 'https://openeo.org'
12+
email: [email protected]
13+
license:
14+
name: Apache 2.0
15+
url: 'http://www.apache.org/licenses/LICENSE-2.0.html'
16+
externalDocs:
17+
url: https://github.com/Open-EO/openeo-api/blob/draft/extensions/workspaces/README.md
18+
tags:
19+
- name: Capabilities
20+
description: General information about the API implementation and other supported capabilities provided by the back-end.
21+
- name: Data Processing
22+
description: Organizes and manages data processing on the back-end, either as synchronous on-demand computation or batch jobs.
23+
- name: Batch Jobs
24+
description: Management of batch processing tasks (jobs) and their results.
25+
- name: Secondary Services
26+
description: On-demand access to data using other web service protocols.
27+
servers:
28+
- url: 'https://openeo.example/api/{version}'
29+
description: >-
30+
The URL of the API MAY freely be chosen by the back-end providers. The
31+
path, including API versioning, is a *recommendation* only. Nevertheless,
32+
all servers MUST support HTTPS as the authentication methods are not
33+
secure with HTTP only!
34+
variables:
35+
version:
36+
default: v1
37+
description: >-
38+
API versioning is RECOMMENDED. As the openEO API is following
39+
[SemVer](https://semver.org/) only the MAJOR part of the stable
40+
version numbers (i.e. versions >= 1.0.0) SHOULD be used for API
41+
versioning in the URL. The reason is that backward-incompatible
42+
changes are usually introduced by major changes. Therefore, the
43+
version number in the URL MUST not be used by the clients to detect
44+
the version number of the API. Use the version number returned from
45+
`GET /` instead.
46+
paths:
47+
/processing_parameters:
48+
get:
49+
summary: Additional processing parameters
50+
operationId: list-processing-parameters
51+
description: |-
52+
Lists additional custom processing parameters
53+
that a back-end offers for the different processing modes (synchronous processing, batch jobs, secondary web services).
54+
The parameters specified here can be added to the corresponding `POST` requests at the top-level of the object that is sent as the payload.
55+
All parameters SHOULD explicitly be made optional with reasonable defaults as otherwise the interoperability between the implementations decreases.
56+
tags:
57+
- Capabilities
58+
- Data Processing
59+
- Batch Jobs
60+
- Secondary Services
61+
security:
62+
- {}
63+
- Bearer: []
64+
responses:
65+
'200':
66+
description: >-
67+
An object with a list of parameters per processing mode.
68+
content:
69+
application/json:
70+
schema:
71+
description: Processing parameters per processing mode.
72+
type: object
73+
properties:
74+
create_job_parameters:
75+
$ref: '#/components/schemas/processing_create_parameters'
76+
create_service_parameters:
77+
$ref: '#/components/schemas/processing_create_parameters'
78+
create_synchronous_parameters:
79+
$ref: '#/components/schemas/processing_create_parameters'
80+
components:
81+
schemas:
82+
processing_create_parameters:
83+
title: Creation Parameters
84+
description: |-
85+
List of additional custom parameters that a back-end offers during the creation
86+
of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively.
87+
type: array
88+
items:
89+
$ref: '../../openapi.yaml#/components/schemas/parameter'
90+
example:
91+
- name: memory
92+
description: Maximum amount of memory that will be allocated for processing, in gigabytes.
93+
optional: true
94+
default: 32
95+
schema:
96+
type: integer
97+
minimum: 1
98+
securitySchemes:
99+
Bearer:
100+
$ref: '../../openapi.yaml#/components/securitySchemes/Bearer'

‎extensions/processing-parameters/package.json

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"name": "@openeo/api-extension-processing-parameters",
3+
"version": "0.1.0",
4+
"author": "openEO Consortium",
5+
"license": "Apache-2.0",
6+
"description": "The openEO API specification.",
7+
"homepage": "https://openeo.org",
8+
"bugs": {
9+
"url": "https://github.com/Open-EO/openeo-api/issues"
10+
},
11+
"repository": {
12+
"type": "git",
13+
"url": "git+https://github.com/Open-EO/openeo-api.git"
14+
},
15+
"devDependencies": {
16+
"@stoplight/spectral-cli": "^6.6.0",
17+
"redoc-cli": "^0.13.21"
18+
},
19+
"scripts": {
20+
"start": "redoc-cli serve openapi.yaml --watch --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true",
21+
"build": "redoc-cli bundle openapi.yaml -o redoc.html --title \"openEO API - Processing Parameters Extension\" --cdn --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true",
22+
"test": "spectral lint openapi.yaml"
23+
}
24+
}

‎extensions/remote-process-definition/README.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,10 @@ This extension enables user to load user-defined processes that are hosted exter
1313
The openEO API defines the `namespace` property in a process node of a process graph as follows:
1414

1515
> The following options are predefined by the openEO API, but additional namespaces may be introduced by back-ends or in a future version of the API.
16-
> * `null` [...]
17-
> * `backend` [...]
18-
> * `user` [...]
16+
>
17+
> - `null` [...]
18+
> - `backend` [...]
19+
> - `user` [...]
1920
2021
This makes it possible for this extension to add additional allowed values to the `namespace` property.
2122

0 commit comments

Comments
 (0)