Improve Elasticsearch mappings #181

philvarner · 2022-02-22T19:09:34Z

Related Issue(s):

Proposed Changes:

replace several older field names with current ones (e.g., eo:gsd is now gsd)
map numeric values to float (instead of long by default)
disable indexing of projjson field
re-enabled eslint check for quoted JSON values, and fix existing uses of that.

PR Checklist:

I have added my changes to the CHANGELOG or a CHANGELOG entry is not required.

Co-authored-by: Marc <[email protected]> Co-authored-by: Marc Huffnagle <[email protected]>

* PR template: put related issues in list for rich formatting * fix path of api.yaml in webpack, add npm build to GH Action

) * fix service-desc link relation type to use OpenAPI yaml media type

* define routes for transaction endpoints

Co-authored-by: Phil Varner <[email protected]>

) * Maintain created time when reingesting an item * Remove typechecking of tests * Update CHANGELOG.md Co-authored-by: Phil Varner <[email protected]> * Consolidate tests * Fix test * Replace items rather than parti-update * Remove ts-check in test * Update docs Co-authored-by: Phil Varner <[email protected]>

marchuffnagle · 2022-02-28T19:11:42Z

It looks like the main branch needs to be pulled into this branch. It's unclear what needs to be reviewed.

philvarner · 2022-02-28T19:16:14Z

Yeah, I though I had, and I'm not sure what even happened there. It's up to date now.

philvarner · 2022-02-28T19:16:48Z

docs/api.md

@@ -1703,7 +1703,7 @@ null
    ],
    "exclude": [
      [
-        "eo:sun_azimuth"
+        "view:sun_azimuth"


this field used to be in the EO extension, is now in the View extension.

philvarner · 2022-02-28T19:17:19Z

fixtures/collections.js

@@ -1,14 +1,15 @@
-const common = require('./common')
+const dynamicTemplates = require('./dynamicTemplates')


this was the only thing in common, so I made it more explicit.

philvarner · 2022-02-28T19:19:19Z

fixtures/dynamicTemplates.js

+    numerics: {
+      match_mapping_type: 'long',
+      mapping: { type: 'float' }
+    }


we now default to mapping any field that's not otherwise explicitly mapped to float instead of long, to avoid a situation where the first time the field and has a non-decimal value that it gets mapped to long, and then all other values for that field, even if they're decimal, will be truncated to longs in the index

philvarner · 2022-02-28T19:20:30Z

fixtures/dynamicTemplates.js

+    }
+  },
+  {
+    proj_projjson: {


this is a complex object with several different variations. attempting to index a variation that is different than the first one indexed can result in an error, see #108

philvarner · 2022-02-28T19:21:04Z

src/lambdas/api/api.yaml

@@ -562,7 +562,7 @@ components:
            max: '2019-07-10T13:44:56Z'
          'sci:citation':
            - 'Copernicus Sentinel data [Year]'
-          'eo:gsd':
+          'gsd':


gsd used to be in the EO extension, now it's part of the STAC Commons fields that have no prefix.

philvarner · 2022-02-28T19:43:57Z

okay, now it's ready for review @marchuffnagle

fixtures/dynamicTemplates.js

src/lib/api.js

tests/system/test-ingest.js

marchuffnagle

Looks good, just a couple questions.

philvarner · 2022-03-01T17:49:08Z

PR comments resolved.

Phil Varner and others added 16 commits February 22, 2022 14:09

initial mappings

ff2feff

support open-ended datetime intervals, fix camelcase violations (#180)

71b63bc

Co-authored-by: Marc <[email protected]> Co-authored-by: Marc Huffnagle <[email protected]>

Use Express for API routing (#182)

e900f1f

Return the correct API content types (#184)

f0555bb

Return empty results when searching for a nonexistent collection (#185)

75c4047

Fix npm build problem (#188)

de6995b

* PR template: put related issues in list for rich formatting * fix path of api.yaml in webpack, add npm build to GH Action

fix service-desc link relation type to use OpenAPI yaml media type (#189

48c36fc

) * fix service-desc link relation type to use OpenAPI yaml media type

Split system tests into separate files (#191)

b0b49ea

Clean up a couple skipped tests (#192)

ae4f6a5

Compress API responses (#193)

c18ece6

define routes for transaction endpoints (#194)

5c48647

* define routes for transaction endpoints

Maintain created time when reingesting an item (#196)

a229079

Co-authored-by: Phil Varner <[email protected]>

Tests handle their own API server

91d2c09

Move addEndpoint middleware (#198)

fa3f16b

provide a more accurate mapping for ES fields

29c7817

philvarner requested review from marchuffnagle and matthewhanson February 28, 2022 19:10

Merge branch 'main' into pv/es-mapping

bfc8067

philvarner marked this pull request as ready for review February 28, 2022 19:16

philvarner commented Feb 28, 2022

View reviewed changes

Phil Varner added 3 commits February 28, 2022 14:30

add messages, fix integral/decimal test

0f24793

use two different projjsons

5e62f9b

changelog

6a56ddd

marchuffnagle reviewed Feb 28, 2022

View reviewed changes

fixtures/dynamicTemplates.js Outdated Show resolved Hide resolved

marchuffnagle reviewed Feb 28, 2022

View reviewed changes

src/lib/api.js Show resolved Hide resolved

marchuffnagle reviewed Feb 28, 2022

View reviewed changes

tests/system/test-ingest.js Outdated Show resolved Hide resolved

marchuffnagle suggested changes Feb 28, 2022

View reviewed changes

philvarner changed the title ~~initial mappings~~ Update Elasticsearch mappings Mar 1, 2022

philvarner changed the title ~~Update Elasticsearch mappings~~ Improve Elasticsearch mappings Mar 1, 2022

philvarner requested a review from marchuffnagle March 1, 2022 17:49

Phil Varner added 2 commits March 1, 2022 12:51

minor changes

b8b8f6b

linting

7d5a83c

marchuffnagle approved these changes Mar 1, 2022

View reviewed changes

philvarner merged commit 5f54fbc into main Mar 2, 2022

philvarner deleted the pv/es-mapping branch March 2, 2022 00:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve Elasticsearch mappings #181

Improve Elasticsearch mappings #181

Uh oh!

philvarner commented Feb 22, 2022 •

edited

Loading

Uh oh!

marchuffnagle commented Feb 28, 2022

Uh oh!

philvarner commented Feb 28, 2022

Uh oh!

philvarner Feb 28, 2022

Uh oh!

philvarner Feb 28, 2022

Uh oh!

philvarner Feb 28, 2022

Uh oh!

philvarner Feb 28, 2022

Uh oh!

philvarner Feb 28, 2022

Uh oh!

philvarner commented Feb 28, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marchuffnagle left a comment

Uh oh!

philvarner commented Mar 1, 2022

Uh oh!

Uh oh!

		@@ -1,14 +1,15 @@
		const common = require('./common')
		const dynamicTemplates = require('./dynamicTemplates')

Improve Elasticsearch mappings #181

Improve Elasticsearch mappings #181

Uh oh!

Conversation

philvarner commented Feb 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marchuffnagle commented Feb 28, 2022

Uh oh!

philvarner commented Feb 28, 2022

Uh oh!

philvarner Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

philvarner Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

philvarner Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

philvarner Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

philvarner Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

philvarner commented Feb 28, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marchuffnagle left a comment

Choose a reason for hiding this comment

Uh oh!

philvarner commented Mar 1, 2022

Uh oh!

Uh oh!

philvarner commented Feb 22, 2022 •

edited

Loading