Skip to content

Commit 5cc60e9

Browse files
huyabooruchidhopensearch-changeset-bot[bot]
authored
Feature/data importer (#9602)
* Add data importer plugin to OSD core Signed-off-by: Huy Nguyen <[email protected]> * Update route names Signed-off-by: Huy Nguyen <[email protected]> * Add support for file preview (#9354) Signed-off-by: Huy Nguyen <[email protected]> * Feature/data importer (#9408) * move header to top and define sectiobna nd add preview table * add preview load Signed-off-by: Ruchi Sharma <[email protected]> * fix ui and add preview table Signed-off-by: Ruchi Sharma <[email protected]> * add data source, indices and mismtach in table Signed-off-by: Ruchi Sharma <[email protected]> * revert changes from yml Signed-off-by: Ruchi Sharma <[email protected]> * comment data import flag Signed-off-by: Ruchi Sharma <[email protected]> * fix line Signed-off-by: Ruchi Sharma <[email protected]> * revert validate check and give rows count as constant Signed-off-by: Ruchi Sharma <[email protected]> --------- Signed-off-by: Ruchi Sharma <[email protected]> * Remove system indexes from index name selector (#9462) * Refactor changes Signed-off-by: Huy Nguyen <[email protected]> * Changeset file for PR #9462 created/updated --------- Signed-off-by: Huy Nguyen <[email protected]> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> * Don't cat indices when there is no datasource available (#9468) Signed-off-by: Huy Nguyen <[email protected]> * Add more tests + general refactoring (#9487) Signed-off-by: Huy Nguyen <[email protected]> * Use depth.limit instead of nested_objects.limit as the validation for deeply nested objects (#9512) Signed-off-by: Huy Nguyen <[email protected]> * Add better error handling and messages (#9533) Signed-off-by: Huy Nguyen <[email protected]> * Fix table not rendering data (#9545) Signed-off-by: Huy Nguyen <[email protected]> * Update yarn lock (#9685) Signed-off-by: Huy Nguyen <[email protected]> * Fix + re-enable determineMapping tests (#9686) Signed-off-by: Huy Nguyen <[email protected]> * Renamed IFileParsers to IFileProcessors + address PR comments Signed-off-by: Huy Nguyen <[email protected]> * Fix dummy URLs Signed-off-by: Huy Nguyen <[email protected]> * Fix valid JSON dummy url Signed-off-by: Huy Nguyen <[email protected]> --------- Signed-off-by: Huy Nguyen <[email protected]> Signed-off-by: Ruchi Sharma <[email protected]> Co-authored-by: Ruchi Sharma <[email protected]> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
1 parent 70d7f4e commit 5cc60e9

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+5971
-11
lines changed

‎changelogs/fragments/9602.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
feat:
2+
- Add a new Data Importer Plugin to OSD Core ([#9602](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/9602))

‎config/opensearch_dashboards.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -370,6 +370,9 @@
370370
# Set the value to true to enable dynamic config service to obtain configs from a config store. By default, it's disabled
371371
# dynamic_config_service.enabled: false
372372

373+
# Set the value to true to enable direct data import from a file
374+
# data_importer.enabled: false
375+
373376
# Set the backend roles in groups or users, whoever has the backend roles or exactly match the user ids defined in this config will be regard as dashboard admin.
374377
# Dashboard admin will have the access to all the workspaces(workspace.enabled: true) and objects inside OpenSearch Dashboards.
375378
# The default config is [], and no one will be dashboard admin.

‎docs/_sidebar.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
- [Dashboard](../src/plugins/dashboard/README.md)
4141
- [Data](../src/plugins/data/README.md)
4242
- [Data_explorer](../src/plugins/data_explorer/README.md)
43+
- [Data_importer](../src/plugins/data_importer/README.md)
4344
- [Data_source](../src/plugins/data_source/README.md)
4445
- [Data_source_management](../src/plugins/data_source_management/README.md)
4546
- [Dev_tools](../src/plugins/dev_tools/README.md)

‎package.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,7 @@
209209
"@osd/ui-framework": "1.0.0",
210210
"@osd/ui-shared-deps": "1.0.0",
211211
"@reduxjs/toolkit": "^1.6.1",
212+
"@types/ndjson": "^2.0.4",
212213
"@types/yauzl": "^2.9.1",
213214
"JSONStream": "1.3.5",
214215
"abortcontroller-polyfill": "^1.4.0",
@@ -229,6 +230,7 @@
229230
"elasticsearch": "^16.7.0",
230231
"execa": "^4.0.2",
231232
"expiry-js": "0.1.7",
233+
"fast-csv": "^5.0.2",
232234
"fast-deep-equal": "^3.1.1",
233235
"file-saver": "^2.0.5",
234236
"font-awesome": "4.7.0",
@@ -257,6 +259,7 @@
257259
"moment": "^2.24.0",
258260
"moment-timezone": "^0.5.27",
259261
"mustache": "^2.3.2",
262+
"ndjson": "^2.0.0",
260263
"node-fetch": "^2.6.7",
261264
"node-forge": "^1.3.0",
262265
"p-map": "^4.0.0",

‎src/.i18nrc.json

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
"dashboard": "plugins/dashboard",
1111
"data": "plugins/data",
1212
"dataExplorer": "plugins/data_explorer",
13+
"dataImporter": "plugins/data_importer",
1314
"dataSourcesManagement": "plugins/data_source_management",
1415
"devTools": "plugins/dev_tools",
1516
"discover": "plugins/discover",
@@ -46,14 +47,9 @@
4647
"server": "legacy/server",
4748
"share": "plugins/share",
4849
"statusPage": "legacy/core_plugins/status_page",
49-
"telemetry": [
50-
"plugins/telemetry",
51-
"plugins/telemetry_management_section"
52-
],
50+
"telemetry": ["plugins/telemetry", "plugins/telemetry_management_section"],
5351
"tileMap": "plugins/tile_map",
54-
"timeline": [
55-
"plugins/vis_type_timeline"
56-
],
52+
"timeline": ["plugins/vis_type_timeline"],
5753
"uiActions": "plugins/ui_actions",
5854
"usageCollection": "plugins/usage_collection",
5955
"visAugmenter": "plugins/vis_augmenter",

‎src/plugins/data_importer/README.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Data Importer Plugin
2+
3+
An OpenSearch Dashboards plugin for importing your static data to OpenSearch indexes directly from Dashboards via text box or file upload. Multiple Data Source (MDS) compatible.
4+
5+
https://github.com/user-attachments/assets/2eb590cb-a21a-4c4e-8b3f-5517365ffde5
6+
7+
Supported filetypes:
8+
9+
- JSON (treated as 1 document)
10+
- CSV
11+
- NDJSON
12+
- And more (TBD)
13+
14+
---
15+
16+
## Configurations
17+
18+
This plugin can be configured in your `config/opensearch_dashboards.yml`
19+
20+
```yaml
21+
# Enable the plugin
22+
data_importer.enabled: true
23+
24+
# Configure which file types will be supported (by default, all 3 are enabled)
25+
data_importer.enabledFileTypes: ['csv', 'json', 'ndjson']
26+
27+
# Configure file size upload limit in bytes
28+
data_importer.maxFileSizeBytes: 100000000
29+
30+
# Configure character limit for text data
31+
data_importer.maxTextCount: 10000
32+
33+
# Configure the max document count the data importer will parse when previewing data
34+
data_importer.filePreviewDocumentsCount: 10
35+
```

‎src/plugins/data_importer/common/constants.ts

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
/*
2+
* Copyright OpenSearch Contributors
3+
* SPDX-License-Identifier: Apache-2.0
4+
*/
5+
6+
export const PLUGIN_NAME_AS_TITLE = 'Data Importer';
7+
export const PLUGIN_ID = 'dataImporter';
8+
export const PLUGIN_NAME = 'dataImporter';
9+
10+
export const CSV_FILE_TYPE = 'csv';
11+
export const JSON_FILE_TYPE = 'json';
12+
export const NDJSON_FILE_TYPE = 'ndjson';
13+
14+
export const DEFAULT_SUPPORTED_FILE_TYPES_LIST = [CSV_FILE_TYPE, JSON_FILE_TYPE, NDJSON_FILE_TYPE];
15+
16+
export const CSV_SUPPORTED_DELIMITERS = [',', ';', '\t', '|'];
17+
18+
export enum DYNAMIC_MAPPING_TYPES {
19+
NULL = 'null',
20+
BOOLEAN = 'boolean',
21+
FLOAT = 'float',
22+
DOUBLE = 'double',
23+
INTEGER = 'integer',
24+
OBJECT = 'object',
25+
ARRAY = 'array',
26+
TEXT = 'text',
27+
KEYWORD = 'keyword',
28+
DATE = 'date',
29+
}

‎src/plugins/data_importer/common/index.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
/*
2+
* Copyright OpenSearch Contributors
3+
* SPDX-License-Identifier: Apache-2.0
4+
*/
5+
6+
export * from './constants';

‎src/plugins/data_importer/config.ts

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
/*
2+
* Copyright OpenSearch Contributors
3+
* SPDX-License-Identifier: Apache-2.0
4+
*/
5+
6+
import { schema, TypeOf } from '@osd/config-schema';
7+
import { CSV_FILE_TYPE, JSON_FILE_TYPE, NDJSON_FILE_TYPE } from './common/constants';
8+
9+
export const configSchema = schema.object({
10+
enabled: schema.boolean({ defaultValue: false }),
11+
enabledFileTypes: schema.arrayOf(schema.string(), {
12+
defaultValue: [CSV_FILE_TYPE, JSON_FILE_TYPE, NDJSON_FILE_TYPE],
13+
}),
14+
maxFileSizeBytes: schema.number({
15+
defaultValue: 100000000,
16+
min: 1,
17+
}),
18+
maxTextCount: schema.number({
19+
defaultValue: 10000,
20+
min: 1,
21+
}),
22+
filePreviewDocumentsCount: schema.number({
23+
defaultValue: 10,
24+
min: 1,
25+
}),
26+
});
27+
28+
export type ConfigSchema = TypeOf<typeof configSchema>;
29+
export type PublicConfigSchema = Omit<ConfigSchema, 'enabled'>;

‎src/plugins/data_importer/opensearch_dashboards.json

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"id": "dataImporter",
3+
"version": "8.0.0",
4+
"opensearchDashboardsVersion": "opensearchDashboards",
5+
"server": true,
6+
"ui": true,
7+
"requiredPlugins": ["navigation"],
8+
"optionalPlugins": ["dataSource", "dataSourceManagement"]
9+
}

0 commit comments

Comments
 (0)