Skip to content

[Manual Backport 2.x] update vpc flow with flint-s3 based DDL assets and dashboard #1721

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 18, 2024
Merged

[Manual Backport 2.x] update vpc flow with flint-s3 based DDL assets and dashboard #1721

merged 2 commits into from
Apr 18, 2024

Conversation

RyanL1997
Copy link
Collaborator

Description

[Manual Backport 2.x] update vpc flow with flint-s3 based DDL assets and dashboard

Issues Resolved

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@YANG-DB @Swiddis @RyanL1997
update vpc flow with flint-s3 based DDL assets and dashboard (opensea…
39f757e 
…rch-project#1691)

* update vpc flow with flint-s3 based DDL assets and dashboard

Signed-off-by: YANGDB <[email protected]>

* update MV to use auto sync refresh

Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_30min_connections_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_total-bytes_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_cardinality_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_live_week_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_60min_connections_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* remove comments from MV queries

Signed-off-by: YANGDB <[email protected]>

* update to remove HIVE as not supported yet in current EMR version

Signed-off-by: YANGDB <[email protected]>

* update refresh sync rate

Signed-off-by: YANGDB <[email protected]>

* update watermark_delay

Signed-off-by: YANGDB <[email protected]>

* add live only workflow & dashboard
add pre-agg + live workflow & dashboard
add refresh-workflow for the pre-agg queries

Signed-off-by: YANGDB <[email protected]>

* add live only workflow & dashboard
add pre-agg + live workflow & dashboard
add refresh-workflow for the pre-agg queries

Signed-off-by: YANGDB <[email protected]>

* add live all only workflow & dashboard

Signed-off-by: YANGDB <[email protected]>

* correct table name hard coded issue

Signed-off-by: YANGDB <[email protected]>

* add vpc table creation based on CSV format

Signed-off-by: YANGDB <[email protected]>

---------

Signed-off-by: YANGDB <[email protected]>
Co-authored-by: Simeon Widdis <[email protected]>
@RyanL1997 RyanL1997 added enhancement New feature or request integrations Used to denote items related to the Integrations project labels Apr 18, 2024
RyanL1997
RyanL1997 commented Apr 18, 2024
View reviewed changes
server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_live_all_mv-1.0.0.sql
{table_name}
WITH (
auto_refresh = true,
refresh_interval = '15 Minute',
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will forward port this change into main since #1717 doesn't include this.

@YANG-DB YANG-DB merged commit e97df5f into opensearch-project:2.x Apr 18, 2024
9 of 22 checks passed
@RyanL1997 RyanL1997 mentioned this pull request Apr 18, 2024
Merged
6 tasks
amsiglan pushed a commit to amsiglan/dashboards-observability that referenced this pull request Jun 7, 2024
@RyanL1997 @YANG-DB
@Swiddis @sejli
[Manual Backport 2.x] update vpc flow with flint-s3 based DDL assets …
4fd1273 
…and dashboard (opensearch-project#1721)

* update vpc flow with flint-s3 based DDL assets and dashboard (opensearch-project#1691)

* update vpc flow with flint-s3 based DDL assets and dashboard

Signed-off-by: YANGDB <[email protected]>

* update MV to use auto sync refresh

Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_30min_connections_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_total-bytes_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_cardinality_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_live_week_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_60min_connections_mv-1.0.0.sql

Co-authored-by: Simeon Widdis <[email protected]>
Signed-off-by: YANGDB <[email protected]>

* remove comments from MV queries

Signed-off-by: YANGDB <[email protected]>

* update to remove HIVE as not supported yet in current EMR version

Signed-off-by: YANGDB <[email protected]>

* update refresh sync rate

Signed-off-by: YANGDB <[email protected]>

* update watermark_delay

Signed-off-by: YANGDB <[email protected]>

* add live only workflow & dashboard
add pre-agg + live workflow & dashboard
add refresh-workflow for the pre-agg queries

Signed-off-by: YANGDB <[email protected]>

* add live only workflow & dashboard
add pre-agg + live workflow & dashboard
add refresh-workflow for the pre-agg queries

Signed-off-by: YANGDB <[email protected]>

* add live all only workflow & dashboard

Signed-off-by: YANGDB <[email protected]>

* correct table name hard coded issue

Signed-off-by: YANGDB <[email protected]>

* add vpc table creation based on CSV format

Signed-off-by: YANGDB <[email protected]>

---------

Signed-off-by: YANGDB <[email protected]>
Co-authored-by: Simeon Widdis <[email protected]>

* Change the vpc flow's refresh interval to 15mins and watermark delay to 1 min

Signed-off-by: Ryan Liang <[email protected]>

---------

Signed-off-by: YANGDB <[email protected]>
Signed-off-by: Ryan Liang <[email protected]>
Co-authored-by: YANGDB <[email protected]>
Co-authored-by: Simeon Widdis <[email protected]>
(cherry picked from commit e97df5f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ps48 ps48 ps48 approved these changes

@YANG-DB YANG-DB YANG-DB approved these changes

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@rupal-bq rupal-bq Awaiting requested review from rupal-bq

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

Assignees
No one assigned
Labels
enhancement New feature or request integrations Used to denote items related to the Integrations project
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@RyanL1997 @ps48 @YANG-DB