EMR container self sufficient example DAG #242

bharanidharan14 · 2022-04-18T07:25:48Z

Add Example DAG to be self sufficient

Add Docker commands to install eksctl, kubectl
Add bash script to create clusters, roles, user and delete the cluster
Trigger the bash script via operator in the DAG

Closes #199

- Add steps to DAG, so that DAG can be self sufficient - Rename DAG file - Add steps to install AWS eks, kubectl packages in Docker file

- Add clean up script - Remove the python script to delete the EKS cluster and EMR virtual cluster

.circleci/integration-tests/Dockerfile

astronomer/providers/amazon/aws/example_dags/example_create_emr_on_eks_cluster.sh

astronomer/providers/amazon/aws/example_dags/example_emr_eks_containers_job.py

astronomer/providers/amazon/aws/example_dags/example_create_emr_on_eks_cluster.sh

.circleci/integration-tests/Dockerfile

astronomer/providers/amazon/aws/example_dags/example_emr_eks_containers_job.py

codecov · 2022-04-18T11:34:49Z

Codecov Report

Merging #242 (ee8fbda) into main (5cf53b2) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #242      +/-   ##
==========================================
- Coverage   96.34%   96.33%   -0.01%     
==========================================
  Files          45       45              
  Lines        2541     2540       -1     
==========================================
- Hits         2448     2447       -1     
  Misses         93       93

Impacted Files	Coverage Δ
astronomer/providers/amazon/aws/triggers/emr.py	`100.00% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5cf53b2...ee8fbda. Read the comment docs.

astronomer/providers/amazon/aws/example_dags/example_create_emr_on_eks_cluster.sh

astronomer/providers/amazon/aws/example_dags/example_emr_eks_containers_job.py

pankajkoti

Nice one 👍🏽 Some minor suggestions!

.circleci/integration-tests/Dockerfile

astronomer/providers/amazon/aws/example_dags/example_create_emr_on_eks_cluster.sh

astronomer/providers/amazon/aws/example_dags/example_emr_eks_containers_job.py

bharanidharan14 added 5 commits April 11, 2022 23:42

Add step for the DAG to be self sufficient DAG

4204e99

- Add steps to DAG, so that DAG can be self sufficient - Rename DAG file - Add steps to install AWS eks, kubectl packages in Docker file

Merge branch 'main' into emr_container_example_dag

b51dd03

Add Docker file change to install kubectl

49dbc13

Add clean up script for the EMR on EKS cluster

f16f3b7

- Add clean up script - Remove the python script to delete the EKS cluster and EMR virtual cluster

Merge branch 'main' into emr_container_example_dag

8c33be5

bharanidharan14 changed the title ~~Emr container example dag~~ EMR container self sufficient example DAG Apr 18, 2022