Add documentation

Author: mhuguesaws

4.validation_and_observability/4.prometheus-grafana/1click-dashboards-deployment/README.md

@@ -0,0 +1,48 @@
+# Machine Learning Infrastructure Monitoring for Slurm based cluster <!-- omit from toc -->
+
+This solution provides a "1 click" deployment observability stack to monitor your slurm based machine learning infrastructure. It automatically creates:
+- Amazon Managed Grafana
+- Amazon Managed Prometheus
+- Setup a Prometheus Agent Collector
+- Create data source and dashboard in Grafana
+
+
+## Prerequisites
+
+Install AWS Serverless Application Model Command Line Interface (AWS SAM CLI) version **>=1.135.0** by following the [instructions](<https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html>)
+
+
+## Architecture
+TBD
+
+## Deploy
+You will begin by installing the necessary Python package needed for the lambda function.
+In your shell:
+
+```bash
+cd dashboards
+pip install -r requirements.txt -t .
+cd ..
+```
+
+You are now ready to deploy the serverless application, run the following in your shell:
+
+```
+OBS_DASHBOARD_NAME="ml-obs-dashboard"
+sam build
+sam deploy --stack-name ${OBS_DASHBOARD_NAME} \
+    --guided \
+    --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
+    --parameter-overrides \
+    ParameterKey=PCClusterName,ParameterValue=<CLUSTER_NAME> \
+    ParameterKey=SubnetId,ParameterValue=<SUBNET_ID> \
+    ParameterKey=VpcId,ParameterValue=<VPC_ID>
+```
+
+
+## Clean up
+To delete the SAM application deployment, you can use the terminal and enter:
+
+```bash
+sam delete
+```

4.validation_and_observability/4.prometheus-grafana/1click-dashboards-deployment/cluster-observability.yaml

@@ -0,0 +1,71 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Description: "Setup to monitor sagemaker hyperpod clusters on AWS. Amazon Managed Prometheus and Amazon Manged Grafana workspaces with associated IAM roles are deployed in the AWS Account. Prometheus and exporter services are set up on Cluster Nodes. Author: Matt Nightingale - nghtm@"
+
+    
+Resources:
+  AmazonGrafanaWorkspaceIAMRole:
+    Type: 'AWS::IAM::Role'
+    Properties:
+      AssumeRolePolicyDocument:
+        Version: 2012-10-17
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service:
+                - grafana.amazonaws.com
+            Action:
+              - 'sts:AssumeRole'
+      RoleName: !Sub ${AWS::StackName}-Grafana-Role
+  
+  AmazonGrafanaPrometheusPolicy:
+    Type: AWS::IAM::Policy
+    Properties:
+      PolicyName: AmazonGrafana_Prometheus_policy
+      PolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+        - Effect: Allow
+          Action:
+          - aps:ListWorkspaces
+          - aps:DescribeWorkspace
+          - aps:QueryMetrics
+          - aps:GetLabels
+          - aps:GetSeries
+          - aps:GetMetricMetadata
+          Resource: "*"
+      Roles: [!Ref AmazonGrafanaWorkspaceIAMRole]   
+
+  AmazonGrafanaWorkspace:
+    Type: 'AWS::Grafana::Workspace'
+    Properties:
+      AccountAccessType: CURRENT_ACCOUNT
+      Name: !Sub ${AWS::StackName}-Dashboard
+      Description: Amazon Grafana Workspace to monitor SageMaker Cluster
+      AuthenticationProviders:
+        - AWS_SSO
+      PermissionType: SERVICE_MANAGED
+      RoleArn: !GetAtt 
+        - AmazonGrafanaWorkspaceIAMRole
+        - Arn
+      DataSources: ["CLOUDWATCH","PROMETHEUS"]
+      OrganizationRoleName: "ADMIN"
+
+  APSWorkspace:
+    Type: AWS::APS::Workspace
+    Properties:
+      Alias: !Sub ${AWS::StackName}-Hyperpod-WorkSpace
+      Tags:
+      - Key: Name
+        Value: SageMaker Hyperpod PrometheusMetrics
+
+Outputs:
+  Region:
+    Value: !Ref "AWS::Region"
+  AMPRemoteWriteURL:
+    Value: !Join ["" , [ !GetAtt APSWorkspace.PrometheusEndpoint , "api/v1/remote_write" ]]
+  AMPEndPointUrl:
+    Value: !GetAtt APSWorkspace.PrometheusEndpoint
+  GrafanWorkspaceURL:
+    Value: !Join ["" , [ "https://", !GetAtt AmazonGrafanaWorkspace.Endpoint ]]
+  GrafanWorkspaceId:
+    Value: !GetAtt AmazonGrafanaWorkspace.Id

4.validation_and_observability/4.prometheus-grafana/dashboards/create_ml_dashboards.py 4.validation_and_observability/4.prometheus-grafana/1click-dashboards-deployment/dashboards/create_ml_dashboards.py

@@ -119,40 +119,3 @@ def lambda_handler(event, context):
                                                workspaceId=workspace_id)
 
     return {'statusCode': 200, 'body': json.dumps('Dashboards created')}
-
-
-def main():
-    aws_region = 'ap-southeast-2'
-    grafana_key_name = "CreateDashpa01"
-    grafana_url = 'https://g-182a00efff.grafana-workspace.ap-southeast-2.amazonaws.com'
-    workspace_id = 'g-182a00efff'
-    prometheus_url = 'https://aps-workspaces.ap-southeast-2.amazonaws.com/workspaces/ws-e4384558-d586-46ec-bd12-173d7019119e/'
-
-    client = boto3.client('grafana')
-    response = client.create_workspace_api_key(keyName=grafana_key_name,
-                                               keyRole='ADMIN',
-                                               secondsToLive=60,
-                                               workspaceId=workspace_id)
-
-    try:
-        grafana = GrafanaApi.from_url(
-            url=grafana_url,
-            credential=TokenAuth(token=response['key']),
-        )
-
-        prometheus_datasource = create_prometheus_datasource(
-            grafana, prometheus_url, aws_region)
-
-        for i in PROM_DASHBOARDS_URL:
-            dashboard_payload = mk_dash(prometheus_datasource['uid'], i)
-            response = grafana.dashboard.update_dashboard(dashboard_payload)
-
-    except Exception as e:
-        print(e)
-
-    response = client.delete_workspace_api_key(keyName=grafana_key_name,
-                                               workspaceId=workspace_id)
-
-
-if __name__ == '__main__':
-    main()