Cluster deployment issues

[heerener]

We regularly see failures when deploying CloudFormation stack. Some investigation has already happened. An experiment with deploying 10 clusters at the same time shows around 60% failure rate.

The main symptom shows up as a "DRA did not stabilize" error:

Resource handler returned message: 
"Resource of type 'AWS::FSx::DataRepositoryAssociation' with identifier 'dra-[...]' did not stabilize." 
(RequestToken: [...], HandlerErrorCode: NotStabilized)

A closer look at the DRA failure state reveals the true error to be

Amazon FSx does not have the required permissions for the S3 bucket. 
Verify that FSx has correct permissions by checking the bucket policy.`

All buckets that get mounted now have this in their permission set:

{
    "Sid": "AllowFSX",
    "Effect": "Allow",
    "Principal": {
         "Service": "fsx.amazonaws.com"
    },
    "Action": "s3:*",
    "Resource": [
         "arn:aws:s3:::[...]/**",
         "arn:aws:s3:::[...]"
    ]
}

Second issue is that when deploying multiple clusters at the same time, the DRA doesn't stabilize in 4 hours. It is unclear whether that's expected and if so what the timeout should be.

To be discussed in the AWS call this Thursday.

[evakennyobi]

This should have a priority & time estimate (days) please @heerener

[heerener]

Several suggestions came out of the call with AWS, I've filed follow-up tickets:

• Get rid of ** in S3 policies #33 : to test whether it helps
• See if there's a link between subnets and cluster failures #34 : to investigate
• Waiting for DRAs might not be necessary after all #35 : deployment optimization, won't help with the failure but will speed up deployment
• Set timeouts to more realistic values #36 : cleanup, to be picked up in-between
• Don't create separate FSx filesystems for the read-only buckets #37 : cost optimization if it works

Ramin has also contacted the FSx team for them to have a look.

[heerener]

While the subtasks are all done, they were simply the practical things we could try after our call with AWS. The actual issue still persists.

[heerener]

Currently waiting for support case 174738741700196

[heerener]

Following up with support; the issue is known on their end. In order to work around it for our own development needs we can try to create the cluster with one DRA and then manually update it for each extra DRA we need, effectively serializing DRA creation.

This might help us work around the issue for now, while we wait for a real fix and the platform doesn't need to deploy clusters, but is not a viable long-term solution.

As an extra potential workaround, we've also asked for the involvement of the aws-parallelcluster team to look at making it possible to deploy multiple DRAs sequentially instead of in parallel.