AWS not enough capacity error message

[hashibot]

This issue was originally opened by @alexmatsuk as hashicorp/terraform#15906. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform v0.10.2

AWS didn't have enough capacity today in us-west-2a on R4.xlarge instances.
Error message I got when running 'terraform apply'

aws_instance.node.4: Still creating... (10s elapsed)
aws_instance.node.0: Still creating... (10s elapsed)
aws_instance.node.2: Still creating... (10s elapsed)
aws_instance.node.2: Still creating... (20s elapsed)
aws_instance.node.0: Still creating... (20s elapsed)
aws_instance.node.4: Still creating... (20s elapsed)
aws_instance.node.2: Still creating... (30s elapsed)
aws_instance.node.4: Still creating... (30s elapsed)
aws_instance.node.0: Still creating... (30s elapsed)
aws_instance.node.4: Still creating... (40s elapsed)
aws_instance.node.2: Still creating... (40s elapsed)
aws_instance.node.0: Still creating... (40s elapsed)
aws_instance.node.0: Still creating... (50s elapsed)
aws_instance.node.2: Still creating... (50s elapsed)
aws_instance.node.4: Still creating... (50s elapsed)
aws_instance.node.0: Still creating... (1m0s elapsed)
aws_instance.node.2: Still creating... (1m0s elapsed)
aws_instance.node.4: Still creating... (1m0s elapsed)
Error applying plan:

3 error(s) occurred:

• aws_instance.node[2]: 1 error(s) occurred:

• aws_instance.node.2: Error launching source instance: timeout while waiting for state to become 'success' (timeout: 30s)

• aws_instance.node[4]: 1 error(s) occurred:

• aws_instance.node.4: Error launching source instance: timeout while waiting for state to become 'success' (timeout: 30s)

• aws_instance.node[0]: 1 error(s) occurred:

• aws_instance.node.0: Error launching source instance: timeout while waiting for state to become 'success' (timeout: 30s)

Is it possible for terraform to return error message from AWS?
How do others usually handle this error message without login to AWS console?

[akuznetsov-gridgain]

+1 We also facing this issue. Could this be fixed?

[akuznetsov-gridgain]

Hi,

It possible to return non zero exit code or throw appropriate exception?

[bflad]

Could someone provide a Gist of a relevant Terraform debug log when this occurs? I have a hunch that what might be happening is:

• Call RunInstances (wrapped in Terraform logic for handling IAM eventual consistency)
• SDK receives InsufficientCapacity error (likely 5XX) and automatically retries itself
• SDK keeps retrying for >30 seconds before returning (so Terraform sees it), which the 30 second Terraform timeout for this call triggers the timeout error logic timeout while waiting for state to become 'success'
• We return that semi-erroneous error

[bflad]

Sorry, meant to include the relevant code block: https://github.com/terraform-providers/terraform-provider-aws/blob/17b7168bd4e6745ff24e32395f059b84abc270a1/aws/resource_aws_instance.go#L511-L535

[drehelis]

@bflad, Here's a gist when this occurs.
https://gist.github.com/autogun/3a254960db0eb6363f6d609cddb529b0

Trying to create m4.4xlarge instance in AZ with not enough capacity.

I wish there was a terraform way to fallback to a lower type.

[aglarendil]

Hit this issue also - what is bad about it is that the error message that tf throws completely hides the real issue making it completely cryptic.

[CaseyLabs]

+1 as well.... would love to get the actually AWS error message back from Terraform. Right now, I'm having to manually launch an instance to verify that it's a problem with EC2 capacity, and not an issue with my Terraform template.

[bflad]

Hi again, folks! 👋 Thanks for providing the additional information, we can likely get the error handling added in, but it will not really be possible to test it, so it may take a few iterations. The important part here is:

<Response><Errors><Error><Code>InsufficientInstanceCapacity</Code><Message>We currently do not have sufficient m4.4xlarge capacity in the Availability Zone you requested (ap-northeast-1d). Our system will be working on provisioning additional capacity. You can currently get m4.4xlarge capacity by not specifying an Availability Zone in your request or choosing ap-northeast-1c, ap-northeast-1a.</Message></Error></Errors><RequestID>--OMITTED--</RequestID></Response>

While the handling gets sorted out within Terraform, I think its probably worth mentioning two newer EC2 features that can help in terms of capacity planning/management: EC2 Fleets and EC2 Capacity Reservations. Fleets can be configured to automatically choose different instance types based on available capacity. Capacity Reservations can be used to pre-allocate capacity to ensure you don't receive insufficient capacity errors.

In AWS' words:

https://aws.amazon.com/about-aws/whats-new/2018/04/introducing-amazon-ec2-fleet/

Amazon EC2 Fleet is a new feature that simplifies the provisioning of Amazon EC2 capacity across different Amazon EC2 instance types, Availability Zones and across On-Demand, Amazon EC2 Reserved Instances (RI) and Amazon EC2 Spot purchase models. With a single API call, now you can provision capacity across EC2 instance types and across purchase models to achieve desired scale, performance and cost.

https://aws.amazon.com/about-aws/whats-new/2018/10/Amazon-EC2-now-offers-On-Demand-Capacity-Reservations/

You can now reserve Amazon EC2 capacity for any duration with On-Demand Capacity Reservations. Amazon EC2 provides a wide selection of instance types as well as over 50 global Availability Zones where you can run your applications. Some customers however have very specific requirements for portions of their applications around instance type, size, and capability, as well as the Availability Zone the instance needs to run in, and need very high levels of assurance that they will always be able to launch that specific capacity. Previously in order to achieve a capacity reservation, customers had to leverage Zonal Reserved Instances and commit to that capacity for a minimum of one year. Now with On-Demand Capacity Reservations, customers can reserve the exact capacity they need, in the location they need, and can keep it only for as long as they need it.

Managing both of these is supported as of version 1.42.0 of the AWS provider via the aws_ec2_fleet and aws_ec2_capacity_reservation resources. 👍

[CaseyLabs]

Thanks for the follow-up, Brian! Good to know about the new aws_ec2_fleet and aws_ec2_capacity_reservation. That said, I encountered the capacity error with a new AWS test account, just trying to launch just 2 T3.medium instances... ec2_fleet might be a little overkill in my case ;-)

[ivanpedersen]

Just adding an additional example of the error message for trying to launch t3.small instances.

2018-11-19T08:10:51.230Z [DEBUG] plugin.terraform-provider-aws_v1.45.0_x4: 2018/11/19 08:10:51 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2018-11-19T08:10:51.230Z [DEBUG] plugin.terraform-provider-aws_v1.45.0_x4: <Response><Errors><Error><Code>InsufficientInstanceCapacity</Code><Message>We currently do not have sufficient t3.small capacity in the Availability Zone you requested (eu-west-1c). Our system will be working on provisioning additional capacity. You can currently get t3.small capacity by not specifying an Availability Zone in your request or choosing eu-west-1b.</Message></Error></Errors><RequestID>7c126b9a-0f1c-497c-9d81-3480090045cb</RequestID></Response>

[jayleverett]

For anyone struggling with this, one thing to check is what kind of tenancy you're using.

We were running into this frequently, but after changing instance_tenancy="dedicated" to instance_tenancy="default" in resource "aws_vpc", the issue disappeared.

[jayleverett]

(Looks like there's also a tenancy argument in aws_instance FYI)