Error: error creating MongoDB Cluster: unexpected EOF

[adam-pasabi]

Terraform CLI and Terraform MongoDB Atlas Provider Version

Terraform v1.0.5
+ provider registry.terraform.io/mongodb/mongodbatlas v0.9.1

(also occurs under mongodbatlas v1.2.0)

Terraform Configuration File

resource "mongodbatlas_project" "project" {
  name   = data.google_project.project.name
  org_id = [REDACTED]
}

resource "mongodbatlas_network_peering" "test" {
  project_id     = mongodbatlas_project.project.id
  container_id   = mongodbatlas_cluster.cluster.container_id
  provider_name  = "GCP"
  gcp_project_id = data.google_project.project.id
  network_name   = "default"
}

resource "mongodbatlas_cluster" "cluster" {
  project_id   = mongodbatlas_project.project.id
  name         = data.google_project.project.name
  cluster_type = "REPLICASET"
  replication_specs {
    num_shards = 1
    regions_config {
      region_name     = "EUROPE_WEST_2"
      electable_nodes = 3
      priority        = 7
      read_only_nodes = 0
    }
  }
  provider_backup_enabled                         = true
  auto_scaling_disk_gb_enabled                    = true
  mongo_db_major_version                          = "5.0"
  auto_scaling_compute_enabled                    = true
  provider_auto_scaling_compute_min_instance_size = "M10"
  provider_auto_scaling_compute_max_instance_size = "M30"
  auto_scaling_compute_scale_down_enabled         = true

  lifecycle {
    ignore_changes = [provider_instance_size_name]
  }

  //Provider Settings "block"
  provider_name               = "GCP"
  disk_size_gb                = 10
  provider_instance_size_name = "M10"
}

resource "time_sleep" "wait_for_mongo_private_connection_string" {
  depends_on      = [mongodbatlas_network_peering.test, google_compute_network_peering.peering]
  create_duration = "300s"
}

data "mongodbatlas_cluster" "cluster" {
  depends_on = [time_sleep.wait_for_mongo_private_connection_string]
  project_id = mongodbatlas_cluster.cluster.project_id
  name       = mongodbatlas_cluster.cluster.name
}

resource "mongodbatlas_database_user" "mongo_user" {
  username           = random_string.mongo_user_username.result
  password           = random_password.mongo_user_password.result
  project_id         = mongodbatlas_cluster.cluster.project_id
  auth_database_name = "admin"

  roles {
    role_name     = "readWrite"
    database_name = [REDACTED]
  }
}

data "mongodbatlas_cluster" "database" {
  project_id = mongodbatlas_cluster.cluster.project_id
  name       = mongodbatlas_cluster.cluster.name
}

resource "mongodbatlas_project_ip_access_list" "access" {
  project_id = mongodbatlas_cluster.cluster.project_id
  cidr_block = [REDACTED]
  comment    = [REDACTED]
}

data "mongodbatlas_project_ip_access_list" "access" {
  project_id = mongodbatlas_project_ip_access_list.access.project_id
  cidr_block = mongodbatlas_project_ip_access_list.access.cidr_block
}

Steps to Reproduce

1. terraform init
2. terraform apply

Expected Behavior

It should create the cluster without erroring

Actual Behavior

terraform apply failed with an error (see below)

Debug Output

terraform.log

Running with TF_LOG=debug

2022-02-10T11:29:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: -----------------------------------------------------
2022-02-10T11:29:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: 2022/02/10 11:29:35 [DEBUG] status for MongoDB cluster: dashboard-counts: CREATING
2022-02-10T11:29:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: 2022/02/10 11:29:35 [TRACE] Waiting 1m0s before next try
2022-02-10T11:30:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: 2022/02/10 11:30:35 [DEBUG] MongoDB Atlas API Request Details:
2022-02-10T11:30:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: ---[ REQUEST ]---------------------------------------
2022-02-10T11:30:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: GET /api/atlas/v1.0/groups/[REDACTED]/clusters/[REDACTED] HTTP/1.1
2022-02-10T11:30:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: Host: cloud.mongodb.com
2022-02-10T11:30:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: User-Agent: terraform-provider-mongodbatlas/devel
2022-02-10T11:30:35.994Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: Accept: application/json
2022-02-10T11:30:35.995Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: Accept-Encoding: gzip
2022-02-10T11:30:35.995Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: 
2022-02-10T11:30:35.995Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: 
2022-02-10T11:30:35.995Z [DEBUG] provider.terraform-provider-mongodbatlas_v0.9.1: -----------------------------------------------------
2022-02-10T11:30:36.499Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-02-10T11:30:36.499Z [DEBUG] provider.terraform-provider-google_v4.10.0_x5: 2022/02/10 11:30:36 [DEBUG] [transport] transport: http2Server.HandleStreams failed to read frame: read unix /var/folders/7f/w84b0r3d3db30p472087v9zc0000gn/T/plugin229942681->: use of closed network connection 
2022-02-10T11:30:36.499Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-02-10T11:30:36.499Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-02-10T11:30:36.500Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-02-10T11:30:36.502Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/null/3.1.0/darwin_amd64/terraform-provider-null_v3.1.0_x5 pid=74467
2022-02-10T11:30:36.502Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/time/0.7.2/darwin_amd64/terraform-provider-time_v0.7.2_x5 pid=74473
2022-02-10T11:30:36.502Z [DEBUG] provider: plugin exited
2022-02-10T11:30:36.502Z [DEBUG] provider: plugin exited
2022-02-10T11:30:36.503Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/mongodb/mongodbatlas/0.9.1/darwin_amd64/terraform-provider-mongodbatlas_v0.9.1 pid=74469
2022-02-10T11:30:36.503Z [DEBUG] provider: plugin exited
2022-02-10T11:30:36.545Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.10.0/darwin_amd64/terraform-provider-google_v4.10.0_x5 pid=74474
2022-02-10T11:30:36.545Z [DEBUG] provider: plugin exited

Console output

mongodbatlas_cluster.cluster: Still creating... [9m10s elapsed]
mongodbatlas_cluster.cluster: Still creating... [9m20s elapsed]
mongodbatlas_cluster.cluster: Still creating... [9m30s elapsed]
mongodbatlas_cluster.cluster: Still creating... [9m40s elapsed]
mongodbatlas_cluster.cluster: Still creating... [9m50s elapsed]
╷
│ Error: error creating MongoDB Cluster: Get "https://cloud.mongodb.com/api/atlas/v1.0/groups/[REDACTED]/clusters/[REDACTED]": unexpected EOF
│ 
│   with mongodbatlas_cluster.cluster,
│   on main.tf line 1054, in resource "mongodbatlas_cluster" "cluster":
│ 1054: resource "mongodbatlas_cluster" "cluster" {
│ 
╵

Additional Context

This is running on GCP

References

Initially mentioned on: #110

Fwiw the workaround is to use https://registry.terraform.io/providers/mongodb/mongodbatlas/latest/docs/resources/cluster#import and then run terraform apply again.

[nikhil-mongo]

@adam-pasabi We are reviewing it and will get back to you on this soon.

[nikhil-mongo]

@adam-pasabi

Unable to repro, this is working in my case.

Terraform v1.0.5
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.11.0
+ provider registry.terraform.io/hashicorp/time v0.7.2
+ provider registry.terraform.io/mongodb/mongodbatlas v1.2.0

Error

2022-02-10T11:30:36.499Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-02-10T11:30:36.499Z [DEBUG] provider.terraform-provider-google_v4.10.0_x5: 2022/02/10 11:30:36 [DEBUG] [transport] transport: http2Server.HandleStreams failed to read frame: read unix /var/folders/7f/w84b0r3d3db30p472087v9zc0000gn/T/plugin229942681->: use of closed network connection 

Often this error message indicative of a crash in the provider (which runs in a separate process).
This could potentially be at the google-terraform-provider side as indicative in the logs and they could look to improve, and data source updates could be done quicker than managed resources.

• There are lot of data resources for google-terraform-provider in your configuration and I believe it could be one of possibility for this issue.

Could you please reach out to GCP Support for the terraform issue and see if they have been observing this with their provider versions. However, with the mongodbatlas-provider there is no such problem I could see.

@themantissa FYI.

Thanks.

[adam-pasabi]

hey @nikhil-mongo, thanks for looking into it. I'll raise a ticket with google-terraform-provider to see what they suggest (I'm on holiday after today so it won't be immediately).
One other thought - as this bug is very consistent for me - is there any way I can add debugging to a local copy of this provider and run that locally? I am familiar with Go, but not with developing for Terraform. Perhaps if I'm able to isolate the issue I might be able to write a fix for it. Any pointers in this regard would be much appreciated.

[julienvincent]

I am also running into this very consistently (haven't got a single apply to work). I am using this via the Pulumi provider, but underlying issue seems to be the same

[themantissa]

Internal ticket INTMDB-304

[adam-pasabi]

I am also running into this very consistently (haven't got a single apply to work). I am using this via the Pulumi provider, but underlying issue seems to be the same

hey @julienvincent - how long does it take before it fails? For me, it's around 10 mins. I wonder if the 10 min wait is somehow part of the problem.

I haven't had any feedback from @nikhil-mongo regarding trying to debug this locally, but it really is a pain for me and my team, so I hope he does reply at some point 😎

[nikhil-mongo]

@adam-pasabi To debug this locally, go through the contribution guide for the terraform provider here.

The problem as @julienvincent mentioned is that you are using pulumi as the provider and we do not officially support Pulumi. Is it possible for you to try the same set of configuration directly and not use Pulumi? If it happens on direct execution, then we can definitely start to look into it.

Also, a debug.log for the failure scenarios on direct run would be helpful.

[julienvincent]

The problem as @julienvincent mentioned is that you are using pulumi as the provider and we do not officially support Pulumi.

This issue was originally opened by @adam-pasabi who is using Terraform directly and is running into this problem. So I think it's clear that this is not the problem.

While I am using Pulumi myself, under the hood Pulumi is executing the same code as with Terraform - so the underlying issue is the same and is seemingly unrelated to Pulumi.

[adam-pasabi]

I can confirm I am not using Pulumi or any other layers - I am using the mongodb provider directly in Terraform. I can also confirm it happens every time I create a new environment.

In fact, I have just setup mongo via terraform on another project - this currently only provisions a mongo atlas instance and I get the same error 😬 .

@themantissa / @nikhil-mongo would you like me to create a new issue with the logs from my new project, or should I add it to this issue?

[themantissa]

@adam-pasabi just put the logs here so @martinstibbe can see them.

[adam-pasabi]

@themantissa / @martinstibbe

Here's the logs (the last bit anyway):

-----------------------------------------------------: timestamp=2022-03-09T20:23:19.518Z
2022-03-09T20:23:19.518Z [INFO]  provider.terraform-provider-mongodbatlas_v1.2.0: 2022/03/09 20:23:19 [DEBUG] status for MongoDB cluster: [REDACTED]: CREATING: timestamp=2022-03-09T20:23:19.518Z
2022-03-09T20:23:19.518Z [INFO]  provider.terraform-provider-mongodbatlas_v1.2.0: 2022/03/09 20:23:19 [TRACE] Waiting 1m0s before next try: timestamp=2022-03-09T20:23:19.518Z
2022-03-09T20:24:19.524Z [INFO]  provider.terraform-provider-mongodbatlas_v1.2.0: 2022/03/09 20:24:19 [DEBUG] MongoDB Atlas API Request Details:
---[ REQUEST ]---------------------------------------
GET /api/atlas/v1.0/groups/[REDACTED]/clusters/[REDACTED] HTTP/1.1
Host: cloud.mongodb.com
User-Agent: terraform-provider-mongodbatlas/1.2.0 go-mongodbatlas/0.14.0 (darwin;amd64)
Accept: application/json
Accept-Encoding: gzip


-----------------------------------------------------: timestamp=2022-03-09T20:24:19.523Z
2022-03-09T20:24:19.533Z [ERROR] vertex "module.infrastructure.mongodbatlas_cluster.cluster" error: error creating MongoDB Cluster: Get "https://cloud.mongodb.com/api/atlas/v1.0/groups/[REDACTED]/clusters/[REDACTED]": unexpected EOF
2022-03-09T20:24:20.013Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-03-09T20:24:20.016Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/mongodb/mongodbatlas/1.2.0/darwin_amd64/terraform-provider-mongodbatlas_v1.2.0 pid=15270
2022-03-09T20:24:20.016Z [DEBUG] provider: plugin exited

If you let me know how, I can send you the account ID - our plan doesn't include support tho', so we can't use that avenue.

[martinstibbe]

@adam-pasabi I was able to take your TF example and attach reference GCP peering example to it and it ran to completion and I was able to connect to DB from vm on GCP side I will attach zip of hybrid example to tes
hybrid.zip
t

[adam-pasabi]

hey @martinstibbe thanks for the sample, and it's great that it works for you, but it fails every. single. time. for me and my team, and even in a new project that currently only runs the mongoatlas side of things (the other side has to be done manually for reasons out of my control).

I wonder if I can give you access to our organisation on atlas and you could try running your code in there to see if it's something specific to the way our organisation is set up.

It's still my intention to get terraform-provider-mongodbatlas code compiled locally so I can add some debugging to understand what is failing, but I haven't been able to carve out the time for it just yet due to other work deadlines etc.

[themantissa]

@adam-pasabi wanted to check in. We can't have the devs get access but if you put in an Atlas support ticket we could perhaps have support help reproduce with you within your org if you are still having issues.

[alonm-totango]

was this solved? im running into this as well

[Zuhairahmed]

hi @alonm-totango can you share your Terraform config file + Logs so we can reproduce this issue on our side?

[Zuhairahmed]

closing this one, but feel free to open new issue with config files + any other details that would be helpful on our side to reproduce this issue. thank you