singlestoredb_private_connection Outbound connection on refresh isnt found

[mattthaber]

Description

• we are currently using singlestoredb_private_connection to create an OUTBOUND connection
• This has worked for about a week , when suddenly on an apply, when refreshing the resource it determined it no longer existed in singlestore, and tried to create it again (not recreate, just a fresh create)

17:37:00.786 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform: singlestoredb_private_connection.outbound["XXXXX"]: Refreshing state... [id=3522df17-ef74-40b4-beaa-35801db33a5f]

17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:   # singlestoredb_private_connection.outbound["XXXXX"] will be created
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:   + resource "singlestoredb_private_connection" "outbound" {
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + active_at           = (known after apply)
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + created_at          = (known after apply)
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + deleted_at          = (known after apply)
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + endpoint            = (known after apply)
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + id                  = (known after apply)
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + outbound_allow_list = (known after apply)
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + service_name        = "XXXXXX"
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + sql_port            = 3306
17:37:01.265 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + status              = (known after apply)
17:37:01.266 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + type                = "OUTBOUND"
17:37:01.266 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + updated_at          = (known after apply)
17:37:01.266 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + web_socket_port     = 443
17:37:01.266 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + workspace_group_id  = "XXXX"
17:37:01.266 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:       + workspace_id        = "XXXXX"
17:37:01.266 STDOUT [multitenant_singlestore/us-west-2/stg/singlestore] terraform:     }

the private connection seemingly still exists in the dashboard , but now we are in a state for which terraform doesnt match whats deployed and it cant be recreated cause of the following error

17:38:01.778 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ Error: SingleStore API client call failed
17:38:01.778 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ 
17:38:01.778 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │   with singlestoredb_private_connection.outbound["com.amazonaws.vpce.us-west-2.vpce-svc-05fd13826333287da"],
17:38:01.778 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │   on workspace.tf line 52, in resource "singlestoredb_private_connection" "outbound":
17:38:01.778 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │   52: resource "singlestoredb_private_connection" "outbound" {
17:38:01.778 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ 
17:38:01.778 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ An unexpected error occurred when calling SingleStore API. If the error is
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ not clear, please report the issue [https://www.singlestore.com/support/.](https://www.singlestore.com/support/)
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ 
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ SingleStore client error: Post
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ "https://api.singlestore.com/v1/privateConnections": giving up after 5
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ attempts, unexpected status code: Internal Server Error, response: failed
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ to create endpoints for workspace group
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ (a0893676-09a0-4735-a0e8-41d2227eb410): an Active Outbound private
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │ connection already exists
17:38:01.779 STDERR [multitenant_singlestore/us-west-2/stg/singlestore] terraform: │

re-importing the private connection back in after this worked, but its concerning to me that this happened in the first place and its not clear why. our terraform state shouldnt be drifting like this

[noprysk-ua]

Hi @mattthaber, thank you for reporting the issue!

I'm glad that re-importing worked. However, we're investigating why it has happened in the first place. We'll inform you once we're done with the investigation.

[noprysk-ua]

@mattthaber our team took a close look at the issue across all the layers (provider, Management API, and internal services) and failed to identify the area that could have caused the drift. Moreover, we re-tested the functionality and couldn't get to the described state.

In case the connection is deleted externally (from portal UI), the provider deletes it from the state file (here) and attempts to create a new connection. If there's already an active outbound connection that's not recognized by the provider, the attempt to re-create fails.

Also, we checked the history of private connections and there were no errors related to outbound connections of the organization starting from 2025-04-09 UTC.

In case you face the issue again or collect more data that may help us debug the issue, please, let us know.

https://memsql.atlassian.net/issues/MCDB-72751