feat: Added support for persisting Atlantis state using EFS #247

MarkIannucci · 2022-01-30T06:59:35Z

Description

Use EFS to store Atlantis state (we are most concerned about locks, but this also reduces repo checkouts in some cases).

Motivation and Context

Fixes #206

Breaking Changes

Reduces the flexibility we offer by requiring the user variable to be in the uid:gid format. This is because I wanted to use the user variable for the posix_user defined in the aws_efs_access_point and I wasn't aware of a method to decompose the more flexible entries we allowed previously into their uid and gid parts. If someone is aware of a more elegant way to do that, I look forward to learning from your contribution! In the meantime, I added a validation to the variable to surface the breaking change at plantime rather than after the deployment.

I also change the default maximum deployment percent and minimum healthy deployment percent to 100 and 0 respectively. I made this change because I found that Atlantis gets angry when there are two instances sharing the same BoltDB database. This means that we have brief interruptions when we make changes that require a new ECS deployment in AWS, but I think it is better to have the interruptions than lost locks.

How Has This Been Tested?

I have tested and validated these changes using one or more of the provided examples/* projects --- I tested this with the github example.

Explore more persistence

antonbabenko

Looks pretty good. Few comments mostly about code style to match the rest of the module(s) we have.

main.tf

Co-authored-by: Anton Babenko <[email protected]>

…orm-aws-atlantis into EFS

Co-authored-by: Anton Babenko <[email protected]>

…orm-aws-atlantis into EFS

MarkIannucci · 2022-02-09T07:31:38Z

@antonbabenko , Thanks for the thorough review. We differed in one place on line 205 -- I left the comment unresolved to highlight it. This is ready for a re-review and potential completion.

antonbabenko

Looks pretty good. Just a comment about default values. Also, please make CI green by running pre-commit run -a and committing the changes.

variables.tf

…initial apply

variables.tf

antonbabenko · 2022-02-10T15:01:16Z

Thanks @MarkIannucci for the contribution!

## [3.10.0](v3.9.0...v3.10.0) (2022-02-10) ### Features * Added support for persisting Atlantis state using EFS ([#247](#247)) ([79d152e](79d152e))

antonbabenko · 2022-02-10T15:01:23Z

This PR is included in version 3.10.0 🎉

borissavelev · 2022-02-14T17:02:01Z

main.tf

+  # we coalescelist in order to specify the resource keys when we create the subnets using the VPC or they're specified for us.  This works around the for_each value depends on attributes which can't be determined until apply error
+  for_each = zipmap(coalescelist(var.private_subnets, var.private_subnet_ids), local.private_subnet_ids)
+
+  file_system_id  = aws_efs_file_system.this[0].id


@MarkIannucci
first of all, thank you! 🙏

Unfortunately this is not working if I want to have enable_ephemeral_storage=true

╷ │ Error: Invalid index │ │ on .terraform/modules/atlantis/main.tf line 438, in resource "aws_efs_mount_target" "this": │ 438: file_system_id = aws_efs_file_system.this[0].id │ ├──────────────── │ │ aws_efs_file_system.this is empty tuple │ │ The given key does not identify an element in this collection value. ╵

there is no condition for var.enable_ephemeral_storage

@borissavelev , thank you for the bug report. I'm sorry about the problem. I created #254 to track the resolution.

no problem! thank you)

borissavelev · 2022-02-14T17:05:38Z

main.tf

+  vpc_id      = local.vpc_id
+  description = "Security group allowing access to the EFS storage"
+
+  ingress_cidr_blocks = [var.cidr]


var.cidr can be empty when is reusing existing VPC

probably ingress_cidr_blocks is unnecessary here because later we have ingress_with_source_security_group_id

…m-aws-modules#247) Co-authored-by: Anton Babenko <[email protected]>

## [3.10.0](terraform-aws-modules/terraform-aws-atlantis@v3.9.0...v3.10.0) (2022-02-10) ### Features * Added support for persisting Atlantis state using EFS ([terraform-aws-modules#247](terraform-aws-modules#247)) ([79d152e](terraform-aws-modules@79d152e))

…m-aws-modules#247) Co-authored-by: Anton Babenko <[email protected]>

## [3.10.0](terraform-aws-modules/terraform-aws-atlantis@v3.9.0...v3.10.0) (2022-02-10) ### Features * Added support for persisting Atlantis state using EFS ([terraform-aws-modules#247](terraform-aws-modules#247)) ([79d152e](terraform-aws-modules@79d152e))

…m-aws-modules#247) Co-authored-by: Anton Babenko <[email protected]>

## [3.10.0](terraform-aws-modules/terraform-aws-atlantis@v3.9.0...v3.10.0) (2022-02-10) ### Features * Added support for persisting Atlantis state using EFS ([terraform-aws-modules#247](terraform-aws-modules#247)) ([79d152e](terraform-aws-modules@79d152e))

github-actions · 2022-11-08T02:34:24Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

MarkIannucci added 6 commits January 18, 2022 11:15

hardcoded EFS implementation proof of concept

298b7a2

prefer EFS, don't build it if ephemeral is specified

684098b

work through syntax errors

5e6c8b2

Merge pull request #1 from MarkIannucci/ExploreMorePersistence

947ff30

Explore more persistence

remove hardcoded mount_point

47f6dfd

Add validation for user variable

0c2b5b2

MarkIannucci changed the title ~~Persist Atlantis state using EFS~~ fix: Persist Atlantis state using EFS Jan 30, 2022

Merge branch 'master' into PersistInEFS

154a95c

MarkIannucci mentioned this pull request Feb 4, 2022

Leverage EFS to persist atlantis locks between deployments? #206

Closed

antonbabenko requested changes Feb 4, 2022

View reviewed changes

antonbabenko and others added 14 commits February 8, 2022 21:45

Merge branch 'master' into PersistInEFS

83db2ad

use conditional for enabling dns

b6d233e

consistentify the count declaration location, add newlines

8f31f4c

Convert names to this

7e6ebbc

Co-authored-by: Anton Babenko <[email protected]>

Merge branch 'PersistInEFS' of https://github.com/MarkIannucci/terraf…

5a7e48f

…orm-aws-atlantis into EFS

finish refactoring the objects to this

f6572b8

Simplify conditional

a748aae

Co-authored-by: Anton Babenko <[email protected]>

Merge branch 'PersistInEFS' of https://github.com/MarkIannucci/terraf…

4331aef

…orm-aws-atlantis into EFS

switch to submodule for nfs/efs defaults

99774f1

add ingress security group rule

7ecc9ce

use correct rule name?

12be167

correct syntax on sg

07ebf5d

Add newlines

36bc94d

use for_each instead of count approach

aa65e66

MarkIannucci requested a review from antonbabenko February 9, 2022 07:28

antonbabenko requested changes Feb 9, 2022

View reviewed changes

variables.tf Show resolved Hide resolved

MarkIannucci added 3 commits February 9, 2022 14:57

remove explicit user configuration. Use default value instead

bd303ff

formatting and other pre-commit updates

05f99b2

switch to local.mount_points

3d6b675

MarkIannucci added 2 commits February 10, 2022 05:20

use variables instead of calculated locals which may not be known on …

b9827af

…initial apply

update formatting

a230370

antonbabenko changed the title ~~fix: Persist Atlantis state using EFS~~ feat: Added support for persisting Atlantis state using EFS Feb 10, 2022

antonbabenko approved these changes Feb 10, 2022

View reviewed changes

variables.tf Show resolved Hide resolved

antonbabenko merged commit 79d152e into terraform-aws-modules:master Feb 10, 2022

antonbabenko pushed a commit that referenced this pull request Feb 10, 2022

chore(release): version 3.10.0 [skip ci]

ef36521

## [3.10.0](v3.9.0...v3.10.0) (2022-02-10) ### Features * Added support for persisting Atlantis state using EFS ([#247](#247)) ([79d152e](79d152e))

borissavelev reviewed Feb 14, 2022

View reviewed changes

MarkIannucci mentioned this pull request Feb 14, 2022

bug: enable_ephemeral_storage broken by update to support EFS to persist Atlantis state #254

Closed

titanlien pushed a commit to Flaconi/terraform-aws-atlantis that referenced this pull request Jun 8, 2022

feat: Added support for persisting Atlantis state using EFS (terrafor…

e282e25

…m-aws-modules#247) Co-authored-by: Anton Babenko <[email protected]>

titanlien pushed a commit to Flaconi/terraform-aws-atlantis that referenced this pull request Jun 8, 2022

feat: Added support for persisting Atlantis state using EFS (terrafor…

db3084a

…m-aws-modules#247) Co-authored-by: Anton Babenko <[email protected]>

titanlien pushed a commit to Flaconi/terraform-aws-atlantis that referenced this pull request Jun 9, 2022

feat: Added support for persisting Atlantis state using EFS (terrafor…

4eade9c

…m-aws-modules#247) Co-authored-by: Anton Babenko <[email protected]>

github-actions bot locked as resolved and limited conversation to collaborators Nov 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Added support for persisting Atlantis state using EFS #247

feat: Added support for persisting Atlantis state using EFS #247

MarkIannucci commented Jan 30, 2022

antonbabenko left a comment

MarkIannucci commented Feb 9, 2022

antonbabenko left a comment

antonbabenko commented Feb 10, 2022

antonbabenko commented Feb 10, 2022

borissavelev Feb 14, 2022

MarkIannucci Feb 14, 2022

borissavelev Feb 14, 2022

borissavelev Feb 14, 2022

borissavelev Feb 14, 2022

github-actions bot commented Nov 8, 2022

feat: Added support for persisting Atlantis state using EFS #247

feat: Added support for persisting Atlantis state using EFS #247

Conversation

MarkIannucci commented Jan 30, 2022

Description

Motivation and Context

Breaking Changes

How Has This Been Tested?

antonbabenko left a comment

Choose a reason for hiding this comment

MarkIannucci commented Feb 9, 2022

antonbabenko left a comment

Choose a reason for hiding this comment

antonbabenko commented Feb 10, 2022

antonbabenko commented Feb 10, 2022

borissavelev Feb 14, 2022

Choose a reason for hiding this comment

MarkIannucci Feb 14, 2022

Choose a reason for hiding this comment

borissavelev Feb 14, 2022

Choose a reason for hiding this comment

borissavelev Feb 14, 2022

Choose a reason for hiding this comment

borissavelev Feb 14, 2022

Choose a reason for hiding this comment

github-actions bot commented Nov 8, 2022