Improve documentation for terraform test

[novekm]

Terraform Version

Terraform v1.10.4
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.94.0

Your version of Terraform is out of date! The latest version
is 1.11.3. You can update by downloading from https://www.terraform.io/downloads.html

Affected Pages

https://developer.hashicorp.com/terraform/cli/commands/test#terraform-test-command

What is the docs issue?

The current documentation for terraform test is not explicit enough with the example usage, especially for the optional parameters such as -filter. The docs mention:

-filter=testfile - Limits the terraform test operation to the specified test files.

However the above doesn't work. With a set of tests within a recommended /tests directory with filenames such as 01_test.tftest.hcl, 02_test.tftest.hcl etc, running terraform test -filter=02_test.tftest.hcl or even terraform test -filter="02_test.tftest.hcl" does not work:

❯ terraform test -filter=02_test.tftest.hcl

Success! 0 passed, 0 failed.

❯ terraform test -filter="02_test.tftest.hcl"

Success! 0 passed, 0 failed.

Instead, you must write the filter as a string, and include the full path to the file, including the /tests directory.

❯ terraform test -filter="tests/02_test.tftest.hcl"
tests/02_test.tftest.hcl... in progress
  run "unit_test"... pass
  run "e2e_test"... pass
tests/02_test.tftest.hcl... tearing down
tests/02_test.tftest.hcl... pass

I could understand this if using a non-standard directory name for your tests, however this is not expected if using the default recommended naming of /tests. One would assume that if terraform test automatically uses test files (*.tftest.hcl) that are contained in a directory with this name, then the simple filter should function the same way.

Proposal

Please update the docs with more examples and explicitly mention the way you must use filters to reference specific test files, and call out that it's required to directly reference the directory name, even if it is the default value. In the longterm, perhaps it would be possible to omit having to pass in the directory name in the filter if using the default naming of /tests 🙂

References

terraform test documentation

[liamcervante]

Hi @novekm, thanks for filing this. We do have examples of using the -filter option within Test Directory section. The docs team might take a look at your feedback and iterate on things anyway, but I did want to highlight that.

One thing to consider is that you can also always write tests directly within the root configuration directory. Terraform will pick up tests both within the tests directory and the root directory. This is why the full path to the test file is required within the -filter option, so that is unlikely to change in future iterations.

[novekm]

Thanks Liam, that makes sense. Related to some of the other flags for Terraform test - are there any examples for targeting speficic directories for tests beyond the default /tests directory, and any limitations on naming/nesting?

I've been trying terraform test -test-directory=<my-desired-folder> but for some reason when using this I can't successful run terraform init. Rather, I run into a cycle where I receive a success message about the initialization, but then with a subsequent run of terraform test am prompted to initlize again.

Additional challenges arose when trying to have a folder named modules within a /tests directory. This was done to allow for a single tests directory and test a variety of modules, as well as larger solutions that use a collection of modules. This was the folder structue:

─ cloud-game-development-toolkit/
    ├── modules
    └── tests/
        ├── modules/
        │   ├── jenkins/
        │   │   ├── 01_mandatory_test.tftest.hcl
        │   │   └── 02_custom_test.tftest.hcl
        │   ├── perforce/
        │   │   ├── 01_mandatory_test.tftest.hcl
        │   │   └── 02_custom_test.tftest.hcl
        │   └── unreal/
        │       └── horde/
        │           ├── 01_mandatory_test.tftest.hcl
        │           └── 02_custom_test.tftest.hcl
        └── solutions/
            ├── simple-build-pipeline/
            │   ├── 01_mandatory_test.tftest.hcl
            │   └── 02_custom_test.tftest.hcl
            └── audio-pipeline/
                ├── 01_mandatory_test.tftest.hcl
                └── 02_custom_test.tftest.hcl

The goal was to be able to group these tests and with a GitHub action (comment on PR), trigger tests within specific sub-directories for the modules with something like "/test-perforce" (during updates to the module) or "/test-all-modules" (when cutting a new version release for the entire project). Running terraform test and trying to target specific directories when having the name of modules within the test directory threw this error:

This persisted even after renaming the sub directory and re-initializing Terraform. I ended up having to delete the entire .Terraform directory and the lock file, restart my IDE, and re-run Terraform init to resolve. I'm assuming modules is a reserved name and made Terraform expect to see a module configuration instead of tests for the modules.

[liamcervante]

One thing to bear in mind is you need to specify the test directory when initialising as well: terraform init -test-directory=<whatever>.

The init command can't discover all the required dependencies for tests that aren't in the default locations without that extra hint.

There should be no restrictions on naming or structure other than the test directory has to be "beneath" the configuration directory. It's also worth noting that Terraform can't load tests from multiple directories at a time and it doesn't load directories recursively.

[novekm]

Hi @liamcervante, unfortunately I am still having some issues trying to use the -test-directory flag. This is what my folder structure looks like (condensed to relevant directories/files):

cloud-game-development-toolkit
├── modules
│   ├── README.md
│   ├── jenkins
│   │   ├── README.md
│   │   ├── alb.tf
│   │   ├── asg.tf
│   │   ├── data.tf
│   │   ├── ecs.tf
│   │   ├── efs.tf
│   │   ├── examples
│   │   │   └── complete
│   │   ├── fsxz.tf
│   │   ├── iam.tf
│   │   ├── local.tf
│   │   ├── outputs.tf
│   │   ├── s3.tf
│   │   ├── sg.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   ├── perforce
│   │   ├── examples
│   │   │   └── complete
│   │   ├── helix-authentication-service
│   │   │   ├── README.md
│   │   │   ├── alb.tf
│   │   │   ├── data.tf
│   │   │   ├── iam.tf
│   │   │   ├── local.tf
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── variables.tf
│   │   │   └── versions.tf
│   │   ├── helix-core
│   │   │   ├── README.md
│   │   │   ├── data.tf
│   │   │   ├── iam.tf
│   │   │   ├── locals.tf
│   │   │   ├── main.tf
│   │   │   ├── outputs.tf
│   │   │   ├── variables.tf
│   │   │   └── versions.tf
│   │   └── helix-swarm
│   │       ├── README.md
│   │       ├── alb.tf
│   │       ├── data.tf
│   │       ├── elasticache.tf
│   │       ├── iam.tf
│   │       ├── locals.tf
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── sg.tf
│   │       ├── variables.tf
│   │       └── versions.tf
│   ├── s3
│   │   ├── examples
│   │   │   ├── advanced
│   │   │   └── basic
│   │   ├── main.tf
│   │   ├── tests
│   │   │   ├── 01_mandatory_basic.tftest.hcl
│   │   │   └── 02_mandatory_advanced.tftest.hcl
│   │   └── variables.tf
│   ├── teamcity
│   │   ├── README.md
│   │   ├── examples
│   │   │   └── simple
│   │   ├── local.tf
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── variables.tf
│   │   └── versions.tf
│   └── unreal
│       ├── horde
│       │   ├── README.md
│       │   ├── alb.tf
│       │   ├── asg.tf
│       │   ├── config
│       │   ├── docdb.tf
│       │   ├── ecs.tf
│       │   ├── elasticache.tf
│       │   ├── examples
│       │   ├── iam.tf
│       │   ├── local.tf
│       │   ├── outputs.tf
│       │   ├── sg.tf
│       │   ├── variables.tf
│       │   └── versions.tf
│       └── unreal-cloud-ddc
│           ├── unreal-cloud-ddc-infra
│           └── unreal-cloud-ddc-intra-cluster

The s3 module/directory is what I'm currently trying to test:

├── examples
│   ├── advanced
│   │   └── main.tf
│   └── basic
│       └── main.tf
├── main.tf
├── tests
│   ├── 01_mandatory_basic.tftest.hcl
│   └── 02_mandatory_advanced.tftest.hcl
└── variables.tf

The s3 module and related tests are very basic currently, just to try to test the correct usage of the flags with terraform test. All the module does is create an S3 bucket with a supplied variable for the bucket prefix.

basic example (main.tf)

module "s3_bucket" {
  source = "../../"

  bucket_prefix = "kevon-test-basic"
}

For the tests, they simply reference the examples in the examples directories and use that for 2 run blocks:

01_mandatory_basic.tftest.hcl

run "unit_test" {
  command = plan
  module {
    source = "./examples/basic"
  }
}

run "e2e_test" {
  command = apply
  module {
    source = "./examples/basic"
  }
}

02_mandatory_advanced.tftest.hcl

run "unit_test" {
  command = plan
  module {
    source = "./examples/advanced"
  }
}

run "e2e_test" {
  command = apply
  module {
    source = "./examples/advanced"
  }
}

If I understand what you're saying, to run a test against that specific /tests directory, I need to do the following (run from the root of the project directory (cloud-game-development-toolkit):

1. Initialize Terraform with context to the module you want to test

terraform init -test-directory=modules/s3/tests

2. Run all terraform tests that are present in the target directory

terraform test -test-directory=modules/s3/tests

However, when I try to run the first command, I get a response that the Terraform was initialized in an empty directory:

❯ terraform init -test-directory=modules/s3/tests
Terraform initialized in an empty directory!

To get this to work, I instead had to use the -chdir flag in terraform and do the following:

terraform -chdir=modules/s3 init

Initializing the backend...
Initializing modules...
- test.tests.02_mandatory_advanced.unit_test in examples/advanced
- test.tests.02_mandatory_advanced.unit_test.s3_bucket in .
- test.tests.02_mandatory_advanced.e2e_test in examples/advanced
- test.tests.02_mandatory_advanced.e2e_test.s3_bucket in .
- test.tests.01_mandatory_basic.unit_test in examples/basic
- test.tests.01_mandatory_basic.unit_test.s3_bucket in .
- test.tests.01_mandatory_basic.e2e_test in examples/basic
- test.tests.01_mandatory_basic.e2e_test.s3_bucket in .
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v5.94.1...
- Installed hashicorp/aws v5.94.1 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Then I didn't even have to use the -test-directory flag since the s3 module is using the default /tests naming. I just had to run:

terraform -chdir=modules/s3 test

tests/01_mandatory_basic.tftest.hcl... in progress
  run "unit_test"... pass
  run "e2e_test"... pass
tests/01_mandatory_basic.tftest.hcl... tearing down
tests/01_mandatory_basic.tftest.hcl... pass
tests/02_mandatory_advanced.tftest.hcl... in progress
  run "unit_test"... pass
  run "e2e_test"... pass
tests/02_mandatory_advanced.tftest.hcl... tearing down
tests/02_mandatory_advanced.tftest.hcl... pass

Success! 4 passed, 0 failed.

I'm pretty confused on which should be used since I was never able to get the -test-directory flag to work.