Validation of resource factories files

[sruffilli]

Resource factories are supported by many modules and heavily used in FAST, but YAML files are currently not validated, neither on documentation examples, nor on FAST tests.

Running terraform plan on modules containing such assets is not a comprehensive test, as a typo on an optional attribute wouldn't be caught by terraform, but might potentially have destructive consequences on the infrastructure, if the plan goes through without a human review.

Formal validation is the only way to catch such errors - however maintaining the schema up-to-date (e.g. subnets) is a toil, which means any manually-generated schema which is not automatically tested is prones to become obsolete.

We could:

• Introduce schema validation on documentation, forcing PR authors to ensure that any change which includes an example is validated. This action is non comprehensive as it doesn't cover undocumented scenarios.
• Automatically generate a validation schema based on the factory variables, and test examples/assets against the generated schema.
• Abandon validation altogether (true as of today)

[sruffilli]

• Introduce schema validation on documentation, forcing PR authors to ensure that any change which includes an example is validated. This action is non comprehensive as it doesn't cover undocumented scenarios.

We've ruled this out

• Automatically generate a validation schema based on the factory variables, and test examples/assets against the generated schema.

There's no easy way to do this - any approach has to go through recursively parsing object types (which the hcl2 python parser doesn't do), which is quite complex and outweights the benefits.

• Abandon validation altogether (true as of today)

I'm afraid this is it :)