Flaky test: TestAccNetworkSecuritySecurityProfile_networkSecuritySecurityProfileOverridesExample

[duvni]

Community Note

Error example:

=== NAME  TestAccNetworkSecuritySecurityProfile_networkSecuritySecurityProfileOverridesExample
    resource_network_security_security_profile_generated_test.go:83: Step 1/2 error: After applying this test step, the plan was not empty.
        stdout:
        
        
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        
        Terraform will perform the following actions:
        
          # google_network_security_security_profile.default will be updated in-place
          ~ resource "google_network_security_security_profile" "default" {
                id               = "organizations/123456789/locations/global/securityProfiles/tf-test-my-security-profile8u00tq84h2"
                name             = "tf-test-my-security-profile8u00tq84h2"
                # (10 unchanged attributes hidden)
        
              ~ threat_prevention_profile {
                  ~ severity_overrides {
                      ~ action   = "DENY" -> "ALLOW"
                      ~ severity = "HIGH" -> "INFORMATIONAL"
                    }
                  ~ severity_overrides {
                      ~ action   = "ALLOW" -> "DENY"
                      ~ severity = "INFORMATIONAL" -> "HIGH"
                    }
        
                    # (1 unchanged block hidden)
                }
            }
        
        Plan: 0 to add, 1 to change, 0 to destroy.
--- FAIL: TestAccNetworkSecuritySecurityProfile_networkSecuritySecurityProfileOverridesExample (29.45s)

This happens because the severityOverrides and threatOverrides properties of threatPreventionProfile are incorrectly marked as Array instead of Set which is their real internal representation.

Terraform Version & Provider Version(s)

Terraform v1.9.6
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.10.4. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

google_network_security_security_profile

Terraform Configuration

No response

Debug Output

No response

Expected Behavior

No response

Actual Behavior

No response

Steps to reproduce

make testacc TEST=./google/services/networksecurity TESTARGS='-run=TestAccNetworkSecuritySecurityProfile_networkSecuritySecurityProfileOverridesExample'

Important Factoids

No response

References

No response

b/391927028

[duvni]

I've created GoogleCloudPlatform/magic-modules#12814 to fix this.

[ggtisc]

Hi @duvni

Could you please share with us the terraform code you are using WITHOUT USING MODULES, VARIABLES or LOCALS to ensure this is an issue as well as being clearer with the steps? For sensitive data you could use examples like:

1. project = "project-20970"
2. member = "user:user-20970@domain-20970.com"

[duvni]

Hi [@duvni](https://github.com/duvni)

Could you please share with us the terraform code you are using WITHOUT USING MODULES, VARIABLES or LOCALS to ensure this is an issue as well as being clearer with the steps? For sensitive data you could use examples like:

1. project = "project-20970"
2. member = "user:user-20970@domain-20970.com"

Sorry, I'll share a more detailed reproduction steps that I take, I'm not making any changes to the code just running one of the existing tests:

1. Clone a fresh provider using: git clone https://github.com/hashicorp/terraform-provider-google.git $GOPATH/src/github.com/hashicorp/terraform-provider-google
2. Navigate to the provider: cd $GOPATH/src/github.com/hashicorp/terraform-provider-google
3. Make sure to set all relevant variables for the local run, specifically this test is testing an org-level resource, so we will need an organization number and project for billing (quota for the API call):

• export USER_PROJECT_OVERRIDE=true
• export GOOGLE_BILLING_PROJECT=your-project-id
• export GOOGLE_USE_DEFAULT_CREDENTIALS=true
• export GOOGLE_ORG=123456789 (your organization number)
• export GOOGLE_PROJECT=your-project-id
• export GOOGLE_REGION=us-central1
• export GOOGLE_ZONE=us-central1-a

4. Run the SP Overrides tests multiple times, the test should be flaky: go clean -testcache && make testacc TEST=./google/services/networksecurity TESTARGS='-run=TestAccNetworkSecuritySecurityProfile_networkSecuritySecurityProfileOverridesExample'

An example of a failure:

    resource_network_security_security_profile_generated_test.go:83: Step 1/2 error: After applying this test step, the plan was not empty.
        stdout:
        
        
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        
        Terraform will perform the following actions:
        
          # google_network_security_security_profile.default will be updated in-place
          ~ resource "google_network_security_security_profile" "default" {
                id               = "organizations/123456789/locations/global/securityProfiles/tf-test-my-security-profile98rjzrvl76"
                name             = "tf-test-my-security-profile98rjzrvl76"
                # (10 unchanged attributes hidden)
        
              ~ threat_prevention_profile {
                  ~ severity_overrides {
                      ~ action   = "DENY" -> "ALLOW"
                      ~ severity = "HIGH" -> "INFORMATIONAL"
                    }
                  ~ severity_overrides {
                      ~ action   = "ALLOW" -> "DENY"
                      ~ severity = "INFORMATIONAL" -> "HIGH"
                    }
        
                    # (1 unchanged block hidden)
                }
            }
        
        Plan: 0 to add, 1 to change, 0 to destroy.
--- FAIL: TestAccNetworkSecuritySecurityProfile_networkSecuritySecurityProfileOverridesExample (30.79s)

Alternatively, you can run a terraform script, creating only one resource:

resource "google_network_security_security_profile" "default" {
  name        = "test-security-profile"
  parent      = "organizations/%{org_number_here}"
  description = "my description"
  type        = "THREAT_PREVENTION"

  threat_prevention_profile {
    severity_overrides {
      action   = "ALLOW"
      severity = "INFORMATIONAL"
    }

    severity_overrides {
      action   = "DENY"
      severity = "HIGH"
    }

    threat_overrides {
      action    = "ALLOW"
      threat_id = "280647"
    }
  }
}

You will see that after initially applying this and creating the security profile, subsequent calls to terraform plan may show diffs which are unexpected.

[ggtisc]

Hi @duvni

I got a 404 trying to clone the repository and this could be a more complex process than just copy-pasting the code of the resources, but after using the shared code which is the same as the terraform registry examples (link here) everything works as expected without errors or diffs, even if there are no changes and just running new terraform apply commands many times.

Could you please share the hashicorp/google provider version you are using to make a new try?

Also please confirm us if you are using modules, locals or variables for your base resource

[duvni]

Not sure why you would get a 404 trying to clone the repository, this is the official link to clone this repository in GitHub's UI (via the Code drop-down menu on the main page).

The steps to run the tests are based on the official dev-guide for working with magic modules:
https://googlecloudplatform.github.io/magic-modules/test/run-tests/
And looking at other bugs for similar tests it seemed like this repository is the right place to create the issue, but if it isn't we can close it and I'll find another way.

My recent attempt was with:

Terraform v1.12.0-dev
on linux_amd64

But I also tried Terraform v1.9.6

As for modules/locals/variable, I'm not using any, it's a simple resource used as-is like the example.
The only thing to note is that this is an org-level resource so it's not the usual work of setting up a project for it, you will need to setup an organization, have a proper IAM permission (network admin in the org-level), and also have a project set for billing. This is all taken care of by the export variables I specified in my previous comment when running the test.

Are you sure you terraform apply did actually create the SP with the overrides? You can verify with gcloud using:

gcloud network-security security-profiles describe sp-id --organization=123456789 --billing-project=proj-to-bill-id --format=json

You should also see that if you describe the resource multiple times it's not guaranteed that the specific overrides will always return in the same order.

[ggtisc]

Thanks for your support here and through chat @duvni

I confirmed the permadiff issue changing the Google provider version to 6.16.0 with a terraform init -upgrade