Provider produced inconsistent final plan with automated logpush ownership challenge

[Arnall]

Confirmation

• My issue isn't already found on the issue tracker.
• I have replicated my issue using the latest version of the provider and it is still present.

Terraform and Cloudflare provider version

Terraform v1.5.1
on linux_amd64

• provider registry.terraform.io/cloudflare/cloudflare v4.14.0
• provider registry.terraform.io/hashicorp/google v4.81.0

Affected resource(s)

• cloudflare_logpush_job
• cloudflare_logpush_ownership_challenge

Terraform configuration files

resource "cloudflare_logpush_ownership_challenge" "ownership_challenge" {
  zone_id = module.cloudflare_id.zone_id[0]

  destination_conf = local.destination_conf
}
data "google_storage_bucket_object_content" "challenge_file" {
  bucket = "<bucket_name>"
  name   = cloudflare_logpush_ownership_challenge.ownership_challenge.ownership_challenge_filename
}
resource "cloudflare_logpush_job" "http_logs" {
  zone_id = module.cloudflare_id.zone_id[0]

  dataset                     = "http_requests"
  destination_conf            = local.destination_conf
  enabled                     = true
  filter                      = var.filter
  frequency                   = var.frequency
  logpull_options             = "fields=${local.http_fields}&timestamps=unixnano&CVE-2021-44228=${var.log4j}"
  max_upload_bytes            = var.max_upload_bytes
  max_upload_interval_seconds = var.max_upload_interval_seconds
  ownership_challenge         = data.google_storage_bucket_object_content.challenge_file.content
}

Link to debug output

https://gist.github.com/Arnall/5a5823566c96191a4cbb1ec5ae7f90b4

Panic output

No response

Expected output

no error during apply

Actual output

"cloudflare_logpush_job.http_logs" error: Provider produced inconsistent final plan

When expanding the plan for cloudflare_logpush_job.http_logs to include new
values learned so far during apply, provider
"registry.terraform.io/cloudflare/cloudflare" produced an invalid new value
for .ownership_challenge: was null, but now
cty.StringVal("[redacted]..[redacted]").

This is a bug in the provider, which should be reported in the provider's
own issue tracker.

Steps to reproduce

Just try to create cloudflare_logpush_job and cloudflare_logpush_ownership_challenge resources with one plan/apply.

Additional factoids

gist for the apply : https://gist.github.com/Arnall/d58bc2b172fd995a415cdc61b6aa2ae4

The problem seems to be during the plan, the provider tries to resolve ownership_challenge in the cloudflare_logpush_job resource. But the challenge does not exist at plan time , so ownership_challenge = null (should be know after apply?). Finally at apply time, the challenge is created, and ownership_challenge now has an actual value different from the plan, which causes the error...

References

No response

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[jacobbednarz]

by the looks of this, you need an explicit depends_on for the ownership challenge resource in the GCS object to make sure it waits for it to complete. otherwise, it will just be an empty lookup.

[Arnall]

hi @jacobbednarz , sorry but adding a dependency changes nothing...

So far i tried:

resource "cloudflare_logpush_ownership_challenge" "ownership_challenge" {
  zone_id = module.cloudflare_id.zone_id[0]

  destination_conf = local.destination_conf
}
data "google_storage_bucket_object_content" "challenge_file" {
  depends_on = [resource.cloudflare_logpush_ownership_challenge.ownership_challenge]

  bucket = "[REDACTED]"
  name   = cloudflare_logpush_ownership_challenge.ownership_challenge.ownership_challenge_filename
}

or

data "google_storage_bucket_object_content" "challenge_file" {
  bucket = "[REDACTED]"
  name   = cloudflare_logpush_ownership_challenge.ownership_challenge.ownership_challenge_filename
}
resource "cloudflare_logpush_job" "http_logs" {
  depends_on = [data.google_storage_bucket_object_content.challenge_file]

  zone_id = module.cloudflare_id.zone_id[0]

  dataset                     = "http_requests"
  destination_conf            = local.destination_conf
  enabled                     = true
  filter                      = var.filter
  frequency                   = var.frequency
  logpull_options             = "fields=${local.http_fields}&timestamps=unixnano&CVE-2021-44228=${var.log4j}"
  max_upload_bytes            = var.max_upload_bytes
  max_upload_interval_seconds = var.max_upload_interval_seconds
  ownership_challenge         = data.google_storage_bucket_object_content.challenge_file.content
}

or

resource "cloudflare_logpush_ownership_challenge" "ownership_challenge" {
  zone_id = module.cloudflare_id.zone_id[0]

  destination_conf = local.destination_conf
}

resource "cloudflare_logpush_job" "http_logs" {
  depends_on = [resource.cloudflare_logpush_ownership_challenge.ownership_challenge]

  zone_id = module.cloudflare_id.zone_id[0]

  dataset                     = "http_requests"
  destination_conf            = local.destination_conf
  enabled                     = true
  filter                      = var.filter
  frequency                   = var.frequency
  logpull_options             = "fields=${local.http_fields}&timestamps=unixnano&CVE-2021-44228=${var.log4j}"
  max_upload_bytes            = var.max_upload_bytes
  max_upload_interval_seconds = var.max_upload_interval_seconds
  ownership_challenge         = data.google_storage_bucket_object_content.challenge_file.content
}

it always lead to the same plan :

  # data.google_storage_bucket_object_content.challenge_file will be read during apply
  # (config refers to values not yet known)
 <= data "google_storage_bucket_object_content" "challenge_file" {
      + bucket              = "[REDACTED]"
      + cache_control       = (known after apply)
      + content_disposition = (known after apply)
      + content_encoding    = (known after apply)
      + content_language    = (known after apply)
      + content_type        = (known after apply)
      + crc32c              = (known after apply)
      + customer_encryption = (known after apply)
      + detect_md5hash      = (known after apply)
      + event_based_hold    = (known after apply)
      + id                  = (known after apply)
      + kms_key_name        = (known after apply)
      + md5hash             = (known after apply)
      + media_link          = (known after apply)
      + metadata            = (known after apply)
      + name                = (known after apply)
      + output_name         = (known after apply)
      + self_link           = (known after apply)
      + source              = (known after apply)
      + storage_class       = (known after apply)
      + temporary_hold      = (known after apply)
    }

  # cloudflare_logpush_job.http_logs will be created
  + resource "cloudflare_logpush_job" "http_logs" {
      + dataset                     = "http_requests"
      + destination_conf            = "gs://[REDACTED]/test/test.com/{DATE}"
      + enabled                     = true
      + frequency                   = "low"
      + id                          = (known after apply)
      + logpull_options             = "fields=CacheCacheStatus,CacheResponseBytes,CacheTieredFill,ClientCountry,ClientIP,ClientIPClass,ClientRequestHost,ClientRequestMethod,ClientRequestProtocol,ClientRequestReferer,ClientRequestSource,ClientRequestURI,ClientRequestUserAgent,ClientSrcPort,EdgeColoCode,EdgeEndTimestamp,EdgePathingSrc,EdgePathingStatus,EdgeResponseBytes,EdgeResponseStatus,EdgeStartTimestamp,EdgeTimeToFirstByteMs,OriginIP,OriginResponseBytes,OriginResponseDurationMs,OriginResponseHeaderReceiveDurationMs,OriginResponseStatus,ParentRayID,RayID,SmartRouteColoID,UpperTierColoID,WorkerCPUTime,WorkerStatus&timestamps=unixnano&CVE-2021-44228=false"
      + max_upload_bytes            = 104857600
      + max_upload_interval_seconds = 300
      + zone_id                     = "[redacted]"
    }

  # cloudflare_logpush_ownership_challenge.ownership_challenge will be created
  + resource "cloudflare_logpush_ownership_challenge" "ownership_challenge" {
      + destination_conf             = "gs://[REDACTED]/test/test.com/{DATE}"
      + id                           = (known after apply)
      + ownership_challenge_filename = (known after apply)
      + zone_id                      = "[redacted]"
    }

Plan: 2 to add, 0 to change, 0 to destroy

As you can see in the plan the attribute ownership_challenge in cloudflare_logpush_job.http_logs does not appear (it's considered null) instead of appearing with (known after apply) as other attributes in cloudflare_logpush_ownership_challenge.ownership_challenge or data.google_storage_bucket_object_content.challenge_file.

If i do something wrong please tell me...
Thanks.

[aegiacometti]

this works
resource "cloudflare_logpush_ownership_challenge" "ownership_challenge" {
zone_id = var.cloudflare_zone_id
destination_conf = var.gcp_bucket_destination_url
}

resource "null_resource" "fetch_gcp_bucket_file_content" {
depends_on = [cloudflare_logpush_ownership_challenge.ownership_challenge]
provisioner "local-exec" {
command = "gsutil cat ${var.gcp_bucket_destination_url}/${cloudflare_logpush_ownership_challenge.ownership_challenge.ownership_challenge_filename} > ${path.module}/${cloudflare_logpush_ownership_challenge.ownership_challenge.ownership_challenge_filename}"
}
}

data "local_file" "bucket_file" {
depends_on = [null_resource.fetch_gcp_bucket_file_content]
filename = "${path.module}/${cloudflare_logpush_ownership_challenge.ownership_challenge.ownership_challenge_filename}"
}

resource "cloudflare_logpush_job" "job_http_requests" {
depends_on = [data.local_file.bucket_file]
enabled = true
zone_id = var.cloudflare_zone_id
name = var.lougpush_name
destination_conf = var.gcp_bucket_destination_url
logpull_options = var.logpull_job_options
ownership_challenge = data.local_file.bucket_file.content
dataset = "http_requests"
}

[Kaitou786]

@Arnall I am running into the same issue even after putting an explicit apply, were you able to solve this?

[Kaitou786]

@jacobbednarz
adding on top of it, it works without problem with S3 but fails with gcs for some reason.