failed to create AWS account: status: 404, body: {"message":"Not found"}

[johncblandii]

Terraform Version

1.5.6

Terraform Grafana Provider Version

3.14.1

Grafana Version

Cloud

Affected Resource(s)

• grafana_cloud_provider_aws_account

Terraform Configuration Files

(multiple files are combined to show the core of the set-up and just what's necessary)

variable "grafana_account_id" {
  type        = string
  description = "The AWS account ID of Grafana Labs."
  default     = "008923505280"
}

locals {
  grafana_account_id = var.grafana_account_id
  iam_role_name      = module.this.id
}

data "aws_iam_policy_document" "trust_grafana" {
  statement {
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${local.grafana_account_id}:root"]
    }
    actions = ["sts:AssumeRole"]
    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = [coalesce(var.prometheus_user_id, module.grafana_stack.outputs.grafana_cloud_stack.prometheus_user_id)]
    }
  }
}

resource "aws_iam_role" "this" {
  name        = local.iam_role_name
  description = "Role used by Grafana AWS integration."
  # Allow Grafana Labs' AWS account to assume this role.
  assume_role_policy = data.aws_iam_policy_document.trust_grafana.json
}

# This policy allows the role to discover metrics via tags and export them.
resource "aws_iam_role_policy" "this" {
  name = local.iam_role_name
  role = aws_iam_role.this.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "apigateway:GET",
          "aps:ListWorkspaces",
          "autoscaling:DescribeAutoScalingGroups",
          "cloudwatch:GetMetricData",
          "cloudwatch:ListMetrics",
          "dms:DescribeReplicationInstances",
          "dms:DescribeReplicationTasks",
          "ec2:DescribeSpotFleetRequests",
          "ec2:DescribeTransitGatewayAttachments",
          "shield:ListProtections",
          "storagegateway:ListGateways",
          "storagegateway:ListTagsForResource",
          "tag:GetResources",
        ]
        Resource = "*"
      }
    ]
  })
}

resource "grafana_cloud_provider_aws_account" "this" {
  stack_id = module.grafana_stack.outputs.grafana_cloud_stack.id
  role_arn = aws_iam_role.this.arn

  regions = [
    "us-east-1",
  ]
}

Expected Behavior

It should create the AWS connection as expected in Grafana.

Actual Behavior

│ Error: Failed to create AWS Account
│ 
│   with grafana_cloud_provider_aws_account.this,
│   on main.tf line 77, in resource "grafana_cloud_provider_aws_account" "this":
│   77: resource "grafana_cloud_provider_aws_account" "this" {
│ 
│ failed to create AWS account: status: 404, body: {"message":"Not found"}

Here is the debug view of the request (not much more info, tbh):

�[0m�[1mgrafana_cloud_provider_aws_account.this: Creating...�[0m�[0m
2024-12-10T00:38:31.919Z [INFO]  Starting apply for grafana_cloud_provider_aws_account.this
2024-12-10T00:38:31.919Z [DEBUG] grafana_cloud_provider_aws_account.this: applying the planned Create change
2024-12-10T00:38:31.920Z [DEBUG] provider.terraform-provider-grafana_v3.14.1: 2024/12/10 00:38:31 [DEBUG] POST https://somewhere.grafana.net/api/v2/stacks/0123456789/aws/accounts
2024-12-10T00:38:32.113Z [ERROR] provider.terraform-provider-grafana_v3.14.1: Response contains error diagnostic: diagnostic_detail="failed to create AWS account: status: 404, body: {"message":"Not found"}
" diagnostic_summary="Failed to create AWS Account" tf_req_id=83e4236a-d56d-0z65-ce40-c119008977d3 tf_rpc=ApplyResourceChange @module=sdk.proto diagnostic_severity=ERROR tf_proto_version=5.7 tf_provider_addr=registry.terraform.io/grafana/grafana tf_resource_type=grafana_cloud_provider_aws_account @caller=github.com/hashicorp/terraform-plugin-go@v0.25.0/tfprotov5/internal/diag/diagnostics.go:58 timestamp=2024-12-10T00:38:32.111Z
2024-12-10T00:38:32.114Z [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-12-10T00:38:32.114Z [ERROR] vertex "grafana_cloud_provider_aws_account.this" error: Failed to create AWS Account

Steps to Reproduce

1. Follow https://grafana.com/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/aws/cloudwatch-metrics/config-cw-metrics/#configure-automatically-with-terraform

Important Factoids

I ran through the test on the "add a new scrape job" web version /a/grafana-csp-app/aws/configuration/cloudwatch/create and the connection auth validates, but I'm not 100% sure if that request is simply checking my user's ability to submit such a request or if it is actually testing the account.

I went through the IAM role to see if that was being checked and it seems it isn't. I butchered the role (removed permissions) and it still showed a green check so I'm not 100% sure on what that part is doing.

I also verified the token had the proper permissions for integration management and stack reading. I tried the test with 2 different keys as well just to see if maybe I was losing my mind. lol

References

No response

[johncblandii]

@tristanburgess your message says to add a label, but I cannot add any labels. Can you?

[tristanburgess]

Hello @johncblandii, hope you are doing well!. Thanks for pointing this out, I will add the label and edit my message. It looks like the issue from your debug output is an invalid endpoint and stack ID for accessing the Grafana Cloud Provider API

2024-12-10T00:38:31.920Z [DEBUG] provider.terraform-provider-grafana_v3.14.1: 2024/12/10 00:38:31 [DEBUG] POST https://somewhere.grafana.net/api/v2/stacks/0123456789/aws/accounts

Thus it seems to me that the Grafana provider specification needs to be adjusted in accordance with https://registry.terraform.io/providers/grafana/grafana/latest/docs#managing-cloud-provider

and the stack ID output module.grafana_stack.outputs.grafana_cloud_stack.id checked for validity (it seems to be 0123456789, likely a default sentinel value).

[johncblandii]

Hello @johncblandii, hope you are doing well!. Thanks for pointing this out, I will add the label and edit my message. It looks like the issue from your debug output is an invalid endpoint and stack ID for accessing the Grafana Cloud Provider API

2024-12-10T00:38:31.920Z [DEBUG] provider.terraform-provider-grafana_v3.14.1: 2024/12/10 00:38:31 [DEBUG] POST https://somewhere.grafana.net/api/v2/stacks/0123456789/aws/accounts

Thus it seems to me that the Grafana provider specification needs to be adjusted in accordance with https://registry.terraform.io/providers/grafana/grafana/latest/docs#managing-cloud-provider

and the stack ID output module.grafana_stack.outputs.grafana_cloud_stack.id checked for validity (it seems to be 0123456789, likely a default sentinel value).

That id is just me editing it. The stack id is valid. I confirmed that.

The code matches the suggested setup and the values are valid for each input.

[johncblandii]

Oh and is there a way to get the cloud provider URL without calling the API?

I'll verify if that's the issue, but it'd be better if creating a stack gave you that value as an output vs mixing in a manual step.

[johncblandii]

Ok...the cloud_provider_url isn't the stack url. I think that was my confusion on the provider. In watching the demo video, I thought the url was their chosen stack url. 🤦🏽

Should I submit a feature request to have that returned on the stack?

[tristanburgess]

Hello @johncblandii, no problem, glad to hear that you sorted it out. You may file a feature request, although it may be helpful to know that we are aware of this issue, we just don't currently have a timeline for fixing it, yet. In any case, your request can be used to help with prioritization, so it would be much appreciated. Thanks!

[johncblandii]

Hello @johncblandii, no problem, glad to hear that you sorted it out. You may file a feature request, although it may be helpful to know that we are aware of this issue, we just don't currently have a timeline for fixing it, yet. In any case, your request can be used to help with prioritization, so it would be much appreciated. Thanks!

I created this one: #1950.

Thanks for the quick help, @tristanburgess.