Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Creating multiple apps simultaneously 503 #159

Open
talbright opened this issue Dec 17, 2018 · 3 comments
Open

Creating multiple apps simultaneously 503 #159

talbright opened this issue Dec 17, 2018 · 3 comments

Comments

@talbright
Copy link
Contributor

talbright commented Dec 17, 2018 •
edited
Loading

Terraform Version

terraform -v
Terraform v0.11.10
+ provider.heroku (unversioned)
+ provider.random v2.0.0

Heroku Provider Version

master (hashicorp@08f5dba)

Affected Resource(s)

Please list the resources as a list, for example:

  • heroku_app

Terraform Configuration Files

resource "heroku_space" "default" {
  name         = "test-terraform-provider-heroku"
  organization = "${var.organization}"
  region       = "${var.region}"
}

resource "random_pet" "default" {}

resource "heroku_app" "default" {
  count  = 10
  name   = "${format("test-app-%d-%s",count.index,random_pet.default.id)}"
  space  = "${heroku_space.default.name}"
  region = "${heroku_space.default.region}"

  organization = {
    name = "${heroku_space.default.organization}"
  }
}

Debug Output

heroku_app.default.8: Still creating... (10s elapsed)
heroku_app.default.9: Still creating... (10s elapsed)
heroku_app.default.1: Still creating... (10s elapsed)
heroku_app.default.2: Still creating... (10s elapsed)
heroku_app.default.7: Still creating... (10s elapsed)
heroku_app.default.0: Still creating... (10s elapsed)
heroku_app.default.5: Still creating... (10s elapsed)
heroku_app.default.3: Still creating... (10s elapsed)
heroku_app.default.4: Still creating... (10s elapsed)
heroku_app.default.6: Still creating... (10s elapsed)
heroku_app.default[7]: Creation complete after 11s (ID: test-app-7-measured-mastodon)
heroku_app.default[1]: Creation complete after 15s (ID: test-app-1-measured-mastodon)
heroku_app.default[9]: Creation complete after 19s (ID: test-app-9-measured-mastodon)
heroku_app.default.5: Still creating... (20s elapsed)
heroku_app.default.2: Still creating... (20s elapsed)
heroku_app.default.0: Still creating... (20s elapsed)
heroku_app.default.8: Still creating... (20s elapsed)
heroku_app.default.6: Still creating... (20s elapsed)
heroku_app.default.3: Still creating... (20s elapsed)
heroku_app.default.4: Still creating... (20s elapsed)
heroku_app.default[8]: Creation complete after 23s (ID: test-app-8-measured-mastodon)
heroku_app.default[0]: Creation complete after 25s (ID: test-app-0-measured-mastodon)
heroku_app.default[4]: Creation complete after 28s (ID: test-app-4-measured-mastodon)
heroku_app.default[6]: Creation complete after 29s (ID: test-app-6-measured-mastodon)
heroku_app.default.5: Still creating... (30s elapsed)
heroku_app.default.2: Still creating... (30s elapsed)
heroku_app.default.3: Still creating... (30s elapsed)
heroku_app.default[3]: Creation complete after 32s (ID: test-app-3-measured-mastodon)
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: 2018/12/23 01:25:22 HTTP/2.0 503 Service Unavailable
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Content-Length: 561
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Cache-Control: no-cache, no-store
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Content-Type: text/html; charset=utf-8
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Date: Sun, 23 Dec 2018 06:25:22 GMT
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: Via: 2.0 spaces-router (d316c7e8a021)
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: 
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: <!DOCTYPE html>
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: <html>
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:   <head>
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:     <meta name="viewport" content="width=device-width, initial-scale=1">
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:     <meta charset="utf-8">
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:     <title>Application Error</title>
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:     <style media="screen">
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:       html,body,iframe {
2018/12/23 01:25:22 [TRACE] root: eval: *terraform.EvalWriteState
2018/12/23 01:25:22 [TRACE] root: eval: *terraform.EvalApplyProvisioners
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         margin: 0;
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         padding: 0;
2018/12/23 01:25:22 [TRACE] root: eval: *terraform.EvalIf
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:       html,body {
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         height: 100%;
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         overflow: hidden;
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:       }
2018/12/23 01:25:22 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* heroku_app.default.9: Post https://api.heroku.com/teams/apps: encountered an error : 503 Service Unavailable
2018/12/23 01:25:22 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* heroku_app.default.9: Post https://api.heroku.com/teams/apps: encountered an error : 503 Service Unavailable
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku: 
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:       iframe {
2018-12-23T01:25:22.776-0500 [DEBUG] plugin.terraform-provider-heroku:         width: 100%;
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:         height: 100%;
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:         border: 0;
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:       }
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:     </style>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:   </head>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:   <body>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:     <iframe src="//www.herokucdn.com/error-pages/application-error.html"></iframe>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku:   </body>
2018-12-23T01:25:22.777-0500 [DEBUG] plugin.terraform-provider-heroku: </html>

Error: Error applying plan:

2 error(s) occurred:

* heroku_app.default[5]: 1 error(s) occurred:

* heroku_app.default.5: Post https://api.heroku.com/teams/apps: encountered an error : 503 Service Unavailable
* heroku_app.default[2]: 1 error(s) occurred:

* heroku_app.default.2: Post https://api.heroku.com/teams/apps: encountered an error : 503 Service Unavailable

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error

Expected Behavior

All ten apps should have been created.

Actual Behavior

Some were created, then received a 503 error from the platform API.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply
@talbright talbright mentioned this issue Dec 20, 2018
Closed
@talbright
Copy link
Contributor Author

@mars should we add 503 as a condition to heroku/heroku-go@3b96879? I suspect the error handling may be off in the back-end, for it to respond with HTML instead of JSON.

@mars
Copy link
Member

mars commented Dec 23, 2018

Hi @talbright 🎄😄

I'd be curious if this is still reproducible, as it appears to be evidence of an incident with the platform API.

I agree that adding a special 503 response status case to the client would be beneficial, so that the provider can sanely handle this "API is offline" condition.

Although I am not familiar with the specific error reported here (creating many apps at once), this kind of Heroku error normally comes from the load balancer when a Heroku app is offline.

@talbright
Copy link
Contributor Author

🎅 yo!

I did reproduce it last night -- the HCL above has recreated this 💯of the time so far.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mars @talbright