Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

progress indicators are wrong when steps run simultaneously #2195

Open
coyotemarin opened this issue Aug 19, 2020 · 1 comment
Open

progress indicators are wrong when steps run simultaneously #2195

coyotemarin opened this issue Aug 19, 2020 · 1 comment
Labels
Milestone

Comments

@coyotemarin
Copy link
Collaborator

_parse_progress_from_resource_manager() assumes that there will be at most one job running on a cluster at the same time, which is wrong now that clusters can run steps concurrently.

If we know a step's StartTime from the ListSteps API, that seems to only be a few seconds off of Start Time in the resource manager UI. So that's a way we could possibly match up step progress correctly.

It would be really nice if there EMR API would tell us the mapping between EMR step IDs and YARN application IDs, but so far I haven't found one.

@coyotemarin coyotemarin added this to the v0.7.4 milestone Aug 19, 2020
@coyotemarin
Copy link
Collaborator Author

Since we now have code to talk to the resource manager API, we can guess the application ID for the step from the apps API (based on start time) and then get its progress from the app API.

@coyotemarin coyotemarin modified the milestones: v0.7.4, v0.7.5 Sep 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant