Sync ADA code with private repo version #8
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update Private Repo
* Created separate SQL queries for dag_id and task_id * Removed duplicate tests * Added DS Store file to gitignore
EPMKEY-27127 - Dag_id api is returning wrong values
This filter will help dag estimation time by providing a minimum and maximum time for dag and task run. Default values: min = 1 minute, max = 1 day
* Since the newest release of pandas creates a warning when sqlalchemy isn't used, the read_sql statement was changed to use psycopg directly. * The convertion to a dataframe is necessary to preserve the format of the output and the increased latency is shown to be negligible in local stress testing * the dockerfile has been formatted following hadolint suggestions
EPMKEY-29076 - Upgrade Python Version and Dependencies
This reverts commit f81cb9f.
Revert "Implementing outliers filters for ADA"
implementing outliers filters
EPMKEY-31877 Receive outliers parameters
EPMKEY-33495 implement task percentage limit
EPMKEY-42497 DevEx pipeline migration
EPMKEY-42497 DevEx tests
EPMKEY-42497 DevEx adjustments
EPMKEY-42497 Rename apps and use nonprod
EPMKEY-42497 Include 'nonprod' changes
* update encrypted credentials * Trigger Build
* new limit and score * fixes
* fix contributing link * fix contents
* EPMKEY-43109 Update build credentials (#30) * update encrypted credentials * Trigger Build * Sync Nonprod with master (#34) * EPMKEY-43109 Update build credentials (#30) (#31) * update encrypted credentials * Trigger Build * EPMP-1849 New Score And Limit (#32) * new limit and score * fixes --------- Co-authored-by: Victoria Resende <victoria.resende@ibm.com> * dag_id and operator changes in query (#36) * dag_id and operator changes in query * fixes --------- Co-authored-by: Victoria Resende <victoria.resende@ibm.com> Co-authored-by: Tobias Conran Zorzetto <tobiaszorzetto@ibm.com>
* EPMKEY-43109 Update build credentials (#30) * update encrypted credentials * Trigger Build * Sync Nonprod with master (#34) * EPMKEY-43109 Update build credentials (#30) (#31) * update encrypted credentials * Trigger Build * EPMP-1849 New Score And Limit (#32) * new limit and score * fixes --------- Co-authored-by: Victoria Resende <victoria.resende@ibm.com> * dag_id and operator changes in query (#36) * dag_id and operator changes in query * fixes * changes to how the query aproximates values, using only ceiling instead as casting as int before (#38) --------- Co-authored-by: Victoria Resende <victoria.resende@ibm.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Since ADA is an ongoing effort within the responsible team at IBM, it is necessary to synchronize the version currently running in production with the one in the public repository. That is the objective of this PR.
Description of change
Description of each PR being synced:
PR Number: 11
PR Title: EPMKEY-27127 - Dag_id api is returning wrong values
PR Description: This change will fix the wrong values that ADA has been returning when dag_id is used for the queries
PR Number: 12
PR Title: Implementing outliers filter
PR Description: This filter will help dag estimation time by providing a minimum and maximum time choice for dag and task run. The default values are tmin = 1 minute, tmax = 1 day
PR Number: 13
PR Title: EPMKEY-29076 - Upgrade Python Version and Dependencies
PR Description: This story aims to solve some of the technical debt in Ada by upgrading the Python version to 3.10 and any dependencies that needs to be upgraded by that migration.
PR Number: 14
PR Title: fixing the outlier commit
PR Description: These outliers filters will help make the estimated time of a dag more accurate, by providing a minimum and maximum time for dag and task run. The default values are outlier_min = 0 minutes and outlier_max = 1 day.
PR Number: 15
PR Title: Revert "Implementing outliers filters for ADA"
PR Description: This reverts commit f81cb9f.
PR Number: 16
PR Title: implementing outliers filters
PR Description: These outliers filters will help make the estimated time of a dag more accurate, by providing a minimum and maximum time for dag and task run. The default values are outlier_min = 0 minutes and outlier_max = 1 day.
PR Number: 17
PR Title: EPMKEY-31877 Receive outliers parameters
PR Description: The command to get the parameter value when sent was added. When nothing is passed, the default values will be used.
Also, if an invalid parameter, for example not an integer, is entered, an exception will be thrown.
PR Number: 18
PR Title: EPMKEY-33495 implement task percentage limit
PR Description: This new filter will help to make sure that there are not so many stuck pods unnecessarily, making score calculations based on a percentage of the most recent runs of a task, keeping these values more updated.
PR Number: 19
PR Title: EPMKEY-37097 include percentage limit for dags
PR Description: Insert a limit in the query to reduce the scope of the amount of dags considered in the calculations, returning a more accurate estimated time.
PR Number: 20
PR Title: Fix requirements
PR Description: Fix issue with python dependencies
PR Number: 21
PR Title: Fix vulnerabilities caused by Flask and Cryptography packages
PR Description: This change will fix the vulnerabilities currently present caused by outdated flask and cryptography packages
PR Number: 22
PR Title: Bump cryptography + Fix UTs
PR Description: The current cryptography package version contains a vulnerability, so this PR aims to fix it
PR Number: 23
PR Title: EPMKEY-41530 Sonarqube
PR Description: Enable Sonarqube
PR Number: 24
PR Title: EPMKEY-42497 DevEx pipeline migration
PR Description: This PR aims to implement the migration of the pipeline from Cirrus to CIO DevEx.
PR Number: 25
PR Title: EPMKEY-42497 DevEx tests
PR Description: This PR aims to test a change for CIO CI/CD migration
PR Number: 26
PR Title: EPMKEY-42497 DevEx adjustments
PR Description: This PR aims to change the names of the applications and their routes to improve overall understanding.
PR Number: 27
PR Title: EPMKEY-42497 Rename apps and use nonprod
PR Description: This PR aims to rename the applications so that there is an application with a route that can be used for testing purposes before the changes reach the master branch. It also intends to update the contribution docs to document the new approach and how to correctly use the branches.
PR Number: 28
PR Title: EPMKEY-42497 Include 'nonprod' changes
PR Description: This PR aims to rename the applications and their routes.
It also intends to update the contribution docs to document the new approach and how to correctly use the branches.
PR Number: 29
PR Title: EPMKEY-43109 Update build credentials
PR Description: This PR aims to update the cirrus encrypted credentials in the build.yml file.
PR Number: 30
PR Title: EPMKEY-43109 Update build credentials
PR Description: This PR aims to update the cirrus encrypted credentials in the build.yml file.
PR Number: 31
PR Title: EPMKEY-43109 Update build credentials
PR Description: This PR aims to update the cirrus encrypted credentials in the build.yml file for prod environment.
PR Number: 32
PR Title: EPMP-1849 New Score And Limit
PR Description: This Pr aims to aims to change the score and the limit used by ada to calculate metrics.
PR Number: 33
PR Title: revert
PR Description: revert pr
PR Number: 34
PR Title: Sync Nonprod with master
PR Description: pushing what should be in nonprod from master
PR Number: 35
PR Title: Fix contribuiting link
PR Description: This pr fixes the link to the contributing file, which was pointing to the old one in the public repo
PR Number: 36
PR Title: dag_id and operator changes in query
PR Description: fix task query to use dag_id and check if operator is CrdTrigger
PR Number: 37
PR Title: ADA - Move To Production
PR Description: Move changes from NonProd to Prod
PR Number: 38
PR Title: STDDEV calculation and formula - Use of ceiling
PR Description: After the last change made on the score, an analysis of its behavior was done and something was noticed. The faster tasks were being wrongfully accused of being stuck much more the the slower ones. That was because they were having scores proportionally smaller comparing to its average than the bigger ones.
After some investigation, it was found that in many cases the stddev used for the score formula was rounded to its floor, which in many cases even made the stddev value to be 0 decreasing dramatically the value of the score, specially for the tasks that average lower minutes, a change by only a unit of stddev can be proportionally really significant to the formula.
One example is: given a task which takes
2.8 minutesin average to run and has a stddev of0.5, if the stddev is rounded to0, then the score will be4, while if the stddev is rounded up, the score will be9, more than double!