-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New text2sql metrics #1584
Closed
Closed
New text2sql metrics #1584
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fix: wml inference with space id only Signed-off-by: Cheng Qian <cheng.qian@ibm.com> Co-authored-by: Cheng Qian <cheng.qian@ibm.com>
Signed-off-by: elronbandel <elronbandel@gmail.com>
Signed-off-by: elronbandel <elronbandel@gmail.com>
Signed-off-by: elronbandel <elronbandel@gmail.com>
…guration Signed-off-by: elronbandel <elronbandel@gmail.com>
…tries Signed-off-by: elronbandel <elronbandel@gmail.com>
Signed-off-by: elronbandel <elronbandel@gmail.com>
Signed-off-by: elronbandel <elronbandel@gmail.com>
Signed-off-by: elronbandel <elronbandel@gmail.com>
* Fix failing tests Signed-off-by: elronbandel <elronbandel@gmail.com> * Fix failing tests Signed-off-by: elronbandel <elronbandel@gmail.com> * Fix failing tests Signed-off-by: elronbandel <elronbandel@gmail.com> --------- Signed-off-by: elronbandel <elronbandel@gmail.com>
Signed-off-by: elronbandel <elronbandel@gmail.com>
add streaming, add tests
* try lazy loadHF first Signed-off-by: dafnapension <dafnashein@yahoo.com> * reduce benchmark profiling to generating the dataset only. Not inferring (that is dome mocking anyhow) and not evaluating (of the mocked results). add trust_remote also to load_dataset_builder Signed-off-by: dafnapension <dafnashein@yahoo.com> * try procrastination for load csv too Signed-off-by: dafnapension <dafnashein@yahoo.com> * added split cache for the generators, and log limit once per data and increase loader cache Signed-off-by: dafnapension <dafnashein@yahoo.com> * make sklearn loader too - a lazy loader Signed-off-by: dafnapension <dafnashein@yahoo.com> * adjust to new readers for csv Signed-off-by: dafnapension <dafnashein@yahoo.com> * Enhance LoadHF class to support optional splits and improve dataset loading logic Signed-off-by: elronbandel <elronbandel@gmail.com> * Refactor LoadHF class to improve dataset loading and implement limit on yielded instances Signed-off-by: elronbandel <elronbandel@gmail.com> * Refactor LoadHF class to streamline dataset loading and enhance split handling Signed-off-by: elronbandel <elronbandel@gmail.com> * Remove unused import and update line number in secrets baseline Signed-off-by: elronbandel <elronbandel@gmail.com> * Refactor load_data method to simplify error handling and remove unnecessary cache checks Signed-off-by: elronbandel <elronbandel@gmail.com> * Merge origin/main Signed-off-by: elronbandel <elronbandel@gmail.com> * Refactor loaders to implement LazyLoader class and update load_iterables method for improved streaming support Signed-off-by: elronbandel <elronbandel@gmail.com> * Update exception handling in test_failed_load_csv to catch general exceptions Signed-off-by: elronbandel <elronbandel@gmail.com> * Refactor LoadHF class to streamline data loading and enhance error handling Signed-off-by: elronbandel <elronbandel@gmail.com> --------- Signed-off-by: dafnapension <dafnashein@yahoo.com> Signed-off-by: elronbandel <elronbandel@gmail.com> Co-authored-by: Elron Bandel <elronbandel@gmail.com>
Signed-off-by: elronbandel <elronbandel@gmail.com>
* Add support for all Granite Guardian risks Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Remove Granite Guardian from LLM as Judge evaluators Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Rename wrong metric name (3.0 version -> 3) Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Add support for custom risks Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Adapt catalog Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Add more examples Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Apply linter Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Add generation params Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Use inference engine instead of internal model Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Add _mock_infer_log_probs to infer_log_prob Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Apply linter Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Bring back breaking catalog names changes Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Add wrongly deleted artifacts Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Only create watsonx inference engine if it is None Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Modularize getting the prompt Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Change default names to what Granite Guardian expects by default Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Adapt rag granite guardian prepare file and catalog Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Adapt metric so it works for all inference engines Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Bring back certainty and improve score naming Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * fixes and format Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Adapt rag catalog Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Adapt WMLInferenceEngineBase credential check: apikey -> api_key Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Use credentials object and pass project and space Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Adapt WML log prob default params Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Adapt granite guardian catalog and fix example Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Apply linter Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Implement inheritance for each risk type Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Apply linter Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * Uncomment log prob params check Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> --------- Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com>
To resolve some merge conflicts, I'm closing this and will do a new PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal is to add additional metrics and results to the output of the text2sql execution accuracy metric implementation. We used to produce just one number:
1
if the dataframe produced by the SQLs in pred and gold are the same,0
otherwise. I've added scores to report 12 scores/outputs:What we used to get (output of
examples/evaluate_text2sql.py
):What we get with the new additions: