Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic instruct questions #100

Merged
merged 1 commit into from
Mar 28, 2024
Merged

Add basic instruct questions #100

merged 1 commit into from
Mar 28, 2024

Conversation

wongjingping
Copy link
Collaborator

@wongjingping wongjingping commented Mar 28, 2024

  • Add basic instruct questions
  • Upgrade openai's runner and standard prompt generator to bypass pruning when num_columns = 0

sample test runs with openai on these data:

$ python main.py \
  -db postgres \
  -q data/instruct_basic_postgres.csv \
  -o results/openai_gpt4_turbo_basic.csv \
  -g oa \
  -f prompts/prompt_openai.md \
  -m gpt-4-0125-preview \
  -c 0 \
  -p 20
/Users/jp/workspace/miniconda3/lib/python3.11/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Preparing questions...
Using all questions from data/instruct_basic_postgres.csv
Correct so far: 38/40 (95.00%): 100%|██████████████████████████| 40/40 [00:16<00:00,  2.38it/s]
                      query_category  num_rows  mean_correct  mean_error_db_exec
0            basic_group_order_limit         8          1.00                 0.0
1  basic_join_date_group_order_limit         8          1.00                 0.0
2                basic_join_distinct         8          1.00                 0.0
3       basic_join_group_order_limit         8          0.75                 0.0
4                    basic_left_join         8          1.00                 0.0
Average correct rate: 0.95

@wongjingping wongjingping requested a review from rishsriv March 28, 2024 03:18
Copy link
Member

@rishsriv rishsriv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! c=0 is super useful. Thank you!

@rishsriv
Copy link
Member

Just need to do a quick fix for the failing tests, then good to merge!

@wongjingping wongjingping force-pushed the jp/instruct_basic branch 3 times, most recently from 62b6892 to e6df160 Compare March 28, 2024 07:41
@wongjingping wongjingping merged commit 90411b4 into main Mar 28, 2024
2 checks passed
@wongjingping wongjingping deleted the jp/instruct_basic branch March 28, 2024 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants