Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add advanced instruct questions #102

Merged
merged 1 commit into from
Apr 2, 2024
Merged

Add advanced instruct questions #102

merged 1 commit into from
Apr 2, 2024

Conversation

wongjingping
Copy link
Collaborator

Add advanced instruct questions.
Our columns include "instructions" (which is meant for inserting into the prompt, as well "correct_instructions" (which is the actual instruction that the model is supposed to follow, added only for easier reference/debugging).

Some sample benchmarks:

python3 main.py \
  -db postgres \
  -q data/instruct_advanced_postgres.csv \
  -o results/openai_gpt4_turbo_advanced.csv \
  -g oa \
  -f prompts/prompt_openai.md \
  -m gpt-4-0125-preview \
  -c 0 \
  -p 20
Preparing questions...
Using all questions from data/instruct_advanced_postgres.csv
Correct so far: 37/64 (57.81%): 100%|███████████████████████████████████████████████████████████████████████████████████████████| 64/64 [00:44<00:00,  1.45it/s]
                 query_category  num_rows  mean_correct  mean_error_db_exec
0         instructions_cte_join        16         0.625              0.0625
1       instructions_cte_window         8         0.375              0.2500
2        instructions_date_join        16         0.500              0.1250
3  instructions_string_matching         8         0.875              0.0000
4            keywords_aggregate         8         0.750              0.0000
5                keywords_ratio         8         0.375              0.0000
Average correct rate: 0.58

python3 main.py \
  -db postgres \
  -q data/instruct_advanced_postgres.csv \
  -o results/openai_gpt3.5_turbo_advanced.csv \
  -g oa \
  -f prompts/prompt_openai.md \
  -m gpt-3.5-turbo-0125 \
  -c 0 \
  -p 20
Preparing questions...
Using all questions from data/instruct_advanced_postgres.csv
Correct so far: 30/64 (46.88%): 100%|███████████████████████████████████████████████████████████████████████████████████████████| 64/64 [00:08<00:00,  7.64it/s]
                 query_category  num_rows  mean_correct  mean_error_db_exec
0         instructions_cte_join        16        0.5625              0.1250
1       instructions_cte_window         8        0.1250              0.5000
2        instructions_date_join        16        0.3750              0.1875
3  instructions_string_matching         8        0.7500              0.0000
4            keywords_aggregate         8        0.6250              0.0000
5                keywords_ratio         8        0.3750              0.0000
Average correct rate: 0.47

Note that this isn't rigorously refined so there will be room for improving on these questions

@wongjingping wongjingping marked this pull request as ready for review April 2, 2024 01:58
Copy link
Member

@rishsriv rishsriv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this!

@rishsriv rishsriv merged commit e0226c6 into main Apr 2, 2024
2 checks passed
@rishsriv rishsriv deleted the jp/adv_data branch April 2, 2024 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants