numeric match and topic tag #72
Conversation
|
|
||
|
|
||
| class AIME_PIPELINETag(AIME_PIPELINE): | ||
| """This class specifies the config for running AIME benchmark 5 repeated times""" |
There was a problem hiding this comment.
update comment so it reflects the functionality of the class
| # Each query is tagged with one or more topics from arithmetic, algebra, counting, geometry, number theory, and probability and other. | ||
| # These topics follow the description on the official website: https://artofproblemsolving.com/wiki/index.php/American_Invitational_Mathematics_Examination?srsltid=AfmBOooSIQ8ua5aJX00ZtYCKDuOAB4I4c-YE9zr1xYZ86fq8x5RL2sEg. | ||
| # In their own words, "The AIME tests mathematical problem solving with arithmetic, algebra, counting, geometry, number theory, and probability and other secondary school math topics" | ||
| return pipeline |
There was a problem hiding this comment.
Since the class inherits from the original AIME_PIPELINE it will continue to run the rest of the AIME_PIPELINE but with the tagging prompt. For example, this means that it will also try to extract an answer and generate the report. There are two options here: 1) Either to not inherit from AIME_PIPELINE, or 2) Inherit from AIME_PIPELINE but then return only the components you need in the pipeline. For example,
return PipelineConfig(
[
self.data_processing_comp,
self.inference_comp,
self.data_post_processing,
],
self.log_dir,
)
In case 2, also requires changing the answer extractor as the marker is different here.
Add a new metric "numeric match" to compare two numeric values.
Add a pipeline to extract topics from a math question.