Evaluating multiple choice questions with GPT (OpenAI Chat Completion API)

[kangqi-ni]

I am wondering is there a way to evaluate GPT (OpenAI Chat Completion API) on a set of multiple choice questions such as the mmlu tasks. I know that the chat completion api does not support log likelihood, but is there a built-in way that allows getting text/json answers from GPT and then post-process to get multiple choice evaluations?

[gonz-mart]

I was wondering the same thing. Has there been any progress?

[jonoillar]

I was wondering the same thing. Has there been any progress?

There is an open PR about it: #2601

But I don't see any progress so far 🥲

[FarisHijazi]

we really need this feature