Add additional evaluations on an already-run experiment?

[omrihar]

I have an experiment I already ran with some evaluations, but would like to add others without having to re-run the entire suite since it's quite long (and token-heavy). Is there a possibility to do that and attach the additional metrics to the experiment?

[dosubot[bot]]

Yes, you can add additional evaluations to an already-run experiment without re-running the entire suite. Use the evaluate_experiment function to append new evaluators to your existing experiment:

from phoenix.experiments import evaluate_experiment

experiment = evaluate_experiment(experiment, evaluators=[contains_keyword, conciseness])

This allows you to attach new metrics to the experiment results ^[1].

To continue talking to Dosu, mention @dosu.

[axiomofjoy]

Hey @omrihar, this makes a lot of sense. We don't currently have a way to do this, but I've filed a ticket here.

[omrihar]

For now I'll try to get it from the sqlite database in the background ;)

[axiomofjoy]

Actually, looks like this is already supported via the get_experiment method on the client. You pass in the experiment_id and then can use evaluate_experiment. Hope this helps!

[axiomofjoy]

Improved docs here.

[omrihar]

Thanks @axiomofjoy , I'll try it out. Another couple of questions regarding experiments:

1. In the UI I don't see a way to filter experiment results based on outcomes. Is this something you plan to add?
2. In the UI I don't see a way to add an example from an experiment to another dataset. this can be useful, for example, to create a dataset of "difficult" examples to focus on in iterations.
3. Is there a way to see both span / document evals performed on the experiment traces together with the evaluation results from the experiments themselves? e.g. if I want to drill down into all of the failed examples according to the experiment evaluation and check whether, for example, the LLM hallucinated based on the span evaluation.

[axiomofjoy]

1. In the UI I don't see a way to filter experiment results based on outcomes. Is this something you plan to add?
2. In the UI I don't see a way to add an example from an experiment to another dataset. this can be useful, for example, to create a dataset of "difficult" examples to focus on in iterations.

NIce suggestions, I've filed issues here and here. Feel free to add color to the tickets.

3. Is there a way to see both span / document evals performed on the experiment traces together with the evaluation results from the experiments themselves? e.g. if I want to drill down into all of the failed examples according to the experiment evaluation and check whether, for example, the LLM hallucinated based on the span evaluation.

This one sounds like it's related to this feature request. It sounds like you want the span and document evaluators to show up as evaluations on the experiment results. Is that right?

[omrihar]

@axiomofjoy about 3 - yes, that sounds like a good idea. I think it makes a lot of sense to be able to deep-dive in this way into the experiment to really use the results properly.

Another question while we're at it - is there a way to download the experiment as a dataframe? So, get the question, ai answer, correct answer and the evaluations run during the experiment as a dataframe to save as csv or do further analysis on?

[axiomofjoy]

Another question while we're at it - is there a way to download the experiment as a dataframe? So, get the question, ai answer, correct answer and the evaluations run during the experiment as a dataframe to save as csv or do further analysis on?

The experiment object returned by run_experiment and evaluate_experiment has as_dataframe and get_evaluations methods that return dataframes for the experiment data and evaluations, respectively.

[omrihar]

Thanks again @axiomofjoy I've tried this, but when I don't use run_experiment or evaluate_experiment I don't see such methods. I've tried to use

experiment = px.Client().get_experiment(experiment_id="....")

but that doesn't work. Is there a way to retrospectively get the experiment object returned from run_experiment or evaluate_experiment?

[axiomofjoy]

Hey @omrihar, we don't currently return the fully hydrated experiment object from get_experiment. Added a feature request here!