[FR]: Support to view thread evaluations in Experiments Tab

[FingerLiu]

Proposal summary

For now we can only see evaluations of traces(grouped by dataset) in experiments, but we can not view thread evaluations in experiments to see overview performance.
This is inconvenient to use thread evaluations(although we can we the score for each single thread in thread)

Motivation

No response

[alexkuzmik]

Hi @FingerLiu! As of today, an experiment as a concept in Opik is suitable when you evaluate some LLM task on a dataset (and every task(dataset_item) call produces a single trace).
This is currently not the case for threads, because you can only evaluate existing logged threads, not read/generate them during the experimentation phase dynamically. However, we are looking into improving the UX in that direction.
Could you please share:

1. Are you using online or SDK-side thread evaluation?
2. What are your main pain points with viewing the evaluated threads' data?
3. What kind of overview performance details/charts/tables do you need for threads?

Thanks for opening the issue! Looking forward to hear back from you :)

[FingerLiu]

@alexkuzmik Thanks for your quick response.

1. I am using SDK-side thread evaluation.
2. I am building AI for science agent, which will build complex parameters and call async jobs. The params are complex, it usually takes multi turn of QA to generate full param and finally call tools to run async jobs. Since then, I need to optimize my agent to build and call async jobs correctly. I think I need a Line chart to view scores between different experiments.
3. I need to see line chart grouped by different experiments.

---

In addition, I’ve also been thinking about and experimenting with best practices for testing multi-turn conversation threads, and I’d like to share some of these ideas with you and hear your thoughts.

When evaluating an agent’s overall performance in multi-turn dialogues, it’s difficult to rely on a fixed dataset of question–answer pairs (like we do for single-turn evaluation). That’s because we can’t know in advance how many turns the conversation will take, nor what specific questions the model will ask while waiting for the user’s responses.

To address this challenge, I see two possible approaches:

1. Direct deployment with user feedback
   We could release the agent and then collect user feedback to score its real-world performance. This approach is straightforward but has downsides: it’s time-consuming, and it makes it hard to systematically test before deployment. As a result, regressions in certain aspects of performance may go unnoticed until they negatively affect user experience.

2. Introduce a “human simulator agent”
   To enable automated testing, we can create a simulated user agent with certain constraints, which interacts with the agent under test in a multi-turn dialogue. After reaching a predefined goal, we can collect the full conversation thread and evaluate it.
   Similar to how trace-scoped evaluation uses a fixed dataset, this gives us a bounded yet dynamic setup with a “human simulator agent” to conduct systematic testing.

In this way, we can also build a dataset of multi-turn threads, consisting of an initial question plus a dynamically responding simulator agent.

[alexkuzmik]

Great feedback! That's for sure will help us with shaping the future features for that direction.

Regarding the 2 approaches, yeah, we also thought about it. Option 1 is for sure really time-consuming and will probably make sense only for some small subset of users who can afford such "experiments". Option 2 is interesting, but the value you can get from synthetic user emulation is arguable, we were not sure that it's worth investing in it for now either. (so here we are, with standalone evaluations but without the experiments yet :)

Actually, if you go with the second approach, you can use our existing opik.evaluate functionality instead of evaluate_threads that works for threads logged to Opik.
The steps will be:

1. Generate a synthetic dataset with conversations
2. Create an Opik dataset from it
3. Define task as a simple extractor of conversation data from the dataset item.
4. Pass all of the required threads metrics to evaluate function.

If you take a look at thread evaluation metrics, you'll see that they all have a similar signature:

from typing import Union, List, Any

from . import types
from .. import base_metric, score_result


class ConversationThreadMetric(base_metric.BaseMetric):
    """Abstract base class for all conversation thread metrics."""

    def score(
        self, conversation: types.Conversation, **kwargs: Any
    ) -> Union[score_result.ScoreResult, List[score_result.ScoreResult]]:
        raise NotImplementedError(
            "Please use concrete metric classes instead of this one."
        )

    async def ascore(
        self, conversation: types.Conversation, **kwargs: Any
    ) -> Union[score_result.ScoreResult, List[score_result.ScoreResult]]:
        """
        Async public method that can be called independently.
        """
        raise NotImplementedError(
            "Please use concrete metric classes instead of this one."
        )

whereas Conversation is essentially a pretty well-known format - List[Dict[Literal["role", "content"], str]].

So, you can have experiments, the path to it is just a bit more unusuall than with regular one-step tasks. You're pre-computing the multi-step input-output pairs, put them into the dataset and just extract them during the experiment inside the task callback.
At least that's what I'd do if I wanted to play with pre-defined datasets today :)

Let me know if that works for you!

[juanferrub]

Hi @FingerLiu
Interesting conversation here!

On top of what @alexkuzmik mentioned, here are a couple of additional thoughts that might help:

• Don't ignore the power of real human sessions. Nothing beats authentic human interactions. If you can collect 50–100 real sessions, that becomes an invaluable foundation— not very scalable, I know, but maybe even better than starting with synthetic data. These conversations are grounded in reality and give you a trustworthy “golden dataset.” Once you have that baseline, you can confidently explore synthetic conversation generation to scale testing.

• Compress multi-turn into single-turn tests. Many multi-turn dialogues can be distilled into targeted single-turn interactions to make evaluation easier. For example, if a conversation spans seven turns, you can compress it into a single structured request that directly probes your agent’s ability on that specific dimension. This helps you test more systematically without always replaying full-length conversations.

Let us know what you think!