[BUG] azure-ai-evaluation: Cannot run built-in evaluators against multi-turn conversation input

[Wixee]

• Package Name: azure-ai-evaluation
• Package Version: 1.15.0
• Operating System: Windows

Describe the bug
Error raised when running the following script.
Error: azure.ai.evaluation._exceptions.EvaluationException: (InternalError) unhashable type: 'list'

To Reproduce
data.csv:

conversation
"{""messages"": [{""role"": ""user"", ""content"": ""Hi""}, {""role"": ""assistant"", ""content"": ""Hello""}]}"
"{""messages"": [{""role"": ""user"", ""content"": ""Hi""}, {""role"": ""assistant"", ""content"": ""Hello""}, {""role"": ""user"", ""content"": ""How are you?""}, {""role"": ""assistant"", ""content"": ""I am fine""}]}"

main.py:

import json
import os
from azure.ai.evaluation import (
    evaluate,
    CoherenceEvaluator,
    AzureOpenAIModelConfiguration,
)
from azure.identity import DefaultAzureCredential

MODEL_CONFIG = AzureOpenAIModelConfiguration(
    azure_endpoint="xxx",
    azure_deployment="xxx",
    api_version="2024-12-01-preview",
)


def target_function(conversation: str):
    return {"conversation": json.loads(conversation)}


if __name__ == "__main__":
    data_path = os.path.join(os.path.dirname(__file__), "data.csv")

    result = evaluate(
        data=data_path,
        target=target_function,
        evaluators={
            "coherence": CoherenceEvaluator(
                model_config=MODEL_CONFIG,
                credential=DefaultAzureCredential(),
                # is_reasoning_model=True,
            )
        },
        evaluator_config={
            "default": {
                "column_mapping": {
                    "conversation": "${target.conversation}",
                }
            }
        },
    )
    print(result)

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @luigiw @needuv @singankit.

[w-javed]

@Wixee Were you able to verify same with jsonl format.
For example:
https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/tests/e2etests/data/evaluate_test_data_conversation.jsonl

[w-javed]

If you want to use CSV format.
Here is an example.
https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/evaluate/evaluate_with_various_inputs/qr_data.csv?plain=1

Please remove unnecessary double quotes.

[Wixee]

Hi @w-javed, the bug only occurs when the input is multi-turn conversation, so the bug cannot repro via https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/evaluate/evaluate_with_various_inputs/qr_data.csv?plain=1

The bug can be repro with https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/tests/e2etests/data/evaluate_test_data_conversation.jsonl