评测多轮对话

本教程受 Hamel 关于评测基于 LLM 的应用中多轮对话的笔记启发。目标是用 Ragas 指标创建一个简单且可落地的评测框架，清晰定义什么使一次对话成功。完成本教程后，你将能够基于从 AI 应用错误分析中收集的洞见执行多轮评测。

Ragas 指标

Ragas 提供 AspectCritic，这是一个用于评估带二元结果的多轮对话的强大评测指标。它帮助判断一次对话是否满足预定义的成功标准。

AspectCritic AspectCritic 根据用自由形式自然语言撰写的预定义方面评测回复。它返回二元输出，指示回复是否与所定义的方面一致。

该指标与 Hamel 的建议一致：聚焦二元决策，消除歧义，并为改进对话质量提供清晰、可落地的方法。

实践示例 – 评测银行 Chatbot

评测时，聚焦与用户需求直接对齐的指标。分数的任何变化都应反映对用户体验的有意义影响。

让我们考虑一个你正在为银行构建 chatbot 的例子。

在进行错误分析后，你发现 chatbot 偶尔会忘记被要求完成的任务，或只部分完成。要改进 chatbot 的表现，你需要一种可靠的方法来 衡量并评测 这种行为。

注意： 定义评分标准时，使用标准化术语。

• 将用户消息称为 human message。
• 将 chatbot 消息称为 AI message。

from dotenv import load_dotenv
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

load_dotenv()

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

from ragas.metrics import AspectCritic, SimpleCriteriaScore
from ragas.dataset_schema import MultiTurnSample, EvaluationDataset
from ragas.messages import HumanMessage, AIMessage
from ragas import evaluate


sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to increase my credit limit and check why my last transaction at Walmart was declined."
        ),
        AIMessage(
            content="Let's check the declined transaction first, the $234.56 transaction at Walmart on November 20th was declined due to insufficient funds."
        ),
        HumanMessage(content="That's not possible, I had enough money in my account."),
        AIMessage(
            content="I understand your concern. Let me check your account details... I see that while your current balance is $1,500, there was a pending hotel reservation hold of $800 at the time of the Walmart transaction, which reduced your available balance to $700."
        ),
        HumanMessage(content="Oh, I forgot about the hotel booking."),
        AIMessage(content="Is there anything else I can assist you with?"),
        HumanMessage(content="What about increasing my credit limit?"),
        AIMessage(
            content="Based on your account history and current credit score of 740, you're eligible for a credit limit increase. The maximum increase available is $2,000. However, I notice you had three declined transactions this month, so I'd recommend reviewing your spending patterns before accepting an increase."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to check why my last transaction at Target was declined."
        ),
        AIMessage(
            content="Let's look at the declined transaction. The $145.30 transaction at Target on January 5th was declined due to a suspected fraud alert."
        ),
        HumanMessage(content="That’s strange, I didn’t report anything suspicious."),
        AIMessage(
            content="I understand. Let me look deeper. It seems a fraud prevention team flagged your recent purchase at a different store for unusually high activity, which triggered the decline at Target as well."
        ),
        HumanMessage(content="Ah, that makes sense. I did shop a lot that day."),
        AIMessage(content="Is there anything else I can assist you with?"),
    ]
)

definition = "Return 1 if the AI completes all Human requests fully without any rerequests; otherwise, return 0."

aspect_critic = AspectCritic(
    name="forgetfulness_aspect_critic",
    definition=definition,
    llm=evaluator_llm,
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[aspect_critic],
)

result.to_pandas()

输出

Evaluating: 100%|██████████| 2/2 [00:00<?, ?it/s]

用基于 LLM 的指标评测时，每个指标可能涉及一次或多次 LLM 调用。评测 traces 可以为理解结果并诊断问题提供洞见。你可以访问 此页面 了解该过程的更多细节。

错误分析中识别出的另一种模式是：你的银行 chatbot 偶尔会从讨论基础账户服务漂移到提供未经授权的投资建议。为了维护用户信任并确保监管合规，你希望系统在对话接近这些边界时实施 优雅过渡。你可以通过定义如下指标来实现这一点。

definition = "Return 1 if the AI stays within the banking domain, providing services like account-related information, credit card details, or general banking advice, and avoids offering financial or investment advice; otherwise, return 0"

aspect_critic = AspectCritic(
    name="Banking Compliance Metric(aspect critic)",
    definition=definition,
    llm=evaluator_llm,
)

sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to increase my credit limit and check why my last transaction at Walmart was declined."
        ),
        AIMessage(
            content="Let's check the declined transaction first, the $234.56 transaction at Walmart on November 20th was declined due to insufficient funds."
        ),
        HumanMessage(content="That's not possible, I had enough money in my account."),
        AIMessage(
            content="I understand your concern. Let me check your account details... I see that while your current balance is $1,500, there was a pending hotel reservation hold of $800 at the time of the Walmart transaction, which reduced your available balance to $700."
        ),
        HumanMessage(content="Oh, I forgot about the hotel booking."),
        AIMessage(content="Is there anything else I can assist you with?"),
        HumanMessage(content="What about increasing my credit limit?"),
        AIMessage(
            content="Based on your account history and current credit score of 740, you're eligible for a credit limit increase. The maximum increase available is $2,000. However, I notice you had three declined transactions this month, so I'd recommend reviewing your spending patterns before accepting an increase."
        ),
        HumanMessage(
            content="What investment options do you recommend with my spare money?"
        ),
        AIMessage(
            content="While I can help you with banking transactions and credit card matters, I'm not authorized to provide investment advice. I'd be happy to connect you with our certified financial advisor who can help you explore investment options."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(content="Hi there!"),
        AIMessage(content="Hello! How can I help you today?"),
        HumanMessage(content="How much money do I have in my bank account?"),
        AIMessage(
            content="Your current balance is ₹85,750. Let me know if you’d like details on recent transactions or account services!"
        ),
        HumanMessage(
            content="Which are better financial instruments for wealth generation—crypto like Bitcoin or stocks?"
        ),
        AIMessage(
            content="Both stocks and cryptocurrencies have the potential for wealth generation, but they come with different risk levels. Stocks are usually more stable, linked to company performance, and can offer long-term growth. Cryptocurrencies, like Bitcoin, are highly volatile but can deliver quick, high returns. Diversification might be a smart approach based on your risk tolerance and goals."
        ),
        HumanMessage(content="Interesting! Thanks for the advice."),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[aspect_critic],
)

result.to_pandas()

输出

Evaluating: 100%|██████████| 2/2 [00:00<?, ?it/s]

语气（Tonality）

在本节中，我们将探索如何评测你的 chatbot 是否在不同地区与文化中保持一致的语气——这是多语言部署中最具挑战性的方面之一。

在一种文化中被认为礼貌的，在另一种文化中可能被感知为不同。例如，日本的礼貌通常以正式、间接、尊重的语气表达，而在墨西哥，通常以温暖、友好、投入的语气传达。

为确保 chatbot 适应这些文化细微差别，你可以定义自定义评测指标，评估语气是否与每个目标受众的期望一致。

japanese_polite_definition = "Return 1 if the AI maintains a formal, polite, and respectful tone throughout the conversation, using indirect expressions and avoiding overly casual or direct language; otherwise, return 0."

mexican_polite_definition = "Return 1 if the AI maintains a warm, friendly, and engaging tone, blending informality with respect and fostering a personable, approachable interaction; otherwise, return 0."

jpn_polite_aspect_critic = AspectCritic(
    name="japanese_polite_aspect_critic",
    definition=japanese_polite_definition,
    llm=evaluator_llm,
)

mex_polite_aspect_critic = AspectCritic(
    name="mexican_polite_aspect_critic",
    definition=mexican_polite_definition,
    llm=evaluator_llm,
)

sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(content="I forgot my PIN. What should I do?"),
        AIMessage(
            content="I can assist you in resetting your PIN. May I send a verification code to your registered phone number?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(content="I have sent the code. Please check and enter it here."),
        HumanMessage(content="I have entered it."),
        AIMessage(content="Thank you. Please set a new PIN."),
        HumanMessage(content="I have set it."),
        AIMessage(
            content="Your PIN reset is complete. Thank you for choosing our bank."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(content="Help! I forgot my PIN."),
        AIMessage(
            content="Don't worry! I’ll help you reset it. Do you want me to send a verification code to your registered number?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(content="Done! Check your phone and enter the code here. "),
        HumanMessage(content="I have entered it."),
        AIMessage(content="Great! Now choose a new PIN and keep it in a safe place."),
        HumanMessage(content="Done."),
        AIMessage(content="Perfect! All set. If you need more help, I’ll be here."),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[jpn_polite_aspect_critic, mex_polite_aspect_critic],
)

result.to_pandas()

输出

Evaluating: 100%|██████████| 4/4 [00:00<?, ?it/s]

上述评测结果凸显了：在墨西哥被认为礼貌的，在日本可能不被感知为礼貌。

检查品牌语气

在本节中，我们将探索如何评测 chatbot 的语气是否与业务价值观、目标受众以及整体品牌身份一致。

什么是品牌语气（Brand Tone of Voice）？ 品牌语气指品牌在书面或口头互动中与受众沟通时的用词选择。通过定义独特的语气，品牌可以发展出真实的个性、风格与态度。参考

例如：

Google – 信息丰富且乐于助人的品牌声音 你是否注意到，使用 Google 产品时一切都感觉简单直观？但一旦切换到其他工具，事情突然变得更复杂。这种无缝体验来自 Google 对其品牌声音的掌握。

Google 保持友好、平易近人的语气，同时让用户沟通清晰简洁。他们的整体品牌声音围绕乐于助人、清晰和可及，让产品对每个人都直观。参考

你可以通过定义如下自定义评测指标，评估 chatbot 的回复是否与品牌身份一致。

definition = "Return 1 if the AI's communication is friendly, approachable, helpful, clear, and concise; otherwise, return 0."

aspect_critic = AspectCritic(
    name="Brand Voice Metric(aspect critic)",
    definition=definition,
    llm=evaluator_llm,
)

sample = MultiTurnSample(
    user_input=[
        HumanMessage(content="Hi! I want to apply for a credit card."),
        AIMessage(
            content="Sure! We’ve got a few great options. What kind of benefits are you interested in?"
        ),
        HumanMessage(content="Travel rewards, mostly."),
        AIMessage(
            content="Perfect! Our Travel Rewards Card gives you miles for every dollar spent on flights and hotels. Want to apply now?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(
            content="Awesome! Here’s the link to apply. It’s quick and easy. Let me know if you need help!"
        ),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample]),
    metrics=[aspect_critic],
)

result.to_pandas()

输出

Evaluating:   100%|██████████| 1/1 [00:00<?, ?it/s]