评测 LlamaIndex Agents

构建能智能使用工具并做出决策的 agents 只是旅程的一半；确保这些 agents 准确、可靠且表现良好，才真正定义它们的成功。LlamaIndex 提供多种创建 agents 的方式，包括 FunctionAgents、CodeActAgents 和 ReActAgents。在本教程中，我们将探索如何使用预构建的 Ragas 指标和自定义评测指标来评测这些不同类型的 agents。

让我们开始。

本教程分为三个全面的部分：

1. 用现成的 Ragas 指标评测 这里我们将考察两个基本评测工具：AgentGoalAccuracy，衡量 agent 识别并达成用户预期目标的效果；以及 Tool Call Accuracy，评估 agent 按正确顺序选择并调用合适工具以完成任务的能力。
2. 用于 CodeActAgent 评测的自定义指标 本节聚焦 LlamaIndex 的预构建 CodeActAgent，演示如何开发量身定制的评测指标，以满足代码生成 agents 的特定要求和能力。
3. Query Engine 工具评估 最后一节探索如何利用 Ragas RAG 指标评测 agents 中的 query engine 功能，在 agents 访问信息系统时提供关于检索有效性和响应质量的洞察。

Ragas Agentic 指标

为了演示使用 Ragas 指标的评测，我们将创建一个带单个 LlamaIndex Function Agent 的简单工作流，并用它覆盖基本功能。

点击查看 Function Agent 设置

from llama_index.llms.openai import OpenAI


async def send_message(to: str, content: str) -> str:
    """Dummy function to simulate sending an email."""
    return f"Successfully sent mail to {to}"

llm = OpenAI(model="gpt-4o-mini")

from llama_index.core.agent.workflow import FunctionAgent

agent = FunctionAgent(
    tools=[send_message],
    llm=llm,
    system_prompt="You are a helpful assistant of Jane",
)

Agent Goal Accuracy

AI agent 的真正价值在于它理解用户想要什么并有效交付的能力。Agent Goal Accuracy 作为基本指标，评估 agent 是否成功完成用户意图。这一度量至关重要，因为它直接反映 agent 解释用户需求并采取适当行动来满足它们的程度。

Ragas 提供该指标的两个关键变体：

• AgentGoalAccuracyWithReference - 二值评估（1 或 0），将 agent 的最终结果与预定义的期望结果进行比较。
• AgentGoalAccuracyWithoutReference - 二值评估（1 或 0），基于推断的意图而非预定义期望，评估 agent 是否达成用户目标。

With Reference 适合期望结果定义明确的场景，例如受控测试环境或针对 ground truth 数据的测试。

from llama_index.core.agent.workflow import (
    AgentInput,
    AgentOutput,
    AgentStream, 
    ToolCall as LlamaToolCall,
    ToolCallResult,
)

handler =  agent.run(user_msg="Send a message to jhon asking for a meeting")

events = []

async for ev in handler.stream_events():
    if isinstance(ev, (AgentInput, AgentOutput, LlamaToolCall, ToolCallResult)):
        events.append(ev)
    elif isinstance(ev, AgentStream):
        print(f"{ev.delta}", end="", flush=True)
    elif isinstance(ev, ToolCallResult):
        print(
            f"\nCall {ev.tool_name} with {ev.tool_kwargs}\nReturned: {ev.tool_output}"
        )

response = await handler

Output:

I have successfully sent a message to Jhon asking for a meeting.

from ragas.integrations.llama_index import convert_to_ragas_messages

ragas_messages = convert_to_ragas_messages(events)

from ragas.metrics import AgentGoalAccuracyWithoutReference
from ragas.llms import LlamaIndexLLMWrapper
from ragas.dataset_schema import MultiTurnSample
from ragas.messages import ToolCall as RagasToolCall

evaluator_llm = LlamaIndexLLMWrapper(llm=llm)

sample = MultiTurnSample(
    user_input=ragas_messages,
)

agent_goal_accuracy_without_reference = AgentGoalAccuracyWithoutReference(llm=evaluator_llm)
await agent_goal_accuracy_without_reference.multi_turn_ascore(sample)

Output:

1.0

from ragas.metrics import AgentGoalAccuracyWithReference

sample = MultiTurnSample(
    user_input=ragas_messages,
    reference="Successfully sent a message to Jhon asking for a meeting"
)


agent_goal_accuracy_with_reference = AgentGoalAccuracyWithReference(llm=evaluator_llm)
await agent_goal_accuracy_with_reference.multi_turn_ascore(sample)

Output:

1.0

Tool Call Accuracy

在 agentic 工作流中，AI agent 的有效性很大程度上取决于它在正确时间选择并使用正确工具的能力。Tool Call Accuracy 指标评估 agent 按正确顺序识别并调用合适工具以完成用户请求的精确程度。这一度量确保 agents 不仅理解有哪些工具可用，还理解如何有效编排它们以实现预期结果。

• ToolCallAccuracy 将 agent 的实际工具使用与期望 tool calls 的参考序列进行比较。如果 agent 的工具选择或顺序与参考不同，该指标返回分数 0，表示未能遵循完成任务的最优路径。

from ragas.metrics import ToolCallAccuracy

sample = MultiTurnSample(
    user_input=ragas_messages,
    reference_tool_calls=[
        RagasToolCall(
            name="send_message",
            args={'to': 'jhon', 'content': 'Hi Jhon,\n\nI hope this message finds you well. I would like to schedule a meeting to discuss some important matters. Please let me know your availability.\n\nBest regards,\nJane'},
        ),
    ],
)

tool_accuracy_scorer = ToolCallAccuracy()
await tool_accuracy_scorer.multi_turn_ascore(sample)

Output:

1.0

评测 LlamaIndex CodeAct Agents

LlamaIndex 提供一个预构建的 CodeAct Agent，可用于编写和执行代码，灵感来自原始 CodeAct 论文。想法是：不是输出一个简单的 JSON 对象，Code Agent 生成一个可执行代码块——通常用 Python 这样的高级语言。用代码而不是类似 JSON 的片段来编写动作，能提供更好的：

• Composability：代码自然允许函数的嵌套和复用；JSON actions 缺乏这种灵活性。
• Object management：代码优雅地处理操作输出（image = generate_image()）；JSON 没有干净的等价物。
• Generality：代码可以表达任何计算任务；JSON 施加了不必要的约束。
• Representation in LLM training data：LLM 已经从训练数据中理解代码，使其成为比专门 JSON 更自然的接口。

点击查看 CodeActAgent 设置

定义函数

from llama_index.llms.openai import OpenAI

# Configure the LLM
llm = OpenAI(model="gpt-4o-mini")


# Define a few helper functions
def add(a: int, b: int) -> int:
    """Add two numbers together"""
    return a + b


def subtract(a: int, b: int) -> int:
    """Subtract two numbers"""
    return a - b


def multiply(a: int, b: int) -> int:
    """Multiply two numbers"""
    return a * b


def divide(a: int, b: int) -> float:
    """Divide two numbers"""
    return a / b

创建 Code Executor

CodeActAgent 需要一个特定的 code_execute_fn 来执行 agent 生成的代码。

from typing import Any, Dict, Tuple
import io
import contextlib
import ast
import traceback


class SimpleCodeExecutor:
    """
    A simple code executor that runs Python code with state persistence.

    This executor maintains a global and local state between executions,
    allowing for variables to persist across multiple code runs.

    NOTE: not safe for production use! Use with caution.
    """

    def __init__(self, locals: Dict[str, Any], globals: Dict[str, Any]):
        """
        Initialize the code executor.

        Args:
            locals: Local variables to use in the execution context
            globals: Global variables to use in the execution context
        """
        # State that persists between executions
        self.globals = globals
        self.locals = locals

    def execute(self, code: str) -> Tuple[bool, str, Any]:
        """
        Execute Python code and capture output and return values.

        Args:
            code: Python code to execute

        Returns:
            Dict with keys `success`, `output`, and `return_value`
        """
        # Capture stdout and stderr
        stdout = io.StringIO()
        stderr = io.StringIO()

        output = ""
        return_value = None
        try:
            # Execute with captured output
            with contextlib.redirect_stdout(
                stdout
            ), contextlib.redirect_stderr(stderr):
                # Try to detect if there's a return value (last expression)
                try:
                    tree = ast.parse(code)
                    last_node = tree.body[-1] if tree.body else None

                    # If the last statement is an expression, capture its value
                    if isinstance(last_node, ast.Expr):
                        # Split code to add a return value assignment
                        last_line = code.rstrip().split("\n")[-1]
                        exec_code = (
                            code[: -len(last_line)]
                            + "\n__result__ = "
                            + last_line
                        )

                        # Execute modified code
                        exec(exec_code, self.globals, self.locals)
                        return_value = self.locals.get("__result__")
                    else:
                        # Normal execution
                        exec(code, self.globals, self.locals)
                except:
                    # If parsing fails, just execute the code as is
                    exec(code, self.globals, self.locals)

            # Get output
            output = stdout.getvalue()
            if stderr.getvalue():
                output += "\n" + stderr.getvalue()

        except Exception as e:
            # Capture exception information
            output = f"Error: {type(e).__name__}: {str(e)}\n"
            output += traceback.format_exc()

        if return_value is not None:
            output += "\n\n" + str(return_value)

        return output

code_executor = SimpleCodeExecutor(
    # give access to our functions defined above
    locals={
        "add": add,
        "subtract": subtract,
        "multiply": multiply,
        "divide": divide,
    },
    globals={
        # give access to all builtins
        "__builtins__": __builtins__,
        # give access to numpy
        "np": __import__("numpy"),
    },
)

设置 CodeAct Agent

from llama_index.core.agent.workflow import CodeActAgent
from llama_index.core.workflow import Context

agent = CodeActAgent(
    code_execute_fn=code_executor.execute,
    llm=llm,
    tools=[add, subtract, multiply, divide],
)

# context to hold the agent's session/state/chat history
ctx = Context(agent)

运行并评测 CodeAct agent

from llama_index.core.agent.workflow import (
    AgentInput,
    AgentOutput,
    AgentStream,
    ToolCall,
    ToolCallResult,
)

handler = agent.run("Calculate the sum of the first 10 fibonacci numbers", ctx=ctx)

events = []

async for event in handler.stream_events():
    if isinstance(event, (AgentInput, AgentOutput, ToolCall, ToolCallResult)):
        events.append(event)
    elif isinstance(event, AgentStream):
        print(f"{event.delta}", end="", flush=True)

The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. I will calculate their sum.

<execute>
def fibonacci(n):
    fib_sequence = [0, 1]
    for i in range(2, n):
        next_fib = fib_sequence[-1] + fib_sequence[-2]
        fib_sequence.append(next_fib)
    return fib_sequence

# Calculate the first 10 Fibonacci numbers
first_10_fib = fibonacci(10)

# Calculate the sum of the first 10 Fibonacci numbers
sum_fib = sum(first_10_fib)
print(sum_fib)
</execute>The sum of the first 10 Fibonacci numbers is 88.

提取 ToolCall

CodeAct_agent_tool_call = events[2]
agent_code = CodeAct_agent_tool_call.tool_kwargs["code"]

print(agent_code)

Output

    def fibonacci(n):
        fib_sequence = [0, 1]
        for i in range(2, n):
            next_fib = fib_sequence[-1] + fib_sequence[-2]
            fib_sequence.append(next_fib)
        return fib_sequence

    # Calculate the first 10 Fibonacci numbers
    first_10_fib = fibonacci(10)

    # Calculate the sum of the first 10 Fibonacci numbers
    sum_fib = sum(first_10_fib)
    print(sum_fib)

评估 CodeAct agents 时，可以从考察基本功能的基础指标开始，例如代码可编译性或合适的参数选择。这些直接的评测在进入更复杂的评估方法之前提供了坚实基础。

Ragas 提供强大的自定义指标能力，随着需求演变，可以实现越来越细致的评测。

• AspectCritic - 提供二值评测（通过/失败），判断 agent 的响应是否满足用户定义的特定标准，使用基于 LLM 的判断给出清晰的成功指示。
• RubricScoreMetric - 根据全面、预定义的质量评分细则和离散评分等级评估 agent 响应，从而在多个维度上一致地评估表现。

def is_compilable(code_str: str, mode="exec") -> bool:
    try:
        compile(code_str, "<string>", mode)
        return True
    except Exception:
        return False

is_compilable(agent_code)

Output

True

from ragas.metrics import AspectCritic
from ragas.dataset_schema import SingleTurnSample
from ragas.llms import LlamaIndexLLMWrapper

llm = OpenAI(model="gpt-4o-mini")
evaluator_llm = LlamaIndexLLMWrapper(llm=llm)

correct_tool_args = AspectCritic(
    name="correct_tool_args",
    llm=evaluator_llm,
    definition="Score 1 if the tool arguements use in the tool call are correct and 0 otherwise",
)

sample = SingleTurnSample(
    user_input="Calculate the sum of the first 10 fibonacci numbers",
    response=agent_code,
)

await correct_tool_args.single_turn_ascore(sample)

Output:

1

评测 Query Engine Tool

用 Ragas 指标评测时，需要确保数据格式适合评测。在 agentic 系统中使用 query engine 工具时，可以像评测任何 retrieval-augmented generation（RAG）系统一样处理。

我们将提取用户交互期间调用 query engine 工具的所有实例。利用这些，我们可以基于事件流数据构建 Ragas RAG 评测数据集。数据集就绪后，就可以应用全套 Ragas 评测指标。在本节中，我们将设置一个带 Query Engine Tools 的 Functional Agent。该 agent 可以访问两个 "tools"：一个查询 2021 Lyft 10-K，另一个查询 2021 Uber 10-K。

点击查看 Agent 设置

设置 LLMs

from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core import Settings

Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

构建 Query Engine Tools

from llama_index.core import StorageContext, load_index_from_storage

try:
    storage_context = StorageContext.from_defaults(
        persist_dir="./storage/lyft"
    )
    lyft_index = load_index_from_storage(storage_context)

    storage_context = StorageContext.from_defaults(
        persist_dir="./storage/uber"
    )
    uber_index = load_index_from_storage(storage_context)

    index_loaded = True
except:
    index_loaded = False

!mkdir -p 'data/10k/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O 'data/10k/lyft_2021.pdf'

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

if not index_loaded:
    # load data
    lyft_docs = SimpleDirectoryReader(
        input_files=["./data/10k/lyft_2021.pdf"]
    ).load_data()
    uber_docs = SimpleDirectoryReader(
        input_files=["./data/10k/uber_2021.pdf"]
    ).load_data()

    # build index
    lyft_index = VectorStoreIndex.from_documents(lyft_docs)
    uber_index = VectorStoreIndex.from_documents(uber_docs)

    # persist index
    lyft_index.storage_context.persist(persist_dir="./storage/lyft")
    uber_index.storage_context.persist(persist_dir="./storage/uber")

lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
uber_engine = uber_index.as_query_engine(similarity_top_k=3)

from llama_index.core.tools import QueryEngineTool

query_engine_tools = [
    QueryEngineTool.from_defaults(
        query_engine=lyft_engine,
        name="lyft_10k",
        description=(
            "Provides information about Lyft financials for year 2021. "
            "Use a detailed plain text question as input to the tool."
        ),
    ),
    QueryEngineTool.from_defaults(
        query_engine=uber_engine,
        name="uber_10k",
        description=(
            "Provides information about Uber financials for year 2021. "
            "Use a detailed plain text question as input to the tool."
        ),
    ),
]

Agent 设置

from llama_index.core.agent.workflow import FunctionAgent, ReActAgent
from llama_index.core.workflow import Context

agent = FunctionAgent(tools=query_engine_tools, llm=OpenAI(model="gpt-4o-mini"))

# context to hold the session/state
ctx = Context(agent)

运行并评测 Agents

from llama_index.core.agent.workflow import (
    AgentInput,
    AgentOutput,
    ToolCall,
    ToolCallResult,
    AgentStream, 
)

handler = agent.run("What's the revenue for Lyft in 2021 vs Uber?", ctx=ctx)

events = []

async for ev in handler.stream_events():
    if isinstance(ev, (AgentInput, AgentOutput, ToolCall, ToolCallResult)):
        events.append(ev)
    elif isinstance(ev, AgentStream):
        print(ev.delta, end="", flush=True)

response = await handler

Output:

In 2021, Lyft generated a total revenue of $3.21 billion, while Uber's total revenue was significantly higher at $17.455 billion.

我们将提取用户交互期间调用 query engine 工具的所有 ToolCallResult 实例，利用这些可以基于事件流数据构建合适的 RAG 评测数据集。

from ragas.dataset_schema import SingleTurnSample

ragas_samples = []

for event in events:
    if isinstance(event, ToolCallResult):
        if event.tool_name in ["lyft_10k", "uber_10k"]:
            sample = SingleTurnSample(
                user_input=event.tool_kwargs["input"],
                response=event.tool_output.content,
                retrieved_contexts=[node.text for node in event.tool_output.raw_output.source_nodes]
                )
            ragas_samples.append(sample)

from ragas.dataset_schema import EvaluationDataset

dataset = EvaluationDataset(samples=ragas_samples)
dataset.to_pandas()

Output:

得到的数据集默认不包含参考答案，因此我们只能使用不需要 references 的指标。不过，如果你希望运行基于参考的评测，可以向数据集添加 reference 列，然后应用相关的 Ragas 指标。

使用 Ragas RAG 指标评测

让我们评估 query engines 的有效性，特别是检索质量和幻觉预防。为完成这一评测，我们将使用两个关键 Ragas 指标：faithfulness 和 context relevance。更多内容可见 此处。

这种评测方法让我们能够识别可能影响整体系统表现的检索质量或响应生成问题。

• Faithfulness - 衡量生成响应在多大程度上忠实于检索上下文中呈现的事实，确保系统提出的主张可以直接被所提供信息支持。
• Context Relevance - 通过双重 LLM 判断机制，评估检索到的信息在多大程度上有效回应用户的特定查询。

from ragas import evaluate
from ragas.metrics import Faithfulness, ContextRelevance
from ragas.llms import LlamaIndexLLMWrapper
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o")
evaluator_llm = LlamaIndexLLMWrapper(llm=llm)

faithfulness = Faithfulness(llm=evaluator_llm)
context_precision = ContextRelevance(llm=evaluator_llm)

result = evaluate(dataset, metrics=[faithfulness, context_precision])

Evaluating: 100%|██████████| 4/4 [00:03<00:00,  1.19it/s]

result.to_pandas()

Output: