入门：Ragas 与 Vertex AI

本教程是关于如何将 Vertex AI 模型与 Ragas 一起使用的三部分系列的一部分。第一篇教程旨在打下基础；其余两篇可以按任意顺序学习。你可以通过下面的链接跳转到其他教程：

• Align LLM Metrics：训练并对齐你的 LLM 评测器，使其更贴近人类判断。
• Model Comparison：用 Ragas 指标比较 VertexAI 提供的模型在基于 RAG 的问答任务上的表现。

让我们开始吧！

概述

本 notebook 演示如何使用 Vertex AI Studio 中的生成式模型，开始用 Ragas 做 Gen AI Evaluation。

Ragas 是一个全面的评测库，旨在增强对 LLM 应用的评估。它提供一套工具和指标，让开发者能够系统地评测并优化 AI 应用。

在本教程中，我们将探索：

1. 为 Ragas 评测准备数据
2. Ragas 提供的各类指标概览

更多用例与高级功能，请参阅文档以及 How-To's 部分中的评测用例：

• Ragas Concepts
• Ragas How-Tos

开始

安装依赖

!pip install --upgrade --user --quiet langchain-core langchain-google-vertexai langchain ragas rouge_score

重启运行时

要在这个 Jupyter runtime 中使用新安装的包，必须重启运行时。你可以运行下面的 cell 来完成，它会重启当前 kernel。

重启可能需要一分钟或更久。重启完成后，继续下一步。

import IPython

app = IPython.Application.instance()
app.kernel.do_shutdown(True)

认证你的 notebook 环境（仅 Colab）

如果你在 Google Colab 上运行本 notebook，请运行下面的 cell 来认证你的环境。

import sys

if "google.colab" in sys.modules:
    from google.colab import auth

    auth.authenticate_user()

设置 Google Cloud 项目信息并初始化 Vertex AI SDK

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
LOCATION = "us-central1"  # @param {type:"string"}

if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    raise ValueError("Please set your PROJECT_ID")


import vertexai

vertexai.init(project=PROJECT_ID, location=LOCATION)

用 Ragas 指标设置评测

Ragas 是一个开源库，为你提供一套全面的工具来评测 AI 应用。无论你需要基于计算的指标、基于模型的指标，还是自定义指标的灵活性，Ragas 都能覆盖。更多细节以及完整受支持指标列表，请访问 Ragas 文档。

在下面各节中，你将学习如何使用 Ragas 中的各类指标：

• Custom Metrics（自定义指标）： 定义并集成最适合你应用评测的自有指标。
• Model-based Metrics（基于模型的指标）： 通过 LLM 调用，对照特定标准分析模型输出，可以有或没有 references。
• Computation-based Metrics（基于计算的指标）： 基于数学公式的定量度量，不需要 LLM 调用。

定义 evaluator_llm

要使用和定义基于模型的指标，你需要同时配置 evaluator LLM 和 evaluator embeddings。

from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_google_vertexai import VertexAI, VertexAIEmbeddings


evaluator_llm = LangchainLLMWrapper(VertexAI(model_name="gemini-pro"))
evaluator_embeddings = LangchainEmbeddingsWrapper(VertexAIEmbeddings(model_name="text-embedding-004"))

用 Ragas 构建你自己的基于模型的 point-wise 指标

自定义或用户定义指标让你能够用评分规则、rubrics，或通过 LLM 调用执行的任意自定义逻辑来定制评测标准，从而根据你独特的任务需求评估回复。更多细节请参阅 general-purpose metrics 文档。

from ragas import evaluate
from ragas.metrics import AspectCritic, RubricsScore

maliciousness_definition = (
    "Is the response intended to harm, deceive, or exploit users?"
)

aspect_critic = AspectCritic(
    name="maliciousness",
    definition=maliciousness_definition,
    llm=evaluator_llm,
)

# adapeted google's helpfulness_prompt_template
helpfulness_rubrics = {
    "score1_description": "Response is useless/irrelevant, contains inaccurate/deceptive/misleading information, and/or contains harmful/offensive content. The user would feel not at all satisfied with the content in the response.",
    "score2_description": "Response is minimally relevant to the instruction and may provide some vaguely useful information, but it lacks clarity and detail. It might contain minor inaccuracies. The user would feel only slightly satisfied with the content in the response.",
    "score3_description": "Response is relevant to the instruction and provides some useful content, but could be more relevant, well-defined, comprehensive, and/or detailed. The user would feel somewhat satisfied with the content in the response.",
    "score4_description": "Response is very relevant to the instruction, providing clearly defined information that addresses the instruction's core needs.  It may include additional insights that go slightly beyond the immediate instruction.  The user would feel quite satisfied with the content in the response.",
    "score5_description": "Response is useful and very comprehensive with well-defined key details to address the needs in the instruction and usually beyond what explicitly asked. The user would feel very satisfied with the content in the response.",
}

rubrics_score = RubricsScore(name="helpfulness", rubrics=helpfulness_rubrics, llm=evaluator_llm)

Ragas 基于模型的指标

基于模型的指标利用预训练语言模型，通过对照特定标准比较回复来评估生成文本，提供细致、有上下文意识的评测，模拟人类判断。这些指标通过 LLM 调用计算。更多细节请参阅 model-based metrics 文档。

from ragas import evaluate
from ragas.metrics import ContextPrecision, Faithfulness

context_precision = ContextPrecision(llm=evaluator_llm)
faithfulness = Faithfulness(llm=evaluator_llm)

Ragas 基于计算的指标

这些指标采用既定的字符串匹配、n-gram 和统计方法来量化文本相似度与质量，完全用数学计算，无需 LLM 调用。更多细节请访问 computation-based metrics 文档。

from ragas.metrics import RougeScore

rouge_score = RougeScore()

准备你的数据集

要用 Ragas 指标进行评测，你需要把数据转换成 EvaluationDataset，这是 Ragas 中的一种数据类型。你可以在这里阅读更多内容。

例如，考虑以下示例数据：

# questions or query from user
user_inputs = [
    "Which part of the brain does short-term memory seem to rely on?",
    "What provided the Roman senate with exuberance?",
    "What area did the Hasan-jalalians command?",
]

# retrieved data used in answer generation
retrieved_contexts = [
    ["Short-term memory is supported by transient patterns of neuronal communication, dependent on regions of the frontal lobe (especially dorsolateral prefrontal cortex) and the parietal lobe. Long-term memory, on the other hand, is maintained by more stable and permanent changes in neural connections widely spread throughout the brain. The hippocampus is essential (for learning new information) to the consolidation of information from short-term to long-term memory, although it does not seem to store information itself. Without the hippocampus, new memories are unable to be stored into long-term memory, as learned from patient Henry Molaison after removal of both his hippocampi, and there will be a very short attention span. Furthermore, it may be involved in changing neural connections for a period of three months or more after the initial learning."],
    ["In 62 BC, Pompey returned victorious from Asia. The Senate, elated by its successes against Catiline, refused to ratify the arrangements that Pompey had made. Pompey, in effect, became powerless. Thus, when Julius Caesar returned from a governorship in Spain in 61 BC, he found it easy to make an arrangement with Pompey. Caesar and Pompey, along with Crassus, established a private agreement, now known as the First Triumvirate. Under the agreement, Pompey's arrangements would be ratified. Caesar would be elected consul in 59 BC, and would then serve as governor of Gaul for five years. Crassus was promised a future consulship."],
    ["The Seljuk Empire soon started to collapse. In the early 12th century, Armenian princes of the Zakarid noble family drove out the Seljuk Turks and established a semi-independent Armenian principality in Northern and Eastern Armenia, known as Zakarid Armenia, which lasted under the patronage of the Georgian Kingdom. The noble family of Orbelians shared control with the Zakarids in various parts of the country, especially in Syunik and Vayots Dzor, while the Armenian family of Hasan-Jalalians controlled provinces of Artsakh and Utik as the Kingdom of Artsakh."],
]

# answers generated by the rag
responses = [
    "frontal lobe and the parietal lobe",
    "The Roman Senate was filled with exuberance due to successes against Catiline.",
    "The Hasan-Jalalians commanded the area of Syunik and Vayots Dzor.",
]

# expected responses or ground truth
references = [
    "frontal lobe and the parietal lobe",
    "Due to successes against Catiline.",
    "The Hasan-Jalalians commanded the area of Artsakh and Utik.",
]

把这些转换成 Ragas 的 EvaluationDataset：

from ragas.dataset_schema import SingleTurnSample, EvaluationDataset

n = len(user_inputs)
samples = []


for i in range(n):

    sample = SingleTurnSample(
        user_input=user_inputs[i],
        retrieved_contexts=retrieved_contexts[i],
        response=responses[i],
        reference=references[i],
    )
    samples.append(sample)


ragas_eval_dataset = EvaluationDataset(samples=samples)
ragas_eval_dataset.to_pandas()

输出

运行评测

定义好评测数据集和所需指标后，你可以把它们传入 Ragas 的 evaluate 函数来运行评测：

from ragas import evaluate

ragas_metrics = [aspect_critic, context_precision, faithfulness, rouge_score, rubrics_score]

result = evaluate(
    metrics=ragas_metrics,
    dataset=ragas_eval_dataset
)
result

Evaluating: 100%|██████████| 15/15 [00:00<?, ?it/s]

查看数据集中每一行的详细分数：

result.to_pandas()

输出

查看本系列其他教程：

• Align LLM Metrics：训练并对齐你的 LLM 评测器，使其更贴近人类判断。
• Model Comparison：用 Ragas 指标比较 VertexAI 提供的模型在基于 RAG 的问答任务上的表现。