RAG 评测 Quickstart

rag_eval 模板提供完整的 RAG 评测设置，包含自定义指标、数据集管理与实验跟踪。

创建项目

# Using uvx (no installation required)
uvx ragas quickstart rag_eval
cd rag_eval

# Or with ragas installed
ragas quickstart rag_eval
cd rag_eval

安装依赖

uv sync

或使用 pip：

pip install -e .

设置 API Key

export OPENAI_API_KEY="your-openai-key"

export ANTHROPIC_API_KEY="your-anthropic-key"

更新 evals.py：

from anthropic import Anthropic
from ragas.llms import llm_factory

client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))
llm = llm_factory("claude-3-5-sonnet-20241022", provider="anthropic", client=client)

export GOOGLE_API_KEY="your-google-api-key"

更新 evals.py：

import google.generativeai as genai
from ragas.llms import llm_factory

genai.configure(api_key=os.environ.get("GOOGLE_API_KEY"))
client = genai.GenerativeModel("gemini-2.0-flash")
llm = llm_factory("gemini-2.0-flash", provider="google", client=client)

from openai import OpenAI
from ragas.llms import llm_factory

client = OpenAI(
    api_key="ollama",
    base_url="http://localhost:11434/v1"
)
llm = llm_factory("mistral", provider="openai", client=client)

运行评测

uv run python evals.py

评测将会：

从 load_dataset() 函数加载测试数据
用测试问题查询你的 RAG 应用
使用自定义指标评测回复
在控制台显示结果
将结果保存为 evals/experiments/ 中的 CSV

项目结构

rag_eval/
├── README.md              # Project documentation
├── pyproject.toml         # Project configuration
├── rag.py                 # RAG application implementation
├── evals.py               # Evaluation workflow
├── __init__.py            # Python package marker
└── evals/
    ├── datasets/          # Test data files
    ├── experiments/       # Evaluation results (CSV)
    └── logs/              # Execution logs and traces

理解代码

RAG 应用（`rag.py`）

一个简单的 RAG 实现，包含：

文档存储：内存中的文档集合
关键词检索：用简单关键词匹配检索文档
回复生成：用 OpenAI API 生成答案
Tracing：记录每次查询以便调试

from rag import default_rag_client

# Initialize with OpenAI client
rag_client = default_rag_client(llm_client=openai_client, logdir="evals/logs")

# Query the RAG system
response = rag_client.query("What is Ragas?")
print(response["answer"])

评测脚本（`evals.py`）

评测工作流：

数据集加载：创建带有问题和评分说明的测试用例
指标定义：用于 pass/fail 评测的自定义 DiscreteMetric
实验执行：运行查询并评测回复
结果存储：保存为 CSV 以便分析

from ragas import Dataset, experiment
from ragas.metrics import DiscreteMetric

# Define your metric
my_metric = DiscreteMetric(
    name="correctness",
    prompt="Check if the response contains points from grading notes...",
    allowed_values=["pass", "fail"],
)

# Run experiment
@experiment()
async def run_experiment(row):
    response = rag_client.query(row["question"])
    score = my_metric.score(llm=llm, response=response["answer"], ...)
    return {**row, "response": response["answer"], "score": score.value}

定制

添加测试用例

编辑 evals.py 中的 load_dataset() 函数：

def load_dataset():
    dataset = Dataset(
        name="test_dataset",
        backend="local/csv",
        root_dir="evals",
    )

    data_samples = [
        {
            "question": "What is Ragas?",
            "grading_notes": "- evaluation framework - LLM applications",
        },
        {
            "question": "How do experiments work?",
            "grading_notes": "- track results - compare runs - store metrics",
        },
        # Add more test cases...
    ]

    for sample in data_samples:
        dataset.append(sample)
    dataset.save()
    return dataset

修改指标

通过更新指标 prompt 来改变评测标准：

my_metric = DiscreteMetric(
    name="quality",
    prompt="""Evaluate the response quality:

Response: {response}
Expected Points: {grading_notes}

Rate as:
- 'excellent': All points covered with clear explanation
- 'good': Most points covered
- 'poor': Missing key points

Rating:""",
    allowed_values=["excellent", "good", "poor"],
)

添加多个指标

为不同评测方面创建额外指标：

from ragas.metrics import DiscreteMetric, NumericalMetric

correctness = DiscreteMetric(
    name="correctness",
    prompt="Is the response factually correct? {response}",
    allowed_values=["correct", "incorrect"],
)

relevance = NumericalMetric(
    name="relevance",
    prompt="Rate relevance 1-5: {response} for question: {question}",
    allowed_values=(1, 5),
)

使用你自己的 RAG 系统

用生产系统替换示例 RAG：

# In evals.py
from your_rag_module import YourRAGClient

rag_client = YourRAGClient(...)

@experiment()
async def run_experiment(row):
    # Call your RAG system
    response = await rag_client.query(row["question"])

    score = my_metric.score(
        llm=llm,
        response=response,
        grading_notes=row["grading_notes"],
    )

    return {
        **row,
        "response": response,
        "score": score.value,
    }

查看结果

结果保存为 evals/experiments/ 中的 CSV 文件。每次实验运行都会创建新文件，包含：

输入数据（问题、评分说明）
模型回复
评测分数
时间戳

import pandas as pd

# Load results
results = pd.read_csv("evals/experiments/your_experiment.csv")

# Calculate pass rate
pass_rate = (results["score"] == "pass").mean()
print(f"Pass rate: {pass_rate:.1%}")

下一步

改进 RAG 指南 - 对比 naive 与 agentic RAG
自定义指标 - 编写你自己的指标
数据集 - 了解数据集管理
实验 - 进阶实验跟踪