R2R 集成

R2R 是面向 AI Retrieval-Augmented Generation（RAG）的一站式方案，具备生产就绪特性，包括多模态内容摄入、混合搜索、用户/文档管理等。

概述

在本教程中，我们将：

• 利用 R2R 的 /rag endpoint，在一个小数据集上执行 Retrieval-Augmented Generation（RAG）。
• 评测生成的响应。
• 分析评测 traces。

R2R 设置

安装依赖

首先，安装必要的包：

%pip install r2r -q

配置本地环境

配置 R2R_API_KEY、OPENAI_API_KEY 和 RAGAS_APP_TOKEN（可选）。

from dotenv import load_dotenv

load_dotenv()

获取数据

dataset = [
    "OpenAI is one of the most recognized names in the large language model space, known for its GPT series of models. These models excel at generating human-like text and performing tasks like creative writing, answering questions, and summarizing content. GPT-4, their latest release, has set benchmarks in understanding context and delivering detailed responses.",
    "Anthropic is well-known for its Claude series of language models, designed with a strong focus on safety and ethical AI behavior. Claude is particularly praised for its ability to follow complex instructions and generate text that aligns closely with user intent.",
    "DeepMind, a division of Google, is recognized for its cutting-edge Gemini models, which are integrated into various Google products like Bard and Workspace tools. These models are renowned for their conversational abilities and their capacity to handle complex, multi-turn dialogues.",
    "Meta AI is best known for its LLaMA (Large Language Model Meta AI) series, which has been made open-source for researchers and developers. LLaMA models are praised for their ability to support innovation and experimentation due to their accessibility and strong performance.",
    "Meta AI with it's LLaMA models aims to democratize AI development by making high-quality models available for free, fostering collaboration across industries. Their open-source approach has been a game-changer for researchers without access to expensive resources.",
    "Microsoft’s Azure AI platform is famous for integrating OpenAI’s GPT models, enabling businesses to use these advanced models in a scalable and secure cloud environment. Azure AI powers applications like Copilot in Office 365, helping users draft emails, generate summaries, and more.",
    "Amazon’s Bedrock platform is recognized for providing access to various language models, including its own models and third-party ones like Anthropic’s Claude and AI21’s Jurassic. Bedrock is especially valued for its flexibility, allowing users to choose models based on their specific needs.",
    "Cohere is well-known for its language models tailored for business use, excelling in tasks like search, summarization, and customer support. Their models are recognized for being efficient, cost-effective, and easy to integrate into workflows.",
    "AI21 Labs is famous for its Jurassic series of language models, which are highly versatile and capable of handling tasks like content creation and code generation. The Jurassic models stand out for their natural language understanding and ability to generate detailed and coherent responses.",
    "In the rapidly advancing field of artificial intelligence, several companies have made significant contributions with their large language models. Notable players include OpenAI, known for its GPT Series (including GPT-4); Anthropic, which offers the Claude Series; Google DeepMind with its Gemini Models; Meta AI, recognized for its LLaMA Series; Microsoft Azure AI, which integrates OpenAI’s GPT Models; Amazon AWS (Bedrock), providing access to various models including Claude (Anthropic) and Jurassic (AI21 Labs); Cohere, which offers its own models tailored for business use; and AI21 Labs, known for its Jurassic Series. These companies are shaping the landscape of AI by providing powerful models with diverse capabilities.",
]

设置 R2R Client

from r2r import R2RClient

client = R2RClient()

摄入数据

ingest_response = client.documents.create(
    chunks=dataset,
)

使用 /rag Endpoint

/rag endpoint 通过将搜索结果与语言模型输出结合，实现 Retrieval-Augmented Generation。生成过程可用 rag_generation_config 参数自定义，检索过程可用 search_settings 配置。

query = "What makes Meta AI’s LLaMA models stand out?"

search_settings = {
        "limit": 2,
        "graph_settings": {"enabled": False, "limit": 2},
    }

response = client.retrieval.rag(
    query=query,
    search_settings=search_settings
)

print(response.results.generated_answer)

Output

Meta AI’s LLaMA models stand out due to their open-source nature, which supports innovation and experimentation by making high-quality models accessible to researchers and developers [1]. This approach democratizes AI development, fostering collaboration across industries and enabling researchers without access to expensive resources to work with advanced AI models [2].

评测

用 Ragas 评测 R2R Client

有了 R2R Client，我们就可以使用 Ragas 的 r2r 集成进行评测。该过程包含以下关键部分：

• 1. R2R Client 与配置 指定 RAG 设置的 R2RClient 和 /rag 配置。
• 2. 评测数据集 你需要一个包含 Ragas 指标所需全部输入的 Ragas EvaluationDataset。
• 3. Ragas 指标 Ragas 提供多种评测指标，用于评估 RAG 的不同方面，例如 faithfulness、answer relevance 和 context recall。完整可用指标列表见 Ragas 文档。

构建 Ragas EvaluationDataset

EvaluationDataset 是 Ragas 中用于表示评测样本的数据类型。关于其结构和用法的更多细节见 核心概念部分。

我们将使用 ragas 的 transform_to_ragas_dataset 函数，为我们的数据得到 EvaluationDataset。

questions = [
    "Who are the major players in the large language model space?",
    "What is Microsoft’s Azure AI platform known for?",
    "What kind of models does Cohere provide?",
]

references = [
    "The major players include OpenAI (GPT Series), Anthropic (Claude Series), Google DeepMind (Gemini Models), Meta AI (LLaMA Series), Microsoft Azure AI (integrating GPT Models), Amazon AWS (Bedrock with Claude and Jurassic), Cohere (business-focused models), and AI21 Labs (Jurassic Series).",
    "Microsoft’s Azure AI platform is known for integrating OpenAI’s GPT models, enabling businesses to use these models in a scalable and secure cloud environment.",
    "Cohere provides language models tailored for business use, excelling in tasks like search, summarization, and customer support.",
]

r2r_responses = []

search_settings = {
    "limit": 2,
    "graph_settings": {"enabled": False, "limit": 2},
}

for que in questions:
    response = client.retrieval.rag(query=que, search_settings=search_settings)
    r2r_responses.append(response)

from ragas.integrations.r2r import transform_to_ragas_dataset

ragas_eval_dataset = transform_to_ragas_dataset(
    user_inputs=questions, r2r_responses=r2r_responses, references=references
)

Output

EvaluationDataset(features=['user_input', 'retrieved_contexts', 'response', 'reference'], len=3)

选择指标

为评测 RAG endpoint，我们将使用以下指标：

• Response Relevancy：衡量响应对用户输入（查询）的相关程度。
• Context Precision：衡量相关文档（或信息片段）被成功检索的比例。
• Faithfulness：衡量响应与检索到的上下文在事实上的一致性。

from ragas.metrics import AnswerRelevancy, ContextPrecision, Faithfulness
from ragas import evaluate
from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper

llm = ChatOpenAI(model="gpt-4o-mini")
evaluator_llm = LangchainLLMWrapper(llm)

ragas_metrics = [AnswerRelevancy(llm=evaluator_llm), ContextPrecision(llm=evaluator_llm), Faithfulness(llm=evaluator_llm)]

results = evaluate(dataset=ragas_eval_dataset, metrics=ragas_metrics)

Output

Querying Client: 100%|██████████| 3/3 [00:00<?, ?it/s]

Evaluating: 100%|██████████| 9/9 [00:00<?, ?it/s]

追踪评测

为了更好地理解评测分数，我们可以用下面的代码获取 traces 以及 verdict 的理由。

results.upload()

Happy Coding