评测一个简单的 RAG 系统
本指南旨在说明用 ragas 测试并评测 RAG 系统的简单工作流。假定你对构建 RAG 系统与评测仅有最少了解。安装 ragas 请参阅我们的安装说明。
基本设置
我们将使用 langchain_openai 来设置构建简单 RAG 所需的 LLM 和嵌入模型。你也可以选择任何其他 LLM 和嵌入模型，做法请参阅 LangChain 中自定义模型。
from langchain_openai import ChatOpenAI
from ragas.embeddings import OpenAIEmbeddings
import openai
llm = ChatOpenAI(model="gpt-4o")
openai_client = openai.OpenAI()
embeddings = OpenAIEmbeddings(client=openai_client)
OpenAI Embeddings API
ragas.embeddings.OpenAIEmbeddings 暴露的是 embed_text（单条）和 embed_texts（批量），而不是某些 LangChain 包装器那样的 embed_query/embed_documents。下面的示例对文档使用 embed_texts，对查询使用 embed_text。请参阅 OpenAI embeddings 实现
构建一个简单的 RAG 系统
要构建一个简单的 RAG 系统，需要定义以下组件：
- 定义一个把文档向量化的方法
- 定义一个检索相关文档的方法
- 定义一个生成响应的方法
点击查看代码
import numpy as np
class RAG:
def __init__(self, model="gpt-4o"):
import openai
self.llm = ChatOpenAI(model=model)
openai_client = openai.OpenAI()
self.embeddings = OpenAIEmbeddings(client=openai_client)
self.doc_embeddings = None
self.docs = None
def load_documents(self, documents):
"""Load documents and compute their embeddings."""
self.docs = documents
self.doc_embeddings = self.embeddings.embed_texts(documents)
def get_most_relevant_docs(self, query):
"""Find the most relevant document for a given query."""
if not self.docs or not self.doc_embeddings:
raise ValueError("Documents and their embeddings are not loaded.")
query_embedding = self.embeddings.embed_text(query)
similarities = [
np.dot(query_embedding, doc_emb)
/ (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb))
for doc_emb in self.doc_embeddings
]
most_relevant_doc_index = np.argmax(similarities)
return [self.docs[most_relevant_doc_index]]
def generate_answer(self, query, relevant_doc):
"""Generate an answer for a given query based on the most relevant document."""
prompt = f"question: {query}\n\nDocuments: {relevant_doc}"
messages = [
("system", "You are a helpful assistant that answers questions based on given documents only."),
("human", prompt),
]
ai_msg = self.llm.invoke(messages)
return ai_msg.content
加载文档
现在，让我们加载一些文档并测试我们的 RAG 系统。
sample_docs = [
"Albert Einstein proposed the theory of relativity, which transformed our understanding of time, space, and gravity.",
"Marie Curie was a physicist and chemist who conducted pioneering research on radioactivity and won two Nobel Prizes.",
"Isaac Newton formulated the laws of motion and universal gravitation, laying the foundation for classical mechanics.",
"Charles Darwin introduced the theory of evolution by natural selection in his book 'On the Origin of Species'.",
"Ada Lovelace is regarded as the first computer programmer for her work on Charles Babbage's early mechanical computer, the Analytical Engine."
]
# Initialize RAG instance
rag = RAG()
# Load documents
rag.load_documents(sample_docs)
# Query and retrieve the most relevant document
query = "Who introduced the theory of relativity?"
relevant_doc = rag.get_most_relevant_docs(query)
# Generate an answer
answer = rag.generate_answer(query, relevant_doc)
print(f"Query: {query}")
print(f"Relevant Document: {relevant_doc}")
print(f"Answer: {answer}")
输出：
Query: Who introduced the theory of relativity?
Relevant Document: ['Albert Einstein proposed the theory of relativity, which transformed our understanding of time, space, and gravity.']
Answer: Albert Einstein introduced the theory of relativity.
收集评测数据
要收集评测数据，我们首先需要一组针对 RAG 运行的查询。我们可以把查询送入 RAG 系统，并为每个查询收集 response、retrieved_contexts。你也可以选择性地为每个查询准备一组黄金答案，用来评测系统表现。
sample_queries = [
"Who introduced the theory of relativity?",
"Who was the first computer programmer?",
"What did Isaac Newton contribute to science?",
"Who won two Nobel Prizes for research on radioactivity?",
"What is the theory of evolution by natural selection?"
]
expected_responses = [
"Albert Einstein proposed the theory of relativity, which transformed our understanding of time, space, and gravity.",
"Ada Lovelace is regarded as the first computer programmer for her work on Charles Babbage's early mechanical computer, the Analytical Engine.",
"Isaac Newton formulated the laws of motion and universal gravitation, laying the foundation for classical mechanics.",
"Marie Curie was a physicist and chemist who conducted pioneering research on radioactivity and won two Nobel Prizes.",
"Charles Darwin introduced the theory of evolution by natural selection in his book 'On the Origin of Species'."
]
dataset = []
for query,reference in zip(sample_queries,expected_responses):
relevant_docs = rag.get_most_relevant_docs(query)
response = rag.generate_answer(query, relevant_docs)
dataset.append(
{
"user_input":query,
"retrieved_contexts":relevant_docs,
"response":response,
"reference":reference
}
)
现在，把数据集加载到 EvaluationDataset 对象中。
from ragas import EvaluationDataset
evaluation_dataset = EvaluationDataset.from_list(dataset)
评测
我们已经成功收集了评测数据。现在，可以用一组常用的 RAG 评测指标，在收集到的数据集上评测我们的 RAG 系统。你可以选择任何模型作为评测用的 evaluator LLM。
from ragas import evaluate
from ragas.llms import LangchainLLMWrapper
evaluator_llm = LangchainLLMWrapper(llm)
from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness
result = evaluate(dataset=evaluation_dataset,metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()],llm=evaluator_llm)
result
输出
{'context_recall': 1.0000, 'faithfulness': 0.8571, 'factual_correctness': 0.7280}
想借助评测改进你的 AI 应用？
过去 2 年里，我们见过并用评测帮助改进了许多 AI 应用。
我们正在把这些知识压缩进一个产品，用评测循环取代凭感觉检查，让你能专注于打造出色的 AI 应用。
如果你想借助评测改进并规模化你的 AI 应用。
🔗 预约一个时段，或写信给我们：founders@vibrantlabs.com。