Ragas Ragas
stable · 中文译文
中文译文 · 原文：https://docs.ragas.io/en/stable/howtos/integrations/_langchain/ · 许可证 Apache-2.0

Langchain

评测 Langchain QA Chains

LangChain 是用于开发由语言模型驱动的应用的框架。它也可用于创建 RAG 系统（在 langchain 中也称为 QA 系统）。如果你想了解如何用 langchain 创建 RAG 系统，可以查看 文档。

借助该集成，你可以轻松用 ragas 提供的指标评测 QA chains

#!pip install ragas langchain_openai python-dotenv

# attach to the existing event loop when using jupyter notebooks
import os

import nest_asyncio
import openai
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()
# IMPORTANT: Remember to create a .env variable containing: OPENAI_API_KEY=sk-xyz where xyz is your key

# Access the API key from the environment variable
api_key = os.environ.get("OPENAI_API_KEY")

# Initialize the OpenAI API client
openai.api_key = api_key

nest_asyncio.apply()

首先加载数据集。我们将在 NYC wikipedia 页面 上构建一个通用 QA 系统。加载数据集并从中创建 VectorstoreIndex 和 RetrievalQA。

from langchain.chains import RetrievalQA
from langchain.indexes import VectorstoreIndexCreator
from langchain_community.document_loaders import TextLoader
from langchain_openai import ChatOpenAI

loader = TextLoader("./nyc_wikipedia/nyc_text.txt")
index = VectorstoreIndexCreator().from_loaders([loader])


llm = ChatOpenAI(temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm,
    retriever=index.vectorstore.as_retriever(),
    return_source_documents=True,
)

/home/jjmachan/.pyenv/versions/ragas/lib/python3.10/site-packages/langchain/indexes/vectorstore.py:128: UserWarning: Using InMemoryVectorStore as the default vectorstore.This memory store won't persist data. You should explicitlyspecify a vectorstore when using VectorstoreIndexCreator
  warnings.warn(



---------------------------------------------------------------------------

ValidationError                           Traceback (most recent call last)

Cell In[2], line 7
      4 from langchain_openai import ChatOpenAI
      6 loader = TextLoader("./nyc_wikipedia/nyc_text.txt")
----> 7 index = VectorstoreIndexCreator().from_loaders([loader])
     10 llm = ChatOpenAI(temperature=0)
     11 qa_chain = RetrievalQA.from_chain_type(
     12     llm,
     13     retriever=index.vectorstore.as_retriever(),
     14     return_source_documents=True,
     15 )


File ~/.pyenv/versions/ragas/lib/python3.10/site-packages/pydantic/main.py:212, in BaseModel.__init__(self, **data)
    210 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks
    211 __tracebackhide__ = True
--> 212 validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
    213 if self is not validated_self:
    214     warnings.warn(
    215         'A custom validator is returning a value other than `self`.\n'
    216         "Returning anything other than `self` from a top level model validator isn't supported when validating via `__init__`.\n"
    217         'See the `model_validator` docs (https://docs.pydantic.dev/latest/concepts/validators/#model-validators) for more details.',
    218         category=None,
    219     )


ValidationError: 1 validation error for VectorstoreIndexCreator
embedding
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing

# testing it out

question = "How did New York City get its name?"
result = qa_chain({"query": question})
result["result"]

为了评测 QA 系统，我们生成了几个相关问题。我们已经为你生成了一些问题，也可以随意添加你想要的问题。

eval_questions = [
    "What is the population of New York City as of 2020?",
    "Which borough of New York City has the highest population?",
    "What is the economic significance of New York City?",
    "How did New York City get its name?",
    "What is the significance of the Statue of Liberty in New York City?",
]

eval_answers = [
    "8,804,190",
    "Brooklyn",
    "New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.",
    "New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.",
    "The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.",
]

examples = [
    {"query": q, "ground_truth": [eval_answers[i]]}
    for i, q in enumerate(eval_questions)
]

介绍 RagasEvaluatorChain

RagasEvaluatorChain 为 ragas 提供的指标（文档见 此处）创建一层包装，从而更容易与 langchain 和 langsmith 一起运行这些评测。

evaluator chain 有以下 API

  • __call__()：直接在 QA chain 的结果上调用 RagasEvaluatorChain。
  • evaluate()：在一组 examples（带输入查询）和 predictions（QA chain 的输出）上评测。
  • evaluate_run()：由 langsmith evaluators 调用、用于评测 langsmith 数据集的方法。

让我们逐个看看它们的实际用法。

result = qa_chain({"query": eval_questions[1]})
result["result"]

result = qa_chain(examples[4])
result["result"]

from ragas.langchain.evalchain import RagasEvaluatorChain
from ragas.metrics import (
    answer_relevancy,
    context_precision,
    context_recall,
    faithfulness,
)

# create evaluation chains
faithfulness_chain = RagasEvaluatorChain(metric=faithfulness)
answer_rel_chain = RagasEvaluatorChain(metric=answer_relevancy)
context_rel_chain = RagasEvaluatorChain(metric=context_precision)
context_recall_chain = RagasEvaluatorChain(metric=context_recall)
  1. __call__()

直接用 QA chain 的结果运行评测 chain。注意，像 context_precision 和 faithfulness 这样的指标需要存在 source_documents。

# Recheck the result that we are going to validate.
result

Faithfulness

eval_result = faithfulness_chain(result)
eval_result["faithfulness_score"]

高 faithfulness_score 意味着源文档与答案之间存在精确一致性。

你可以通过把 result（LLM 的答案）或 source_documents 改成别的内容，来查看更低的 faithfulness 分数。

fake_result = result.copy()
fake_result["result"] = "we are the champions"
eval_result = faithfulness_chain(fake_result)
eval_result["faithfulness_score"]

Context Recall

eval_result = context_recall_chain(result)
eval_result["context_recall_score"]

高 context_recall_score 意味着 ground truth 出现在源文档中。

你可以通过把 source_documents 改成别的内容，来查看更低的 context recall 分数。

from langchain.schema import Document

fake_result = result.copy()
fake_result["source_documents"] = [Document(page_content="I love christmas")]
eval_result = context_recall_chain(fake_result)
eval_result["context_recall_score"]
  1. evaluate()

评测一组 inputs/queries 以及 QA chain 的 outputs/predictions。

# run the queries as a batch for efficiency
predictions = qa_chain.batch(examples)

# evaluate
print("evaluating...")
r = faithfulness_chain.evaluate(examples, predictions)
r

# evaluate context recall
print("evaluating...")
r = context_recall_chain.evaluate(examples, predictions)
r

用 langsmith 评测

Langsmith 是一个帮助调试、测试、评测和监控基于任意 LLM 框架构建的 chains 和 agents 的平台。它也与 LangChain 无缝集成。

Langsmith 还提供构建测试数据集并针对它们运行评测的工具，借助 RagasEvaluatorChain，你也可以用 ragas 指标运行 langsmith 评测。要了解更多关于 langsmith 评测的内容，请查看 quickstart。

让我们从用 eval_questions 中列出的 NYC 问题创建数据集开始。创建一个新的 langsmith 数据集并上传这些问题。

# dataset creation

from langsmith import Client
from langsmith.utils import LangSmithError

client = Client()
dataset_name = "NYC test"

try:
    # check if dataset exists
    dataset = client.read_dataset(dataset_name=dataset_name)
    print("using existing dataset: ", dataset.name)
except LangSmithError:
    # if not create a new one with the generated query examples
    dataset = client.create_dataset(
        dataset_name=dataset_name, description="NYC test dataset"
    )
    for e in examples:
        client.create_example(
            inputs={"query": e["query"]},
            outputs={"ground_truth": e["ground_truth"]},
            dataset_id=dataset.id,
        )

    print("Created a new dataset: ", dataset.name)

如你所见，问题已经上传。现在你可以针对这个测试数据集运行 QA chain，并在 langchain 平台上比较结果。

在调用 run_on_dataset 之前，你需要一个工厂函数，用于创建你要测试的 QA chain 的新实例。这样在针对每个 example 运行时就不会复用内部状态。

# factory function that return a new qa chain
def create_qa_chain(return_context=True):
    qa_chain = RetrievalQA.from_chain_type(
        llm,
        retriever=index.vectorstore.as_retriever(),
        return_source_documents=return_context,
    )
    return qa_chain

现在让我们运行评测

from langchain.smith import RunEvalConfig, run_on_dataset

evaluation_config = RunEvalConfig(
    custom_evaluators=[
        faithfulness_chain,
        answer_rel_chain,
        context_rel_chain,
        context_recall_chain,
    ],
    prediction_key="result",
)

result = run_on_dataset(
    client,
    dataset_name,
    create_qa_chain,
    evaluation=evaluation_config,
    input_mapper=lambda x: x,
)

你可以跟随链接在 langsmith 中打开这次运行的结果。也请查看每个 example 的分数

如果你想更深入了解分数原因以及如何改进，点击任意 example 并打开 feedback 标签页。这里会显示各项分数。

你也可以查看对应的 RagasEvaluatorChain trace，弄清 ragas 为何给出这样的分数。

← LangChainLangGraph →