Ragas Ragas
stable · 中文译文
中文译文 · 原文：https://docs.ragas.io/en/stable/howtos/integrations/_langfuse/ · 许可证 Apache-2.0

Langfuse

Ragas 与 Langfuse 的组合很强大，可以帮助你评测和监控 Retrieval-Augmented Generation（RAG）流水线。

什么是 Langfuse？

Langfuse（GitHub）是一个开源平台，用于 LLM 追踪、prompt 管理 和 评测。它允许你为 traces 和 spans 打分，从而洞察 RAG 流水线的表现。Langfuse 支持多种集成，包括 OpenAI、LangChain 以及 更多。

将 Langfuse 与 Ragas 一起使用的关键好处

  • 为 Traces 打分：为 traces 和 spans 打分，洞察 RAG 流水线的表现。
  • 详细分析：对 traces 进行分段和 分析，识别低质量分数并改进系统表现。
  • 分数报告：深入特定用例和用户分群的详细报告。

Ragas（GitHub）是一个开源工具，可以帮助你对 traces/spans 运行 基于模型的评测，尤其适合 RAG 流水线。Ragas 可以对 RAG 流水线的多个方面进行无需参考的评测。因为它是 reference-free 的，运行评测时不需要 ground-truths，可以在你用 Langfuse 收集的生产 traces 上运行。

入门

本指南将带你走完一个用 Ragas 和 Langfuse 做 RAG 评测的端到端示例。

环境

在 Langfuse 注册 以获取 API keys。

import os

# get keys for your project from https://cloud.langfuse.com
os.environ["LANGFUSE_SECRET_KEY"] = "sk-..."
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-..."

# your openai key
# os.environ["OPENAI_API_KEY"] = "sk-..."

%pip install datasets ragas llama_index python-dotenv --upgrade

数据

本例中，我们将使用一份已经通过查询 RAG 系统并收集其输出而准备好的数据集。如何从 Langfuse 获取生产数据的说明见下文。

该数据集包含以下列：

  • question：list[str] - 这些是 RAG 流水线将要被评测的问题。
  • answer：list[str] - RAG 流水线生成并提供给用户的答案。
  • contexts：list[list[str]] - 传入 LLM 以回答问题的上下文。
  • ground_truth：list[list[str]] - 问题的 ground truth 答案。不过在线评测可以忽略它，因为我们在这种情况下无法获得 ground-truth 数据。
from datasets import load_dataset

amnesty_qa = load_dataset("vibrantlabsai/amnesty_qa", "english_v2")["eval"]
amnesty_qa

Found cached dataset amnesty_qa (/home/jjmachan/.cache/huggingface/datasets/vibrantlabs___amnesty_qa/english_v2/2.0.0/d0ed9800191a31943ee52a5c22ee4305e28a33f5edcd9a323802112cff07cc24)


  0%|          | 0/1 [00:00<?, ?it/s]




Dataset({
    features: ['question', 'ground_truth', 'answer', 'contexts'],
    num_rows: 20
})

指标

本例中，我们将使用 Ragas 库中的以下指标：

  • faithfulness：衡量生成答案相对于给定上下文的事实一致性。
  • answer_relevancy：Answer Relevancy 关注评估生成答案对给定 prompt 的针对性和相关性。
  • context precision：Context Precision 评估上下文中所有与 ground-truth 相关的条目是否被排在更高位置。理想情况下，所有相关 chunks 都应出现在靠前的排名。该指标使用问题和上下文计算，取值范围为 0 到 1，分数越高表示精度越好。
  • aspect_critique：用于根据预定义方面（如 harmlessness 和 correctness）评估提交内容。此外，用户可以灵活定义自己的方面，按特定标准评测提交内容。

请查看 文档 了解这些指标及其工作原理。

# import metrics
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from ragas.metrics.critique import SUPPORTED_ASPECTS, harmfulness

# metrics you chose
metrics = [faithfulness, answer_relevancy, context_precision, harmfulness]

接下来，用你选择的 LLM 和 Embeddings 初始化指标。本例中我们使用 OpenAI。

from ragas.run_config import RunConfig
from ragas.metrics.base import MetricWithLLM, MetricWithEmbeddings


# util function to init Ragas Metrics
def init_ragas_metrics(metrics, llm, embedding):
    for metric in metrics:
        if isinstance(metric, MetricWithLLM):
            metric.llm = llm
        if isinstance(metric, MetricWithEmbeddings):
            metric.embeddings = embedding
        run_config = RunConfig()
        metric.init(run_config)

from langchain_openai.chat_models import ChatOpenAI
from langchain_openai.embeddings import OpenAIEmbeddings

# wrappers
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

llm = ChatOpenAI()
emb = OpenAIEmbeddings()

init_ragas_metrics(
    metrics,
    llm=LangchainLLMWrapper(llm),
    embedding=LangchainEmbeddingsWrapper(emb),
)

设置

你可以用两种方式使用 Ragas 做基于模型的评测：

  1. 为每条 Trace 打分：这意味着你会对每个 trace 项运行评测。这能让你更清楚地了解每次 RAG 流水线调用的表现，但可能较昂贵
  2. 批量打分：这种方法会定期随机抽取 traces 样本并打分。这能降低成本并给出应用表现的粗略估计，但可能漏掉重要样本。

在本 cookbook 中，我们将展示如何设置这两种方式。

为 Trace 打分

让我们以单条 trace 的小例子，看看如何用 Ragas 打分。首先加载数据。

row = amnesty_qa[0]
print("question: ", row["question"])
print("answer: ", row["answer"])

question:  What are the global implications of the USA Supreme Court ruling on abortion?
answer:  The global implications of the USA Supreme Court ruling on abortion can be significant, as it sets a precedent for other countries and influences the global discourse on reproductive rights. Here are some potential implications:

1. Influence on other countries: The Supreme Court's ruling can serve as a reference point for other countries grappling with their own abortion laws. It can provide legal arguments and reasoning that advocates for reproductive rights can use to challenge restrictive abortion laws in their respective jurisdictions.

2. Strengthening of global reproductive rights movements: A favorable ruling by the Supreme Court can energize and empower reproductive rights movements worldwide. It can serve as a rallying point for activists and organizations advocating for women's rights, leading to increased mobilization and advocacy efforts globally.

3. Counteracting anti-abortion movements: Conversely, a ruling that restricts abortion rights can embolden anti-abortion movements globally. It can provide legitimacy to their arguments and encourage similar restrictive measures in other countries, potentially leading to a rollback of existing reproductive rights.

4. Impact on international aid and policies: The Supreme Court's ruling can influence international aid and policies related to reproductive health. It can shape the priorities and funding decisions of donor countries and organizations, potentially leading to increased support for reproductive rights initiatives or conversely, restrictions on funding for abortion-related services.

5. Shaping international human rights standards: The ruling can contribute to the development of international human rights standards regarding reproductive rights. It can influence the interpretation and application of existing human rights treaties and conventions, potentially strengthening the recognition of reproductive rights as fundamental human rights globally.

6. Global health implications: The Supreme Court's ruling can have implications for global health outcomes, particularly in countries with restrictive abortion laws. It can impact the availability and accessibility of safe and legal abortion services, potentially leading to an increase in unsafe abortions and related health complications.

It is important to note that the specific implications will depend on the nature of the Supreme Court ruling and the subsequent actions taken by governments, activists, and organizations both within and outside the United States.

现在初始化 Langfuse client SDK 以为应用埋点。

from langfuse import Langfuse

langfuse = Langfuse()

这里我们定义一个工具函数，用你选择的指标为 trace 打分。

async def score_with_ragas(query, chunks, answer):
    scores = {}
    for m in metrics:
        print(f"calculating {m.name}")
        scores[m.name] = await m.ascore(
            row={"question": query, "contexts": chunks, "answer": answer}
        )
    return scores

question, contexts, answer = row["question"], row["contexts"], row["answer"]
await score_with_ragas(question, contexts, answer)

calculating faithfulness
calculating answer_relevancy


Using 'context_precision' without ground truth will be soon depreciated. Use 'context_utilization' instead


calculating context_precision
calculating harmfulness





{'faithfulness': 0.0,
 'answer_relevancy': 0.9999999999999996,
 'context_precision': 0.9999999999,
 'harmfulness': 0}

你对每个请求计算分数。下面我们概述了一个 dummy 应用，执行以下步骤：

  1. 从用户获取问题
  2. 从数据库或向量存储获取可用于回答用户问题的上下文
  3. 把问题和上下文传给 LLM 生成答案

所有这些步骤都作为 spans 记录在 Langfuse 的单条 trace 中。关于 traces 和 spans 的更多内容，见 Langfuse 文档。

# the logic of the dummy application is
# given a question fetch the correspoinding contexts and answers from a dict

import hashlib


def hash_string(input_string):
    return hashlib.sha256(input_string.encode()).hexdigest()


q_to_c = {}  # map between question and context
q_to_a = {}  # map between question and answer
for row in amnesty_qa:
    q_hash = hash_string(row["question"])
    q_to_c[q_hash] = row["contexts"]
    q_to_a[q_hash] = row["answer"]

# if your running this in a notebook - please run this cell
# to manage asyncio event loops
import nest_asyncio

nest_asyncio.apply()

from langfuse.decorators import observe, langfuse_context
from asyncio import run


@observe()
def retriver(question: str):
    return q_to_c[question]


@observe()
def generator(question):
    return q_to_a[question]


@observe()
def rag_pipeline(question):
    q_hash = hash_string(question)
    contexts = retriver(q_hash)
    generated_answer = generator(q_hash)

    # score the runs
    score = run(score_with_ragas(question, contexts, answer=generated_answer))
    for s in score:
        langfuse_context.score_current_trace(name=s, value=score[s])
    return generated_answer

question, contexts, answer = row["question"], row["contexts"], row["answer"]
generated_answer = rag_pipeline(amnesty_qa[0]["question"])

calculating faithfulness
calculating answer_relevancy


Using 'context_precision' without ground truth will be soon depreciated. Use 'context_utilization' instead


calculating context_precision
calculating harmfulness

在 Langfuse 中分析分数

你可以在 Langfuse UI 中 分析 分数，并按问题或用户下钻。

→ 还没用过 Langfuse？在我们的 交互式 demo 中探索 dashboard。

注意打分是阻塞的，因此请确保在等待分数计算完成之前已经发送了生成的答案。或者你可以在单独的线程中运行 score_with_ragas()，并传入 trace_id 来记录分数。

资源

  • 查看我们关于 基于模型的评测 的指南，了解如何用 Ragas 运行基于模型的评测。
  • 在 这里 了解更多关于分析和改进 LLM 应用的内容。

反馈

如有任何反馈或请求，请创建 GitHub Issue，或在 Discord 上与社区分享你的工作。

← SwarmAthina AI →