Ragas Ragas
stable · 中文译文
中文译文 · 原文：https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/ · 许可证 Apache-2.0

Faithfulness（忠实度）

Faithfulness 指标衡量 response 与 retrieved context 在事实上的一致程度。范围为 0 到 1，分数越高表示一致性越好。

如果回答中的所有 claim 都能被检索到的上下文支持，则认为该回答是 faithful 的。

计算方法：

  1. 识别回答中的所有 claim。
  2. 检查每条 claim 是否可以从检索到的上下文推断出来。
  3. 使用以下公式计算 faithfulness 分数：

[ \text{Faithfulness Score} = \frac{\text{Number of claims in the response supported by the retrieved context}}{\text{Total number of claims in the response}} ]

示例

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

# Create metric
scorer = Faithfulness(llm=llm)

# Evaluate
result = await scorer.ascore(
    user_input="When was the first super bowl?",
    response="The first superbowl was held on Jan 15, 1967",
    retrieved_contexts=[
        "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
    ]
)
print(f"Faithfulness Score: {result.value}")

输出：

Faithfulness Score: 1.0

同步用法

如果你更喜欢同步代码，可以使用 .score() 方法代替 .ascore()：

result = scorer.score(
    user_input="When was the first super bowl?",
    response="The first superbowl was held on Jan 15, 1967",
    retrieved_contexts=[...]
)

计算方式

示例

问题：Where and when was Einstein born?

上下文：Albert Einstein (born 14 March 1879) was a German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time

高忠实度答案：Einstein was born in Germany on 14th March 1879.

低忠实度答案：Einstein was born in Germany on 20th March 1879.

让我们看看使用低忠实度答案时 faithfulness 是如何计算的：

  • 步骤 1： 将生成的答案拆成单独的陈述。

  • 陈述：

    • 陈述 1: "Einstein was born in Germany."
    • 陈述 2: "Einstein was born on 20th March 1879."

  • 步骤 2： 对每个生成的陈述，验证它是否可以从给定上下文推断出来。

  • 陈述 1: Yes

  • 陈述 2: No

  • 步骤 3： 使用上面描述的公式计算 faithfulness。

[ \text{Faithfulness} = { \text{1} \over \text{2} } = 0.5 ]

旧版 Metrics API

以下示例使用旧版 metrics API 模式。对于新项目，我们建议使用上文所示的 collections-based API。

弃用时间表

此 API 将在 0.4 版本中弃用，并在 1.0 版本中移除。请迁移到上文所示的 collections-based API。

使用 SingleTurnSample 的示例

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import Faithfulness

sample = SingleTurnSample(
        user_input="When was the first super bowl?",
        response="The first superbowl was held on Jan 15, 1967",
        retrieved_contexts=[
            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ]
    )
scorer = Faithfulness(llm=evaluator_llm)
await scorer.single_turn_ascore(sample)

输出：

1.0

使用 HHEM-2.1-Open 的 Faithfulness

Vectara's HHEM-2.1-Open 是一个分类器模型（T5），经过训练用于检测 LLM 生成文本中的幻觉。该模型可用于计算 faithfulness 的第二步，即当 claim 与给定上下文交叉核验以判断是否可以从上下文推断时。该模型免费、体积小且开源，在生产用例中非常高效。

要使用该模型计算 faithfulness：

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import FaithfulnesswithHHEM


sample = SingleTurnSample(
        user_input="When was the first super bowl?",
        response="The first superbowl was held on Jan 15, 1967",
        retrieved_contexts=[
            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles."
        ]
    )
scorer = FaithfulnesswithHHEM(llm=evaluator_llm)
await scorer.single_turn_ascore(sample)

你可以通过设置 device 参数将模型加载到指定设备，并使用 batch_size 参数调整推理批次大小。默认情况下，模型加载在 CPU 上，批次大小为 10：

my_device = "cuda:0"
my_batch_size = 10

scorer = FaithfulnesswithHHEM(device=my_device, batch_size=my_batch_size)
await scorer.single_turn_ascore(sample)
← Response RelevancyAnswer Correctness →