Noise Sensitivity（噪声敏感度）

NoiseSensitivity 衡量系统在利用相关或不相关检索文档时，因给出不正确回答而出错的频率。分数范围为 0 到 1，值越低表示性能越好。噪声敏感度使用 user_input、reference、response 以及 retrieved_contexts 计算。

为了估计噪声敏感度，会检查生成回答中的每条 claim，判断它相对于 ground truth 是否正确，以及是否可以归因于相关（或不相关）的检索上下文。理想情况下，答案中的所有 claim 都应由相关检索上下文支持。

[ \text{noise sensitivity (relevant)} = {|\text{Total number of incorrect claims in response}| \over |\text{Total number of claims in the response}|} ]

示例

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import NoiseSensitivity

# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

# Create metric
scorer = NoiseSensitivity(llm=llm)

# Evaluate
result = await scorer.ascore(
    user_input="What is the Life Insurance Corporation of India (LIC) known for?",
    response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
    reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
    retrieved_contexts=[
        "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
        "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
        "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
        "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
    ]
)
print(f"Noise Sensitivity Score: {result.value}")

输出：

Noise Sensitivity Score: 0.3333333333333333

要计算不相关上下文的噪声敏感度，可以将 mode 参数设为 irrelevant：

scorer = NoiseSensitivity(llm=llm, mode="irrelevant")
result = await scorer.ascore(
    user_input="What is the Life Insurance Corporation of India (LIC) known for?",
    response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
    reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
    retrieved_contexts=[
        "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
        "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
        "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
        "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
    ]
)
print(f"Noise Sensitivity (Irrelevant) Score: {result.value}")

输出：

Noise Sensitivity (Irrelevant) Score: 0.0

同步用法

如果你更喜欢同步代码，可以使用 .score() 方法代替 .ascore()：

result = scorer.score(
    user_input="What is the Life Insurance Corporation of India (LIC) known for?",
    response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India...",
    reference="The Life Insurance Corporation of India (LIC) is the largest insurance company...",
    retrieved_contexts=[...]
)

计算方式

示例

问题：What is the Life Insurance Corporation of India (LIC) known for?

Ground truth：The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.

旧版 Metrics API

以下示例使用旧版 metrics API 模式。对于新项目，我们建议使用上文所示的 collections-based API。

弃用时间表

此 API 将在 0.4 版本中弃用，并在 1.0 版本中移除。请迁移到上文所示的 collections-based API。

使用 SingleTurnSample 的示例

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import NoiseSensitivity

sample = SingleTurnSample(
    user_input="What is the Life Insurance Corporation of India (LIC) known for?",
    response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.",
    reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.",
    retrieved_contexts=[
        "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.",
        "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.",
        "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.",
        "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc."
    ]
)

scorer = NoiseSensitivity(llm=evaluator_llm)
await scorer.single_turn_ascore(sample)

输出：

0.3333333333333333

要计算不相关上下文的噪声敏感度，可以将 mode 参数设为 irrelevant：

scorer = NoiseSensitivity(mode="irrelevant")
await scorer.single_turn_ascore(sample)

致谢：噪声敏感度由 RAGChecker 引入