修改指标中的 prompts

Ragas 中每一个使用 LLM 的指标，也会使用一个或多个 prompts 来生成中间结果，这些中间结果再被用来形成分数。在使用基于 LLM 的指标时，可以把 prompts 当作超参数来对待。一个适合你的领域和用例的优化 prompt，可以把基于 LLM 的指标准确率提高 10-20%。由于最优 prompts 取决于所用的 LLM，你可能想要调整驱动每个指标的 prompts。

快速开始：如果你需要一个简单的自定义指标，可以考虑使用 DiscreteMetric 或 NumericMetric，它们直接接受自定义 prompts。示例参见 Discrete Metrics。

本指南介绍如何修改现有 collection metrics（如 Faithfulness、FactualCorrectness）中的 prompts，这些指标使用 BasePrompt 类。继续之前，请确保你理解 Prompt Object 文档。

理解你的指标中的 prompts

对于支持 prompt 定制的指标，Ragas 通过指标实例提供对底层 prompt 对象的访问。来看如何访问 Faithfulness 指标中的 prompts：

from ragas.metrics.collections import Faithfulness
from openai import AsyncOpenAI
from ragas.llms import llm_factory

# Setup dependencies
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

# Create metric instance
scorer = Faithfulness(llm=llm)

# Faithfulness has two prompts:
# 1. statement_generator_prompt - breaks response into atomic statements
# 2. nli_statement_prompt - evaluates each statement against context
print(scorer.statement_generator_prompt)
print(scorer.nli_statement_prompt)

生成并查看 prompt 字符串

来查看将发送给 LLM 的 prompt：

from ragas.metrics.collections.faithfulness.util import StatementGeneratorInput

# Create sample input
sample_input = StatementGeneratorInput(
    question="What is the Eiffel Tower?",
    answer="The Eiffel Tower is located in Paris."
)

# Generate the prompt string
prompt_string = scorer.statement_generator_prompt.to_string(sample_input)
print(prompt_string)

修改 prompts

Ragas 中的现代指标使用模块化的 BasePrompt 类。要定制 prompt：

1. 访问 prompt：prompt 作为指标实例上的属性可用
2. 修改 prompt 类：扩展或子类化 prompt，以定制 instruction 或 examples
3. 更新指标：将你的自定义 prompt 赋给指标的属性

示例：定制 FactualCorrectness prompt

FactualCorrectness 内部使用两个 prompts：

• prompt - ClaimDecompositionPrompt，用于将文本拆成 claims
• nli_prompt - NLIStatementPrompt，用于对照 context 验证 claims

你可以定制其中一个或两个：

from ragas.metrics.collections import FactualCorrectness
from ragas.metrics.collections.factual_correctness.util import (
    ClaimDecompositionPrompt,
    NLIStatementPrompt,
)

# Create a custom claim decomposition prompt by subclassing
class CustomClaimDecompositionPrompt(ClaimDecompositionPrompt):
    instruction = """You are an expert at breaking down complex statements into atomic claims.
Break down the input text into clear, verifiable claims.
Only output valid JSON with a "claims" array."""

# Optionally customize the NLI prompt too
class CustomNLIPrompt(NLIStatementPrompt):
    instruction = """Carefully evaluate if each statement is supported by the context.
Be strict in your verification - only mark as supported if directly stated."""

# Create metric instance and replace prompts
scorer = FactualCorrectness(llm=llm)
scorer.prompt = CustomClaimDecompositionPrompt()
scorer.nli_prompt = CustomNLIPrompt()

# Now the metric will use the custom prompts
result = await scorer.ascore(
    response="The Eiffel Tower is in Paris and was built in 1889.",
    reference="The Eiffel Tower is located in Paris. It was completed in 1889."
)

示例：定制 Faithfulness 的 examples

Few-shot examples 可以极大地影响 LLM 输出。下面是如何修改它们：

from ragas.metrics.collections import Faithfulness
from ragas.metrics.collections.faithfulness.util import (
    NLIStatementInput,
    NLIStatementOutput,
    NLIStatementPrompt,
    StatementFaithfulnessAnswer,
)

# Create custom prompt with domain-specific examples
class DomainSpecificNLIPrompt(NLIStatementPrompt):
    examples = [
        (
            NLIStatementInput(
                context="Machine learning is a field within artificial intelligence that enables systems to learn from data.",
                statements=[
                    "Machine learning is a subset of AI.",
                    "Machine learning uses statistical techniques.",
                ],
            ),
            NLIStatementOutput(
                statements=[
                    StatementFaithfulnessAnswer(
                        statement="Machine learning is a subset of AI.",
                        reason="The context states ML is 'a field within artificial intelligence', supporting this claim.",
                        verdict=1
                    ),
                    StatementFaithfulnessAnswer(
                        statement="Machine learning uses statistical techniques.",
                        reason="The context doesn't mention statistical techniques.",
                        verdict=0
                    ),
                ]
            ),
        ),
    ]

# Update the metric with custom prompt
scorer = Faithfulness(llm=llm)
scorer.nli_statement_prompt = DomainSpecificNLIPrompt()

# Now evaluate with domain-specific prompts
result = await scorer.ascore(
    user_input="How do neural networks work?",
    response="Neural networks are inspired by biological neurons.",
    retrieved_contexts=["Artificial neural networks are computing systems loosely inspired by biological neural networks."]
)

将 prompts 适配到不同语言

你可以使用 adapt 方法将 prompts 适配到不同语言：

from ragas.metrics.collections import Faithfulness

scorer = Faithfulness(llm=llm)

# Adapt the statement generator prompt to Spanish
adapted_prompt = await scorer.statement_generator_prompt.adapt(
    target_language="spanish",
    llm=llm,
    adapt_instruction=False  # Keep instruction in English, only translate examples
)

# Replace the prompt with the adapted version
scorer.statement_generator_prompt = adapted_prompt

# Now use the metric with Spanish examples
result = await scorer.ascore(
    user_input="¿Dónde nació Einstein?",
    response="Einstein nació en Alemania.",
    retrieved_contexts=["Albert Einstein nació en Alemania..."]
)

验证你的定制

下面是如何验证你的 prompt 定制是否生效：

from ragas.metrics.collections.faithfulness.util import NLIStatementInput

# Create sample input to test the prompt
sample_input = NLIStatementInput(
    context="Paris is the capital and most populous city of France.",
    statements=["The capital of France is Paris.", "Paris is in Germany."]
)

# Generate and view the full prompt string
full_prompt = scorer.nli_statement_prompt.to_string(sample_input)
print("Full Prompt:")
print(full_prompt)