Optimizers API Reference

Ragas 提供优化器，通过自动化优化改进指标 prompt。本页记录可用的优化器类及其配置。

Overview

优化器使用带有真实分数的标注数据集来精炼指标 prompt，通过以下方式提升准确率：

• Instruction optimization：找到更好的 prompt 措辞
• Demonstration optimization：选择有效的 few-shot 示例
• Search strategies：高效探索 prompt 空间

Core Classes

Optimizer

Optimizer(metric: Optional[MetricWithLLM] = None, llm: Optional[BaseRagasLLM] = None)

基类：ABC

所有优化器的抽象基类。

optimize

optimize(dataset: SingleMetricAnnotation, loss: Loss, config: Dict[Any, Any], run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, callbacks: Optional[Callbacks] = None, with_debugging_logs=False, raise_exceptions: bool = True) -> Dict[str, str]

为给定指标优化 prompt。

参数：

返回：

源代码位于 src/ragas/optimizers/base.py

@abstractmethod
def optimize(
    self,
    dataset: SingleMetricAnnotation,
    loss: Loss,
    config: t.Dict[t.Any, t.Any],
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    callbacks: t.Optional[Callbacks] = None,
    with_debugging_logs=False,
    raise_exceptions: bool = True,
) -> t.Dict[str, str]:
    """
    Optimizes the prompts for the given metric.

    Parameters
    ----------
    metric : MetricWithLLM
        The metric to optimize.
    train_data : Any
        The training data.
    config : InstructionConfig
        The training configuration.

    Returns
    -------
    Dict[str, str]
        The optimized prompts for given chain.
    """
    raise NotImplementedError("The method `optimize` must be implemented.")

GeneticOptimizer

GeneticOptimizer(metric: Optional[MetricWithLLM] = None, llm: Optional[BaseRagasLLM] = None)

基类：Optimizer

一种在探索与利用之间取得平衡的遗传算法优化器。

DSPyOptimizer

DSPyOptimizer(metric: Optional[MetricWithLLM] = None, llm: Optional[BaseRagasLLM] = None, num_candidates: int = 10, max_bootstrapped_demos: int = 5, max_labeled_demos: int = 5, init_temperature: float = 1.0, auto: Optional[Literal['light', 'medium', 'heavy']] = 'light', num_threads: Optional[int] = None, max_errors: Optional[int] = None, seed: int = 9, verbose: bool = False, track_stats: bool = True, log_dir: Optional[str] = None, metric_threshold: Optional[float] = None, cache: Optional[CacheInterface] = None)

基类：Optimizer

使用 DSPy 的 MIPROv2 的高级 prompt 优化器。

MIPROv2 通过组合以下方式进行精细的 prompt 优化：

• Instruction optimization（prompt 工程）
• Demonstration optimization（few-shot 示例）
• 对两个空间的联合搜索

需要：pip install dspy-ai 或 uv add ragas[dspy]

参数：

optimize

optimize(dataset: SingleMetricAnnotation, loss: Loss, config: Dict[Any, Any], run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, callbacks: Optional[Callbacks] = None, with_debugging_logs: bool = False, raise_exceptions: bool = True) -> Dict[str, str]

使用 DSPy MIPROv2 优化指标 prompt。

步骤：

1. 将 Ragas PydanticPrompt 转换为 DSPy Signature
2. 用 signature 创建 DSPy Module
3. 将数据集转换为 DSPy Examples
4. 运行 MIPROv2 优化
5. 提取优化后的 prompt
6. 转换回 Ragas 格式

参数：

返回：

源代码位于 src/ragas/optimizers/dspy_optimizer.py

def optimize(
    self,
    dataset: SingleMetricAnnotation,
    loss: Loss,
    config: t.Dict[t.Any, t.Any],
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    callbacks: t.Optional[Callbacks] = None,
    with_debugging_logs: bool = False,
    raise_exceptions: bool = True,
) -> t.Dict[str, str]:
    """
    Optimize metric prompts using DSPy MIPROv2.

    Steps:

    1. Convert Ragas PydanticPrompt to DSPy Signature
    2. Create DSPy Module with signature
    3. Convert dataset to DSPy Examples
    4. Run MIPROv2 optimization
    5. Extract optimized prompts
    6. Convert back to Ragas format

    Parameters
    ----------
    dataset : SingleMetricAnnotation
        Annotated dataset with ground truth scores.
    loss : Loss
        Loss function to optimize.
    config : Dict[Any, Any]
        Additional configuration parameters.
    run_config : RunConfig, optional
        Runtime configuration.
    batch_size : int, optional
        Batch size for evaluation.
    callbacks : Callbacks, optional
        Langchain callbacks for tracking.
    with_debugging_logs : bool
        Enable debug logging.
    raise_exceptions : bool
        Whether to raise exceptions during optimization.

    Returns
    -------
    Dict[str, str]
        Optimized prompts for each prompt name.
    """
    if self.metric is None:
        raise ValueError("No metric provided for optimization.")

    if self.llm is None:
        raise ValueError("No llm provided for optimization.")

    if self._dspy is None:
        raise RuntimeError("DSPy module not loaded.")

    if self.cache is not None:
        cache_key = self._generate_cache_key(dataset, loss, config)
        if self.cache.has_key(cache_key):
            logger.info(
                f"Cache hit for DSPy optimization of metric: {self.metric.name}"
            )
            return self.cache.get(cache_key)

    logger.info(f"Starting DSPy optimization for metric: {self.metric.name}")

    from ragas.optimizers.dspy_adapter import (
        create_dspy_metric,
        pydantic_prompt_to_dspy_signature,
        ragas_dataset_to_dspy_examples,
        setup_dspy_llm,
    )

    setup_dspy_llm(self._dspy, self.llm)

    prompts = self.metric.get_prompts()
    optimized_prompts = {}

    for prompt_name, prompt in prompts.items():
        logger.info(f"Optimizing prompt: {prompt_name}")

        signature = pydantic_prompt_to_dspy_signature(prompt)
        module = self._dspy.Predict(signature)
        examples = ragas_dataset_to_dspy_examples(dataset, prompt_name)

        teleprompter = self._dspy.MIPROv2(
            num_candidates=self.num_candidates,
            max_bootstrapped_demos=self.max_bootstrapped_demos,
            max_labeled_demos=self.max_labeled_demos,
            init_temperature=self.init_temperature,
            auto=self.auto,
            num_threads=self.num_threads,
            max_errors=self.max_errors,
            seed=self.seed,
            verbose=self.verbose,
            track_stats=self.track_stats,
            log_dir=self.log_dir,
            metric_threshold=self.metric_threshold,
        )

        metric_fn = create_dspy_metric(loss, dataset.name)

        optimized = teleprompter.compile(
            module,
            trainset=examples,
            metric=metric_fn,
        )

        optimized_instruction = self._extract_instruction(optimized)
        optimized_prompts[prompt_name] = optimized_instruction

        logger.info(
            f"Optimized prompt for {prompt_name}: {optimized_instruction[:100]}..."
        )

    if self.cache is not None:
        cache_key = self._generate_cache_key(dataset, loss, config)
        self.cache.set(cache_key, optimized_prompts)
        logger.info("Cached optimization results")

    return optimized_prompts

GeneticOptimizer

面向 prompt 指令的简单进化优化器。

Parameters

Usage

from ragas.optimizers import GeneticOptimizer
from ragas.config import InstructionConfig

optimizer = GeneticOptimizer(
    max_steps=50,
    population_size=10,
)

config = InstructionConfig(llm=llm, optimizer=optimizer)
metric.optimize_prompts(dataset, config)

How it Works

1. 生成 prompt 变体种群
2. 在标注数据集上评估每一个
3. 选出表现最好的
4. 通过交叉和变异产生下一代
5. 重复 max_steps 次迭代

优点：简单，在数据有限时也能工作 缺点：收敛较慢，仅优化 instruction

DSPyOptimizer

使用 DSPy 的 MIPROv2 算法的高级优化器。

Parameters

Usage

from ragas.optimizers import DSPyOptimizer
from ragas.config import InstructionConfig

optimizer = DSPyOptimizer(
    num_candidates=10,
    max_bootstrapped_demos=5,
    max_labeled_demos=5,
)

config = InstructionConfig(llm=llm, optimizer=optimizer)
metric.optimize_prompts(dataset, config)

How it Works

1. 生成候选 prompt 指令
2. 从数据中 bootstrap few-shot demonstrations
3. 选择最佳人工标注示例
4. 在数据集上评估所有组合
5. 返回表现最好的配置

了解更多 DSPy 概念：

• Signatures - DSPy 定义输入/输出规范的方式
• Optimizers - 改进 prompt 和 LM 权重的算法
• Modules - LLM 程序的构建块

优点：效果更好，同时结合 instructions + demos 缺点：需要安装 DSPy，LLM 调用更多

Installation

DSPy 是可选依赖：

# Using uv (recommended)
uv add "ragas[dspy]"

# Using pip
pip install "ragas[dspy]"

Cost Estimation

每次优化的近似 LLM 调用次数：

Total calls ≈ num_candidates × 30 + max_bootstrapped_demos × 7

示例：

• 默认配置 (10, 5, 5)：约 335 次调用
• 节省配置 (5, 2, 3)：约 164 次调用
• 激进配置 (20, 10, 10)：约 670 次调用

Optimizer Base Class

基类：ABC

所有优化器的抽象基类。

optimize

optimize(dataset: SingleMetricAnnotation, loss: Loss, config: Dict[Any, Any], run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, callbacks: Optional[Callbacks] = None, with_debugging_logs=False, raise_exceptions: bool = True) -> Dict[str, str]

为给定指标优化 prompt。

参数：

返回：

Configuration

两种优化器都与 InstructionConfig 一起使用：

from ragas.config import InstructionConfig

config = InstructionConfig(
    llm=llm,                      # LLM for optimization
    optimizer=optimizer_instance, # Optimizer to use
)

# Use with metric
metric.optimize_prompts(dataset, config)

Dataset Format

优化器需要带有真实分数的标注数据集：

from ragas.dataset_schema import (
    PromptAnnotation,
    SampleAnnotation,
    SingleMetricAnnotation
)

# Create annotated sample
prompt_annotation = PromptAnnotation(
    prompt_input={"user_input": "...", "response": "..."},
    prompt_output={"score": 0.9},
    edited_output=None,  # Optional: corrected output
)

sample = SampleAnnotation(
    metric_input={"user_input": "...", "response": "..."},
    metric_output=0.9,  # Ground truth score
    prompts={"metric_prompt": prompt_annotation},
    is_accepted=True,  # Include in optimization
)

# Create dataset
dataset = SingleMetricAnnotation(
    name="metric_name",
    samples=[sample, ...]  # 20-50+ samples recommended
)

Loss Functions

优化器使用损失函数来评估 prompt 质量：

from ragas.losses import MSELoss, HuberLoss

# Mean Squared Error (default)
loss = MSELoss()

# Huber Loss (robust to outliers)
loss = HuberLoss(delta=1.0)

# Use with config
config = InstructionConfig(llm=llm, optimizer=optimizer, loss=loss)

Comparison

See Also

• DSPy Optimizer Guide - 详细用法
• Metric Customization - 创建指标
• Prompt API Reference - 理解 prompt

Additional Resources

DSPy 文档：

• DSPy Official Documentation - DSPy 完整指南
• MIPROv2 API Reference - MIPROv2 详细文档
• DSPy Optimizers Overview - 全部 DSPy 优化器指南
• DSPy GitHub Repository - 源代码与示例

研究论文：

• Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs - MIPROv2 论文