评测

evaluate()

使用不同指标对数据集执行评测

参数：

名称	类型	说明	默认值
`dataset`	`(Dataset, EvaluationDataset)`	指标用来评测 RAG pipeline 的数据集。	required
`metrics`	`list[Metric]`	用于评测的指标列表。若未提供，ragas 会在一组最佳指标上运行评测，以给出完整视图。	`None`
`llm`	`BaseRagasLLM`	用于生成分数以计算指标的语言模型（LLM）。若未提供，ragas 会对需要 LLM 的指标使用默认语言模型。可被指标级别通过 metric.llm 指定的 LLM 覆盖。	`None`
`embeddings`	`BaseRagasEmbeddings`	用于指标的嵌入模型。若未提供，ragas 会对需要嵌入的指标使用默认嵌入。可被指标级别通过 metric.embeddings 指定的嵌入覆盖。	`None`
`experiment_name`	`str`	要追踪的实验名称。用于在 tracing 工具中追踪本次评测。	`None`
`callbacks`	`Callbacks`	评测期间运行的 Langchain 生命周期 Callbacks。更多信息见 Langchain 文档。	`None`
`run_config`	`RunConfig`	超时与重试等运行时设置。若未提供，使用默认值。	`None`
`token_usage_parser`	`TokenUsageParser`	从 LLM 结果中获取 token 用量的解析器。若未提供，将不计算成本和总 token 数。默认为 None。	`None`
`raise_exceptions`	`False`	是否抛出异常。若为 True，任一指标失败时评测会抛出异常。若为 False，失败的行会返回 np.nan。默认为 False。	`False`
`column_map`	`dict[str, str]`	用于评测的数据集列名。若数据集列名与默认不同，可在此以字典提供映射。例如：若数据集列名为 contexts_v1，可传入 column_map 为 {"contexts": "contexts_v1"}。	`None`
`show_progress`	`bool`	评测期间是否显示进度条。若为 False，则禁用进度条。默认为 True。	`True`
`batch_size`	`int`	批次应有多大。若设为 None（默认），则不分批。	`None`
`return_executor`	`bool`	若为 True，返回 Executor 实例而不运行评测。返回的 executor 可通过调用 executor.cancel() 取消执行。要获取结果，调用 executor.results()。默认为 False。	`False`
`allow_nest_asyncio`	`bool`	是否允许为 Jupyter 兼容性打上 nest_asyncio 补丁。在生产异步应用中设为 False，以避免事件循环冲突。默认为 True。	`True`

类型	说明
`EvaluationResult or Executor`	若 return_executor 为 False，返回包含各指标分数的 EvaluationResult 对象。若 return_executor 为 True，返回可用于取消执行的 Executor 实例。

抛出：

类型	说明
`ValueError`	若校验失败：指标所需列缺失，或列格式不正确。

示例：

基本用法如下：

from ragas import evaluate

>>> dataset
Dataset({
    features: ['question', 'ground_truth', 'answer', 'contexts'],
    num_rows: 30
})

>>> result = evaluate(dataset)
>>> print(result)
{'context_precision': 0.817,
'faithfulness': 0.892,
'answer_relevancy': 0.874}

源代码位于 src/ragas/evaluation.py

@track_was_completed
def evaluate(
    dataset: t.Union[Dataset, EvaluationDataset],
    metrics: t.Optional[t.Sequence[Metric]] = None,
    llm: t.Optional[BaseRagasLLM | LangchainLLM] = None,
    embeddings: t.Optional[
        BaseRagasEmbeddings | BaseRagasEmbedding | LangchainEmbeddings
    ] = None,
    experiment_name: t.Optional[str] = None,
    callbacks: Callbacks = None,
    run_config: t.Optional[RunConfig] = None,
    token_usage_parser: t.Optional[TokenUsageParser] = None,
    raise_exceptions: bool = False,
    column_map: t.Optional[t.Dict[str, str]] = None,
    show_progress: bool = True,
    batch_size: t.Optional[int] = None,
    _run_id: t.Optional[UUID] = None,
    _pbar: t.Optional[tqdm] = None,
    return_executor: bool = False,
    allow_nest_asyncio: bool = True,
) -> t.Union[EvaluationResult, Executor]:
    """
    Perform the evaluation on the dataset with different metrics

    Parameters
    ----------
    dataset : Dataset, EvaluationDataset
        The dataset used by the metrics to evaluate the RAG pipeline.
    metrics : list[Metric], optional
        List of metrics to use for evaluation. If not provided, ragas will run
        the evaluation on the best set of metrics to give a complete view.
    llm : BaseRagasLLM, optional
        The language model (LLM) to use to generate the score for calculating the metrics.
        If not provided, ragas will use the default
        language model for metrics that require an LLM. This can be overridden by the LLM
        specified in the metric level with `metric.llm`.
    embeddings : BaseRagasEmbeddings, optional
        The embeddings model to use for the metrics.
        If not provided, ragas will use the default embeddings for metrics that require embeddings.
        This can be overridden by the embeddings specified in the metric level with `metric.embeddings`.
    experiment_name : str, optional
        The name of the experiment to track. This is used to track the evaluation in the tracing tool.
    callbacks : Callbacks, optional
        Lifecycle Langchain Callbacks to run during evaluation.
        Check the [Langchain documentation](https://python.langchain.com/docs/modules/callbacks/) for more information.
    run_config : RunConfig, optional
        Configuration for runtime settings like timeout and retries. If not provided, default values are used.
    token_usage_parser : TokenUsageParser, optional
        Parser to get the token usage from the LLM result.
        If not provided, the cost and total token count will not be calculated. Default is None.
    raise_exceptions : False
        Whether to raise exceptions or not. If set to True, the evaluation will raise an exception
        if any of the metrics fail. If set to False, the evaluation will return `np.nan` for the row that failed. Default is False.
    column_map : dict[str, str], optional
        The column names of the dataset to use for evaluation. If the column names of the dataset are different from the default ones,
        it is possible to provide the mapping as a dictionary here. Example: If the dataset column name is `contexts_v1`, it is possible to pass column_map as `{"contexts": "contexts_v1"}`.
    show_progress : bool, optional
        Whether to show the progress bar during evaluation. If set to False, the progress bar will be disabled. The default is True.
    batch_size : int, optional
        How large the batches should be. If set to None (default), no batching is done.
    return_executor : bool, optional
        If True, returns the Executor instance instead of running evaluation.
        The returned executor can be used to cancel execution by calling executor.cancel().
        To get results, call executor.results(). Default is False.
    allow_nest_asyncio : bool, optional
        Whether to allow nest_asyncio patching for Jupyter compatibility.
        Set to False in production async applications to avoid event loop conflicts. Default is True.

    Returns
    -------
    EvaluationResult or Executor
        If return_executor is False, returns EvaluationResult object containing the scores of each metric.
        If return_executor is True, returns the Executor instance for cancellable execution.

    Raises
    ------
    ValueError
        if validation fails because the columns required for the metrics are missing or
        if the columns are of the wrong format.

    Examples
    --------
    the basic usage is as follows:
    ```
    from ragas import evaluate

    >>> dataset
    Dataset({
        features: ['question', 'ground_truth', 'answer', 'contexts'],
        num_rows: 30
    })

    >>> result = evaluate(dataset)
    >>> print(result)
    {'context_precision': 0.817,
    'faithfulness': 0.892,
    'answer_relevancy': 0.874}
    ```
    """
    warnings.warn(
        "evaluate() is deprecated and will be removed in a future version. "
        "Use the @experiment decorator instead. "
        "See https://docs.ragas.io/en/latest/concepts/experiment/ for more information.",
        DeprecationWarning,
        stacklevel=2,
    )

    # Create async wrapper for aevaluate
    async def _async_wrapper():
        return await aevaluate(
            dataset=dataset,
            metrics=metrics,
            llm=llm,
            embeddings=embeddings,
            experiment_name=experiment_name,
            callbacks=callbacks,
            run_config=run_config,
            token_usage_parser=token_usage_parser,
            raise_exceptions=raise_exceptions,
            column_map=column_map,
            show_progress=show_progress,
            batch_size=batch_size,
            _run_id=_run_id,
            _pbar=_pbar,
            return_executor=return_executor,
        )

    if not allow_nest_asyncio:
        # Run without nest_asyncio - creates a new event loop
        import asyncio

        return asyncio.run(_async_wrapper())
    else:
        # Default behavior: use nest_asyncio for backward compatibility (Jupyter notebooks)
        from ragas.async_utils import run

        return run(_async_wrapper())