Prompt 评测 Quickstart

prompt_evals 模板通过情感分析评测并比较不同 prompt 变体。

创建项目

ragas quickstart prompt_evals
cd prompt_evals

安装依赖

uv sync

设置 API Key

export OPENAI_API_KEY="your-openai-key"

运行评测

uv run python evals.py

项目结构

prompt_evals/
├── README.md              # Project documentation
├── pyproject.toml         # Project configuration
├── prompt.py              # Prompt implementation
├── evals.py               # Evaluation workflow
├── __init__.py            # Python package marker
└── evals/
    ├── datasets/          # Test datasets
    ├── experiments/       # Evaluation results
    └── logs/              # Execution logs

评测内容

该模板评测情感分类 prompt 的有效性：

• 任务：情感分析（positive/negative）
• 测试用例：带有期望情感标签的电影评论
• 指标：二元准确率（pass/fail）

理解代码

Prompt（prompt.py）

实现情感分析 prompt：

from prompt import run_prompt

sentiment = run_prompt("I loved the movie! It was fantastic.")
# Returns: "positive" or "negative"

评测（evals.py）

测试 prompt 准确率：

@discrete_metric(name="accuracy", allowed_values=["pass", "fail"])
def my_metric(prediction: str, actual: str):
    return (
        MetricResult(value="pass", reason="")
        if prediction == actual
        else MetricResult(value="fail", reason="")
    )

测试数据

数据集包含电影评论：

dataset_dict = [
    {"text": "I loved the movie! It was fantastic.", "label": "positive"},
    {"text": "The movie was terrible and boring.", "label": "negative"},
    # More examples...
]

定制

测试不同 Prompt

修改 prompt.py 以测试变体：

# Version 1: Simple
prompt = f"Is this positive or negative: {text}"

# Version 2: With examples
prompt = f"""Classify sentiment:
Examples:
- "Great movie" -> positive
- "Boring film" -> negative

Text: {text}
Sentiment:"""

# Compare results across versions

添加更多指标

评测额外方面：

from ragas.metrics import NumericalMetric

confidence = NumericalMetric(
    name="confidence",
    prompt="Rate confidence 1-5 in this classification: {prediction}",
    allowed_values=(1, 5),
)

下一步

• Judge Alignment - 衡量 LLM-as-judge 对齐
• LLM 基准测试 - 比较不同模型