Judge Alignment Quickstart

judge_alignment 模板衡量 LLM-as-judge 与人工评测标准的对齐程度。

创建项目

ragas quickstart judge_alignment
cd judge_alignment

安装依赖

uv sync

设置 API Key

export OPENAI_API_KEY="your-openai-key"

运行评测

uv run python evals.py

项目结构

judge_alignment/
├── README.md              # Project documentation
├── pyproject.toml         # Project configuration
├── evals.py               # Evaluation workflow
├── __init__.py            # Python package marker
└── evals/
    ├── datasets/          # Test datasets
    ├── experiments/       # Evaluation results
    └── logs/              # Execution logs

评测内容

该模板评测 LLM judge 对齐：

• 场景：由 LLM judge 评测已有回复
• 人工标签：作为 ground truth 的通过/失败标签
• LLM Judge：用评分标准评测同一批回复
• Alignment Metric：人工判断与 LLM 判断之间的一致性

理解代码

Judge 指标（evals.py）

两种可比较的 judge 实现：

# Baseline judge (simple prompt)
accuracy_metric = DiscreteMetric(
    name="accuracy",
    prompt="Check if response contains points from grading notes...",
    allowed_values=["pass", "fail"],
)

# Improved judge (enhanced with abbreviation guide)
accuracy_metric_v2 = DiscreteMetric(
    name="accuracy",
    prompt="""Evaluate if response covers ALL key concepts...

    ABBREVIATION GUIDE:
    • Financial: val=valuation, post-$=post-money, rev=revenue...
    • Business: mkt=market, reg=regulation...
    """,
    allowed_values=["pass", "fail"],
)

评测

测试与人工判断的对齐：

@discrete_metric(name="alignment", allowed_values=["aligned", "misaligned"])
def alignment_metric(llm_judgment: str, human_judgment: str):
    # Compares LLM judge output with human label
    return "aligned" if llm_judgment == human_judgment else "misaligned"

测试数据

数据集包含：

• 预先评测过的回复
• 人工通过/失败标签
• 带有期望要点的评分说明
• 各种缩写与商业术语

用例

比较 Judge 版本

用两种 judge 运行实验：

# Test baseline judge
results_v1 = await run_with_judge(accuracy_metric)

# Test improved judge
results_v2 = await run_with_judge(accuracy_metric_v2)

# Compare alignment rates

提升 Judge 质量

迭代 judge prompt 以提升对齐：

1. 识别错位模式
2. 用更清晰的标准更新 judge prompt
3. 重新评测对齐
4. 重复直到满意

下一步

• Prompt 评测 - 比较不同 prompt
• LLM 基准测试 - 比较不同模型