Tokenizers

Ragas 支持多种 tokenizer 实现，用于知识图谱操作和测试数据生成过程中的文本切分。

概览

从知识图谱节点提取属性时，文本会按 token 上限切成 chunk。默认情况下，Ragas 使用 tiktoken（OpenAI 的 tokenizer），你也可以使用 HuggingFace tokenizer，以便更好地兼容开源模型。

可用 Tokenizers

TiktokenWrapper

OpenAI tiktoken tokenizer 的包装器。这是默认 tokenizer。

from ragas import TiktokenWrapper

# Using default encoding (o200k_base)
tokenizer = TiktokenWrapper()

# Using a specific encoding
tokenizer = TiktokenWrapper(encoding_name="cl100k_base")

# Using encoding for a specific model
tokenizer = TiktokenWrapper(model_name="gpt-4")

HuggingFaceTokenizer

HuggingFace transformers tokenizer 的包装器。在使用开源模型时选用它。

from ragas import HuggingFaceTokenizer

# Load tokenizer for a specific model
tokenizer = HuggingFaceTokenizer(model_name="meta-llama/Llama-2-7b-hf")

# Use a pre-initialized tokenizer
from transformers import AutoTokenizer
hf_tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
tokenizer = HuggingFaceTokenizer(tokenizer=hf_tokenizer)

注意： HuggingFace tokenizer 需要 transformers 包。安装方式：

pip install transformers
# or
uv add transformers

工厂函数

使用 get_tokenizer() 可简便地创建 tokenizer：

from ragas import get_tokenizer

# Default tiktoken tokenizer
tokenizer = get_tokenizer()

# Tiktoken for a specific model
tokenizer = get_tokenizer("tiktoken", model_name="gpt-4")

# HuggingFace tokenizer
tokenizer = get_tokenizer("huggingface", model_name="meta-llama/Llama-2-7b-hf")

使用自定义 Tokenizers

与基于 LLM 的 Extractor 一起使用

所有基于 LLM 的 extractor 都接受 tokenizer 参数：

from ragas import HuggingFaceTokenizer
from ragas.testset.transforms import (
    SummaryExtractor,
    KeyphrasesExtractor,
    HeadlinesExtractor,
)

# Create a HuggingFace tokenizer for your model
tokenizer = HuggingFaceTokenizer(model_name="meta-llama/Llama-2-7b-hf")

# Use it with extractors
summary_extractor = SummaryExtractor(llm=your_llm, tokenizer=tokenizer)
keyphrase_extractor = KeyphrasesExtractor(llm=your_llm, tokenizer=tokenizer)
headlines_extractor = HeadlinesExtractor(llm=your_llm, tokenizer=tokenizer)

自定义 Tokenizer 实现

你可以通过扩展 BaseTokenizer 创建自己的 tokenizer：

from ragas.tokenizers import BaseTokenizer

class MyCustomTokenizer(BaseTokenizer):
    def __init__(self, ...):
        # Initialize your tokenizer
        pass

    def encode(self, text: str) -> list[int]:
        # Return token IDs
        pass

    def decode(self, tokens: list[int]) -> str:
        # Return decoded text
        pass

API 参考

Ragas 的 tokenizer 抽象。

本模块为不同 tokenizer 实现提供统一接口，同时支持 tiktoken（OpenAI）和 HuggingFace tokenizer。

BaseTokenizer

基类：ABC

tokenizer 的抽象基类。

encode

encode(text: str) -> List[int]

将文本编码为 token ID。

源代码位于 src/ragas/tokenizers.py

@abstractmethod
def encode(self, text: str) -> t.List[int]:
    """Encode text into token IDs."""
    pass

decode

decode(tokens: List[int]) -> str

将 token ID 解码回文本。

源代码位于 src/ragas/tokenizers.py

@abstractmethod
def decode(self, tokens: t.List[int]) -> str:
    """Decode token IDs back into text."""
    pass

count_tokens

count_tokens(text: str) -> int

统计文本中的 token 数量。

源代码位于 src/ragas/tokenizers.py

def count_tokens(self, text: str) -> int:
    """Count the number of tokens in text."""
    return len(self.encode(text))

TiktokenWrapper

TiktokenWrapper(encoding: Optional[Encoding] = None, model_name: Optional[str] = None, encoding_name: Optional[str] = None)

基类：BaseTokenizer

tiktoken encoding（OpenAI tokenizer）的包装器。

参数：

源代码位于 src/ragas/tokenizers.py

def __init__(
    self,
    encoding: t.Optional[tiktoken.Encoding] = None,
    model_name: t.Optional[str] = None,
    encoding_name: t.Optional[str] = None,
):
    """
    Initialize TiktokenWrapper.

    Parameters
    ----------
    encoding : tiktoken.Encoding, optional
        A pre-initialized tiktoken encoding.
    model_name : str, optional
        Model name to get encoding for (e.g., "gpt-4", "gpt-3.5-turbo").
    encoding_name : str, optional
        Encoding name (e.g., "cl100k_base", "o200k_base").

    If none provided, defaults to "o200k_base" encoding.
    """
    if encoding is not None:
        self._encoding = encoding
    elif model_name is not None:
        self._encoding = tiktoken.encoding_for_model(model_name)
    elif encoding_name is not None:
        self._encoding = tiktoken.get_encoding(encoding_name)
    else:
        self._encoding = tiktoken.get_encoding("o200k_base")

encoding

encoding: Encoding

访问底层的 tiktoken encoding。

HuggingFaceTokenizer

HuggingFaceTokenizer(tokenizer: Optional[Any] = None, model_name: Optional[str] = None)

基类：BaseTokenizer

HuggingFace tokenizer 的包装器。

参数：

源代码位于 src/ragas/tokenizers.py

def __init__(
    self,
    tokenizer: t.Optional[t.Any] = None,
    model_name: t.Optional[str] = None,
):
    """
    Initialize HuggingFaceTokenizer.

    Parameters
    ----------
    tokenizer : PreTrainedTokenizer or PreTrainedTokenizerFast, optional
        A pre-initialized HuggingFace tokenizer.
    model_name : str, optional
        Model name or path to load tokenizer from (e.g., "meta-llama/Llama-2-7b").

    One of tokenizer or model_name must be provided.
    """
    if tokenizer is not None:
        self._tokenizer = tokenizer
    elif model_name is not None:
        try:
            from transformers import AutoTokenizer
        except ImportError:
            raise ImportError(
                "transformers package is required for HuggingFace tokenizers. "
                "Install it with: pip install transformers"
            )
        self._tokenizer = AutoTokenizer.from_pretrained(model_name)
    else:
        raise ValueError("Either tokenizer or model_name must be provided")

tokenizer

tokenizer: Any

访问底层的 HuggingFace tokenizer。

get_default_tokenizer

get_default_tokenizer() -> TiktokenWrapper

获取默认 tokenizer，在首次访问时惰性创建。

源代码位于 src/ragas/tokenizers.py

def get_default_tokenizer() -> TiktokenWrapper:
    """Get the default tokenizer, creating it lazily on first access."""
    global _default_tokenizer
    if _default_tokenizer is None:
        _default_tokenizer = TiktokenWrapper(encoding_name="o200k_base")
    return _default_tokenizer

get_tokenizer

get_tokenizer(tokenizer_type: str = 'tiktoken', model_name: Optional[str] = None, encoding_name: Optional[str] = None) -> BaseTokenizer

获取 tokenizer 实例的工厂函数。

参数：

返回：

示例：

>>> # Get default tiktoken tokenizer
>>> tokenizer = get_tokenizer()

>>> # Get tiktoken for a specific model
>>> tokenizer = get_tokenizer("tiktoken", model_name="gpt-4")

>>> # Get HuggingFace tokenizer
>>> tokenizer = get_tokenizer("huggingface", model_name="meta-llama/Llama-2-7b")

源代码位于 src/ragas/tokenizers.py

def get_tokenizer(
    tokenizer_type: str = "tiktoken",
    model_name: t.Optional[str] = None,
    encoding_name: t.Optional[str] = None,
) -> BaseTokenizer:
    """
    Factory function to get a tokenizer instance.

    Parameters
    ----------
    tokenizer_type : str
        Type of tokenizer: "tiktoken" or "huggingface".
    model_name : str, optional
        Model name for the tokenizer.
    encoding_name : str, optional
        Encoding name (only for tiktoken).

    Returns
    -------
    BaseTokenizer
        A tokenizer instance.

    Examples
    --------
    >>> # Get default tiktoken tokenizer
    >>> tokenizer = get_tokenizer()

    >>> # Get tiktoken for a specific model
    >>> tokenizer = get_tokenizer("tiktoken", model_name="gpt-4")

    >>> # Get HuggingFace tokenizer
    >>> tokenizer = get_tokenizer("huggingface", model_name="meta-llama/Llama-2-7b")
    """
    if tokenizer_type == "tiktoken":
        return TiktokenWrapper(model_name=model_name, encoding_name=encoding_name)
    elif tokenizer_type == "huggingface":
        if model_name is None:
            raise ValueError("model_name is required for HuggingFace tokenizers")
        return HuggingFaceTokenizer(model_name=model_name)
    else:
        raise ValueError(f"Unknown tokenizer type: {tokenizer_type}")