detect_backend (registry.rs)

Model-name → tokenizer dispatch. ! ! Mirrors `MODEL_PATTERNS` in `headroom/tokenizers/registry.py`. Three ! backends in priority order: ! ! 1. **HuggingFace** — anything the caller has registered via ! [`register_hf`] for a given model-name prefix. Real BPE/Unigram/ ! WordPiece

//! Model-name → tokenizer dispatch.
//!
//! Mirrors `MODEL_PATTERNS` in `headroom/tokenizers/registry.py`. Three
//! backends in priority order:
//!
//! 1. **HuggingFace** — anything the caller has registered via
//!    [`register_hf`] for a given model-name prefix. Real BPE/Unigram/
//!    WordPiece counts. This is opt-in: tokenizer.json files aren't bundled,
//!    so nothing routes here until the embedding application calls
//!    [`register_hf`] at startup. Wins over the rules below when registered.
//! 2. **Tiktoken** — OpenAI / o-series via `tiktoken-rs`. Byte-identical to
//!    Python `tiktoken`.
//! 3. **Estimation** — `chars / cpt` fallback for Anthropic Claude (3.5),
//!    Gemini / Cohere / Command without an HF registration (4.0), and
//!    everything else (4.0).

use std::c

... (truncated -- full source via MCP)