Extractive Prose Segmentation Unit (extractive/segment.rs)

Provides deterministic segmentation of text into ranking units for extractive compression, keeping structure intact. Reuse when building text summarization or NLP context pruning pipelines.

// Source: https://github.com/yvgude/lean-ctx | License: Apache-2.0 | Copyright You must retain, in the Source form of any Derivative Works
// Provided via justclaw.it.com; keep this notice with the code.
//! Deterministic segmentation of prose into ranking units for extractive
//! compression ([`super`]).
//!
//! A unit is a sentence, except structurally important lines (headings, links,
//! list/quote markers, citations — see [`is_protected_line`]) which are kept
//! whole and never sentence-split or dropped. Blank lines delimit paragraphs so
//! the re-emitted text keeps its shape. The function is a pure, allocation-only
//! transform — no model, no I/O — so the #498 determinism contract holds before
//! any embedding is involved.

use crate::core::web::distill::{is_protected_line, spli

... (truncated -- full source via MCP)