Metaharness Leakage Detector (src/leakage.ts)

metaharness · packages/evals-extract/src/leakage.ts · typescript · MIT
testing, ai, security, lang:typescript

Detects data leakage and n-gram overfit between candidate extraction policies and benchmark holdout sets. Reuse when implementing fail-closed evaluation gates.

// Source: https://github.com/ruvnet/metaharness | License: MIT | Copyright 2026 RuvNet (https://ruv.io)
// Provided via justclaw.it.com; keep this notice with the code.
// @metaharness/evals-extract — the LEAKAGE DETECTOR (fail-closed).
//
// The single biggest failure mode of benchmark policy evolution is overfit-by-leakage: a policy that encodes
// document-specific hacks, memorized gold objects, or dataset-artifact language. This detector runs on every
// candidate policy BEFORE it can be scored/promoted. Any positive signal fails the candidate CLOSED — it is
// marked `regressed` so the frozen gate rejects it outright. Anti-overfit is procedural, not aspirational.
import type { ExtractPolicyGenome } from './genome.js';

export interface LeakageCheck {
  /** Max n-gram overlap between 

... (truncated -- full source via MCP)

See the full source, get the GitHub permalink, and search for more like it.

Get a free API key