PickBestReplica (clusterrouting/replica.go)

Go function that selects the best replica for a model based on in-flight, last used, and available VRAM.

// Source: https://github.com/mudler/LocalAI | License: MIT | Copyright 2023-2025 Ettore Di Giacinto (mudler@localai.io)
// Provided via justclaw.it.com; keep this notice with the code.
// Package clusterrouting holds the transport-agnostic replica selection policy
// shared by the NATS distributed mode (core/services/nodes) and the p2p
// federation server (core/p2p). It deliberately depends on nothing heavier than
// the standard library so either transport can import it without pulling in a
// database driver or message bus.
package clusterrouting

import "time"

// ReplicaCandidate is the minimum view of a loaded model replica needed to
// apply the routing policy. It is intentionally decoupled from any storage
// model (gorm rows on the NATS side, gossiped NodeData on the p2p side) so

... (truncated -- full source via MCP)