Kafka Sticky Partitioner (partitioner/sticky.py)

kafka-python · kafka/partitioner/sticky.py · python · Apache-2.0
infra, integration, realtime, chat, lang:python, tag:partitioner

Implements KIP-480 sticky partitioning strategy for Kafka producers to optimize batch sizes and throughput. Reuse when tuning Kafka message distribution performance.

# Source: https://github.com/dpkp/kafka-python | License: Apache-2.0 | Copyright You must retain, in the Source form of any Derivative Works
# Provided via justclaw.it.com; keep this notice with the code.
"""KIP-480 sticky partitioner.

Records with a non-None key are hashed to a partition just like
:class:`~kafka.partitioner.default.DefaultPartitioner`. Records with a
None key go to a *sticky* partition - i.e. the same partition is reused
for every null-key record on a topic until KafkaProducer signals that a
batch has been completed (via :meth:`StickyPartitioner.on_new_batch`),
at which point a different partition is picked.

The goal is to give the RecordAccumulator larger, denser batches for
null-key sends so per-batch overhead (CRC, compression, broker
round-trip) is amortized across 

... (truncated -- full source via MCP)

See the full source, get the GitHub permalink, and search for more like it.

Get a free API key