Implementing Kafka Static Membership Using Kubernetes Stateful Set

Kubernetes and Kafka are both crucial technologies that are transforming the way companies operate and scale their applications. When orchestrating Kafka clients with Kubernetes, how can we maintain high throughput and low latency given the potential for rebalancing that comes with consumers exiting and joining the group?

Every rebalance results in partition lag. This may be for milliseconds, or it may be much longer than that. For applications where throughput and latency absolutely cannot degrade, rebalances represent a real threat to operations.

Enter Kafka Static Membership and Kubernetes StatefulSet. In this blog post, we'll explore how to implement static membership by using StatefulSet as a means to an end.

I have created a fully functional reference architecture to go along with this post. Feel free to clone the repository and explore as you go. But first, some definitions...

Kafka Static Membership

Kafka Static Membership is an enhancement to reduce the rebalance time in consumer groups. When brokers or consumers go down, and then come back online, Kafka needs to rebalance the consumers. With static membership, the process is optimized by reusing instance ids for group members, which, in practice, virtually eliminates unnecessary rebalances.

Kubernetes StatefulSet

Kubernetes is known for its ephemeral, stateless nature. However, for applications that require persistent storage or unique network identifiers, StatefulSet is the solution. It ensures that pods maintain a sticky, consistent identity based on their ordinal index, even across rescheduling and restarts.

Implementation Overview

By combining static membership with StatefulSet, we can ensure a stable Kafka deployment within Kubernetes. The implementation is quite simple, so I will summarize the steps first before we dig in.

StatefulSet Configuration

Ensure that you use a StatefulSet to deploy consumers. This guarantees that each pod gets a stable hostname.

In Kubernetes, the hostname for a pod in a StatefulSet follows the pattern <statefulset-name>-<ordinal> (e.g., kafka-0).

Static Membership Configuration

Within the Kafka consumer configuration, enable the static membership and set the group.instance.id to the hostname of the pod. Later on we will a Spring Boot based consumer for static membership.

Implementation Details

I mentioned a reference repository earlier: kafka-static-membership. It is a fully functioning Spring Boot based producer-consumer architecture. The README highlights the following elements:

Producer

The producer retrieves a weather report for a random geo coordinate, and then publishes that weather report to a Kafka topic.

The producer module is a Spring Boot non-web application with a Scheduled service that retrieves and then publishes a weather report.

We won't dive into the producer much because it's orthogonal to this post, but feel free to mine it for patterns.

Consumer

The consumer receives weather reports from a Kafka topic, and then logs them.

The consumer module is a Spring Boot non-web application with a KafkaListener service that receives and then logs weather reports.

The consumer is where we hook into static membership:

spring.kafka.consumer.properties.group.instance.id instructs the group manager to add the consumer with the provided id, which in our case will be the stateful HOSTNAME of the pod.

spring.kafka.consumer.properties.session.timeout.ms is also important. It increases the time a consumer is allowed to be unavailable (no heartbeat response) before considering the consumer gone for good and triggering a rebalance. Your mileage will vary depending on several factors, but I suggest being generous here.

Docker Compose

docker-compose.yml contains a 3-broker Kafka cluster definition, as well as definitions for producer and consumer.

Kubernetes

The consumer deployment is configured as a StatefulSet. This is literally the only thing you need to do to opt in for a unique persistent pod name (and thus HOSTNAME).

That's about it friends. The reference repository README details how to setup, build, run, and test - so feel free to explore. Unless you keep a pod offline for more than spring.kafka.consumer.properties.session.timeout.ms you will NOT see rebalancing.

Summary of Benefits

Stability

With static membership, Kafka reduces the frequency of rebalancing, which keeps processing metrics consistent.

Predictability

Using StatefulSet guarantees that Kafka brokers will always have the same identity, making operations and debugging predictable. For example, my-app-0 will predictably be seen processing partitions x, y, and z.

Performance

Minimized rebalances translate to reduced lag and a more efficient Kafka cluster. This is the main feature that draws high throughput, low latency applications to use static membership.

Conclusion

By leveraging the unique and persistent identity provided by Kubernetes StatefulSet and integrating it with Kafka Static Membership, we can craft a resilient and stable Kafka-based architecture. It's a testament to how, by understanding the intricacies of both technologies, we can create powerful, synergistic systems that are more than the sum of their parts.

After Credits...

If you read all the way to the end, please drop me a line @RjaeEaston and let me know what you think. Include "where everybody knows your name" if you get the joke. :wink: