Bitpanda's New Trade Engine Part 5: Optimizing Kafka Clients for High Performance

In our previous article, we explored how we built a coordinated custom saga pattern to handle distributed transactions atomically across our microservices.

Throughout this series, one technology has been at the core of our trade engine and the entire ecosystem: Apache Kafka. On average, almost a billion events flow through Kafka daily in our platform, connecting every microservice, every transaction, and every customer interaction.

Because Kafka is so fundamental to our architecture, we need to treat it with care and optimize it continuously. We don't wait for the first spike to hurt our system - by then, it's too late. At our scale, running hundreds of applications across the platform, making company-wide adoptions is hard and time-consuming. Continuous optimization keeps resource usage low and the system stable, ensuring we have enough buffer for major traffic spikes or unexpected world events.

With that in mind, we want to share what we believe every Kafka client should watch out for.

Tune Your Fetch Settings

Batch consumption in Kafka is not controlled purely by your application - it depends heavily on how the broker delivers data.
Kafka consumers fetch messages asynchronously and buffer them internally. Your batch consumer simply drains this buffer. That means batch size is largely determined by fetch configuration.

Many developers think that batch consumption is controlled by application code:

@KafkaListener(batch = true)
public class TradeListener {
    @Topic("trade-created")
    public void processBatch(List trades) {
        // Process batch of trades
        System.out.println("Batch size: " + trades.size());
    }
}

You might expect consistent batch sizes based on your topic throughput. Instead, you'll see wildly inconsistent results:

• Sometimes batches of 1 message
• Sometimes batches of 500+ messages
• No predictable pattern

The reason? Batch size isn't determined by your application - it's determined by what the Kafka broker delivers.

Here's what actually happens:

1. Your consumer sends a fetch request to the broker
2. The broker looks at two critical settings:

• fetch.min.bytes: Minimum data to accumulate before responding
• fetch.max.wait.ms: Maximum time to wait for that minimum

3. The broker responds when either condition is met
4. Your batch consumer drains whatever arrived

If fetch.min.bytes is too low or fetch.max.wait.ms is too short, the broker responds immediately with whatever it has - often just a handful of messages. Your "batch" consumer processes tiny batches, defeating the purpose entirely.

For high-throughput topics, you should tune:

• fetch.min.bytes - wait until enough data accumulates before responding
• fetch.max.wait.ms - maximum wait time before the broker returns data

These settings allow Kafka to build larger batches, improving efficiency.

For low-throughput and time-critical topics, the priorities shift toward responsiveness:

• Use a low fetch.min.bytes
• Allow a slightly higher fetch.max.wait.ms to avoid excessive polling

Different workloads require different strategies - there is no one-size-fits-all configuration.

Example: Hydration Service (prices consumer)

We receive thousands of price updates every second and therefore we optimise for batch consumption:

fetch.min.bytes=200000
fetch.max.wait.ms=250
fetch.max.bytes=400000

Prefer throughput; tolerate up to 250ms additional wait to allow messages to accumulate while bounding response size.

Example: Offer Service (offer creation consumer)

Latency-oriented configuration for time-sensitive offer creation:

fetch.min.bytes=1
fetch.max.wait.ms=3000

Consume as soon as anything is available; when idle, long-poll up to 3s to reduce churn.

Producer: Use Producer Batching

Kafka producers are asynchronous by design. When you call send(), the record doesn't go over the network immediately. Instead, it's placed in an internal buffer while a background sender thread handles the actual transmission.

This design enables batching - multiple records can be grouped into a single network request. But batching only happens if you allow it.

The default trap is

linger.ms = 5

Most Kafka producer configurations default to very low linger, which means "send the batch as soon as 5 ms pass" (Apache Kafka 3.9 defaulted to 0ms). For applications that produce messages one at a time with gaps between them, this results in tiny batches - often just a single message per network request.

Adding Strategic Delay

Here's a Micronaut producer configuration with minimal batching:

kafka:
  producers:
    default:
      linger.ms: 5

Let's look at a black swan event scenario where we, let’s say, book on average 200 trades/sec across 5 application instances (40 trades/sec per instance).

With linger of 5ms and processing speed of 1 trade every 25ms, there is no time to accumulate more messages in the buffer and therefore:

• Average batch size: 1
• Network requests per second: 40/sec per instance (200/sec for all)

Now increase the linger window like this:

kafka:
  producers:
    default:
      linger.ms: 100

With a linger of 100ms, we can accumulate 4 messages in the buffer and therefore:

• Average batch size: 4
• Network requests per second: 10/sec per instance (50/sec for all)
• CPU usage (producer overhead): significantly reduced

That means that if we can sacrifice a delay of up to 100ms, we can achieve up to 75% better efficiency.

For non-time-critical services like email sending, we use even higher delay since latency is less critical.

Choosing the Right Linger Value

The optimal linger.ms depends on your architecture and latency requirements:

For synchronous processes (like PHP):

If your process is synchronous, there's no need for the default 5ms. In 5ms, you won't be able to process another request anyway. Use linger of 0 for immediate sends, or set it way higher (50-100ms+) if latency isn't critical.

For asynchronous processes (like Java):

For time-critical and user-facing processes use double-digit numbers (10-20ms). If you have background processing that is not user-facing, you can safely go up to 1000ms.

For our trade engine core flows (actual trade execution), we keep linger of 0ms. For non-time-critical services (notifications, emails, analytics), we use 50-100ms.

Measuring Producer Effectiveness

To determine if your Kafka application needs better batching, you can calculate a simple batching efficiency score.

Divide your average records per request by the total number of records sent over a specific window (e.g., one minute):

• A score near 0 (Low): This indicates poor efficiency. Your producer is sending many tiny requests rather than grouping them together, which increases overhead and slows down your system.
• A score near 1 (High): This indicates high efficiency. Your producer is effectively grouping records into large, infrequent batches.

For example, if you send 1 000 records in one minute:

• Inefficient: If you send 1 record per request, your score is 1 / 1 000 \= 0,001. That doesn’t look good.
• Efficient: If you send 20 records per request, your score is 200 / 1 000 \= 0,2. That looks better!

Choose the Right Commit Strategy

In Kafka, offset commits track which messages have been processed. This enables recovery after failures - consumers resume from their last committed offset.

But committing offsets isn't free. Each commit:

• Writes to the internal __consumer_offsets topic
• Requires acknowledgement from the broker
• Can block your consumer thread (if synchronous)

The commit strategy may have a significant impact on throughput and latency.

Scenario 1: Synchronous Commit After Each Message

The simplest approach is synchronous commit after every message:

@KafkaListener
public class TradeListener {
    @Topic("trade-created")
    public void handleTrade(Trade trade) {
        processTrade(trade);
        consumer.commitSync();  // Block until offset is committed
    }
}

This guarantees exactly-once processing within a partition (combined with idempotency), but blocks on every commit, severely limiting throughput.

Scenario 2: Asynchronous Commit After Each Message

To avoid blocking, we should switch to asynchronous commits:

@KafkaListener
public class TradeListener {
    @Topic("trade-created")
    public void handleTrade(Trade trade) {
        processTrade(trade);
        consumer.commitAsync((offsets, exception) -> {
            if (exception != null) {
                log.error("Commit failed: " + offsets, exception);
            }
        });
    }
}

While non-blocking, this approach is not ideal for topics with high throughput or for topics that periodically get updated data (like price updates), as you're still committing after every single message, increasing CPU usage of your application as well as CPU usage of your Broker.

Scenario 3: Asynchronous Commit After Batch

To overcome over-committing, we can commit asynchronously after processing a batch of messages:

@KafkaListener(batch = true)
public class TradeListener {
    @Topic("trade-created")
    public void handleBatch(List trades) {
        trades.stream().forEach(this::processTrade);
        consumer.commitAsync((offsets, exception) -> {
            if (exception != null) {
                log.error("Commit failed: " + offsets, exception);
            }
        });
    }
}

This reduces commit overhead significantly - one non-blocking commit for many messages instead of one per message.

Scenario 4: Commit in Intervals

For topics with very high throughput, commit periodically based on time intervals.

If message loss is acceptable, use auto-commit:

enable.auto.commit=true
auto.commit.interval.ms=5000

Kafka automatically commits the latest processed offset every 5 seconds.

If message loss is not acceptable, you need a custom solution that keeps track of processed offsets per partition and commits them manually every 3-5 seconds. This ensures you only commit offsets for messages that have been fully processed, while still maintaining the performance benefits of interval-based commits.

To recap, this is how we usually decide:

• Very low throughput (like 1 message per hour) -> synchronous commit after each message
• Moderate throughput (like 1 message per second) -> asynchronous commit after each message
• High throughput - can’t lose a single message (like 20 messages per second) -> asynchronous commit after processing a batch of messages
• Very high throughput - can lose messages (like 1 000 messages per second) -> auto commit every couple of seconds

Consolidate Your Consumers

Many Kafka frameworks create one high-level consumer for each topic subscription. While this abstraction is convenient, it can easily hide the real cost. Each consumer brings multiple internal threads (fetching, heartbeats, metadata updates), along with its own network connections and memory overhead. Subscribing to many topics individually can therefore multiply resource usage far beyond what's obvious in the code.

Let's look at what happens when you have multiple topic subscriptions. Here's a common pattern with three separate listeners:

@KafkaListener
public class AssetCreatedListener {
    @Topic("asset-created")
    public void handleAssetCreated(Asset asset) {
        // Handle asset creation
    }
}

@KafkaListener
public class AssetUpdatedListener {
    @Topic("asset-updated")
    public void handleAssetUpdated(Asset asset) {
        // Handle asset updates
    }
}

@KafkaListener
public class AssetFeeChangedListener {
    @Topic("asset-fee-changed")
    public void handleAssetFeeChanged(AssetFee fee) {
        // Handle fee changes
    }
}

This creates three separate consumers, resulting in:
- 9 threads total (3 per consumer: fetcher, heartbeat, metadata updater)
- Network connections multiplied: If topics have multiple partitions and those partition leaders reside on different brokers, we will have a separate connection to each and every broker
- 3× memory overhead for fetch buffers and decompression

Now consolidate them into a single consumer by grouping topics:

@KafkaListener
public class AssetEventListener {
    @Topic("asset-created", "asset-updated", "asset-fee-changed")
    public void handleAssetEvents(String topic, GenericRecord record) {
        switch (topic) {
            case "asset-created":
                handleAssetCreated(record);
                break;
            case "asset-updated":
                handleAssetUpdated(record);
                break;
            case "asset-fee-changed":
                handleAssetFeeChanged(record);
                break;
        }
    }
}

This creates one consumer, resulting in:
- 3 threads total (fetcher, heartbeat, metadata updater)
- Less broker connections (one per broker where partition leaders live)
- 1× memory overhead

By consolidating, you reduce threads by 67% and the sheer amount of connections you need to create to the brokers.

A quick recap:

• For high-throughput topics - one dedicated consumer per topic
• For low-throughput topics - consolidate them into a single consumer (if consumers can share the same configuration settings such as: fetch settings, deserialization strategy or retry configuration)

Epilogue

Many engineers trust default settings with the reasoning: "They know best, so they set good defaults." And they're right - defaults work for the average use case.

But in Fintech, there's no such thing as average.

Today we handle thousands of requests per second. Tomorrow, a black swan or white swan events happen which results in heavy market movements and causes tens of thousands of requests per second. This year we offer 10 000 assets. Next year, we may push for 15 000 assets. Financial markets don't follow predictable patterns, and neither does our traffic.

That's why we can't rely on defaults. We need to understand our specific use case and optimize for it.

These four optimizations - tuning fetch settings, enabling producer batching, choosing the right commit strategy and consolidating consumers - might seem small. But during turbulent times, when market volatility spikes and everyone rushes to trade, these details make the difference between a system that holds strong and one that buckles.

And those moments? They're the ones that define your reputation.