Resolve Kafka/Redpanda Consumer Errors With Message Size Limits
Kafka/Redpanda consumer error? Check message size limits! Increase `fetch.message.max.bytes` to resolve `BrokerTransportFailure`. Default limit is too low for large messages.
If you're using Redpanda (or Kafka) and hitting mysterious consumer errors like:
Message consumption error: BrokerTransportFailure
You might be dealing with a message size limit issue! I ran into this recently, and increasing the fetch.message.max.bytes limit solved it instantly.
The Problem
My Rust consumer, built with rdkafka, was randomly failing with BrokerTransportFailure. Debugging didn’t reveal much rather going back n forth as everything otherwise with broker connection, topic, network reachability was all good; but after experimenting with different configur...