Amazon MSK now supports up to 5x more partitions on Express Brokers

Amazon Managed Streaming for Apache Kafka (Amazon MSK) now supports up to 5x more partitions per Express Broker, improving price-performance by up to 50% for partition-bound workloads. Express Brokers are a broker type in Amazon MSK that provide up to 3x more throughput, up to 20x faster scaling, and up to 90% faster recovery compared to Standard Apache Kafka brokers. With this launch, partition-bound workloads can be hosted on fewer brokers. For example, m7g.16xlarge instances now support up to 20,000 partitions per broker vs 4,000 partitions previously. To further improve the resilience of your clusters, Amazon MSK now also now limits the maximum number of partitions that can be created per Express Broker. These quotas help maintain cluster stability by preventing partition counts from exceeding safe thresholds at these higher limits.

The increased partition support is particularly valuable for streaming applications requiring numerous topics, micro-service architectures with many small topics, and data ingestion pipelines supporting multiple data sources among others. These improvements are now available across all AWS regions where Amazon MSK Express Brokers are offered, at no additional cost. To learn more about partition limits for different instance types and best practices for scaling your Apache Kafka workloads, visit our documentation.

Categories:

Source: Amazon Web Services

Latest Posts