Terraform AWS MSK Kafka Cluster module

Oct 13, 2025 by pynkocean.com 86 views

MSK Express Brokers are Not Supported: A Critical Issue in Terraform AWS MSK Kafka Cluster

Introduction

The Terraform AWS MSK Kafka Cluster module is a powerful tool for provisioning and managing Amazon MSK clusters. However, a critical issue has been identified that prevents users from creating clusters with Express instance types and broker logs enabled. In this article, we will delve into the details of this issue, explore the expected behavior, and provide steps to reproduce the problem.

Describe the Bug

When attempting to create an MSK cluster with Express instance types and a broker_logs block specified, the following error is returned:

Broker logs are not supported for clusters with Express instance types

This error is particularly frustrating, as it prevents users from taking advantage of the Express instance types, which offer significant cost savings and improved performance. Furthermore, when the broker_logs block is not specified, the following error is returned:

missing required field, CreateClusterInput.LoggingInfo.BrokerLogs.

This error is even more puzzling, as it suggests that the broker_logs block is a required field, yet it is not specified in the documentation.

Expected Behavior

Fortunately, a fix has been implemented in the Terraform MSK module, which allows for the creation of clusters with Express instance types and broker logs enabled. The relevant pull request can be found on GitHub:

https://github.com/terraform-aws-modules/terraform-aws-msk-kafka-cluster/pull/48/files

This fix is a significant improvement, as it enables users to take advantage of the Express instance types and broker logs, while also ensuring that the required fields are properly specified.

Steps to Reproduce

To reproduce this issue, follow these simple steps:

Choose any express.xx.xx instance type.
Attempt to create an MSK cluster with a broker_logs block specified.
Observe the error message indicating that broker logs are not supported for clusters with Express instance types.

Alternatively, if you do not specify a broker_logs block, you will encounter the error message indicating that the required field CreateClusterInput.LoggingInfo.BrokerLogs is missing.

Screenshots

Unfortunately, there are no screenshots available to illustrate this issue, as it is a purely text-based error message.

Environment

The environment in which this issue occurs is not specified, as it is likely to be a generic issue that affects all users attempting to create MSK clusters with Express instance types and broker logs enabled.

Additional Context

There is no additional context available to provide further insight into this issue. However, it is worth noting that this issue is likely to be a result of a misconfiguration or a bug in the Terraform AWS MSK Kafka Cluster module.

Conclusion

In conclusion, the MSK Express brokers are not supported issue is a critical problem that prevents users from creating clusters with Express instance types and broker logs enabled. While a fix has been implemented in the Terraform MSK module, it is essential to be aware of this issue and take steps to reproduce it to ensure that it is properly addressed.

Recommendations

To avoid this issue, we recommend the following:

Use a different instance type, such as kafka.m5.xlarge.
Specify a broker_logs block with all logging types disabled.
Use the Terraform MSK module, which has been updated to include the fix for this issue.

By following these recommendations, you can ensure that your MSK clusters are created successfully and that you can take advantage of the Express instance types and broker logs.

Future Development

In the future, it would be beneficial to include additional logging and error handling mechanisms to prevent this issue from occurring. Additionally, it would be helpful to provide more detailed documentation and examples to illustrate the correct usage of the broker_logs block.

Related Issues

This issue is related to the following:

Glossary

MSK: Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that makes it easy to run Apache Kafka clusters in the cloud.
Express instance types: Express instance types are a type of instance that offers significant cost savings and improved performance.
Broker logs: Broker logs are a feature that allows users to log and monitor the activity of their MSK clusters.
Terraform AWS MSK Kafka Cluster module: The Terraform AWS MSK Kafka Cluster module is a powerful tool for provisioning and managing Amazon MSK clusters.
MSK Express Brokers are Not Supported: A Q&A Article

Introduction

In our previous article, we explored the issue of MSK Express brokers not being supported in the Terraform AWS MSK Kafka Cluster module. In this article, we will provide a Q&A section to address some of the most frequently asked questions related to this issue.

Q&A

Q: What is the cause of the MSK Express brokers not being supported issue?

A: The cause of this issue is a misconfiguration or a bug in the Terraform AWS MSK Kafka Cluster module. Specifically, the module does not support the creation of clusters with Express instance types and broker logs enabled.

Q: How can I reproduce the MSK Express brokers not being supported issue?

A: To reproduce this issue, follow these simple steps:

Choose any express.xx.xx instance type.
Attempt to create an MSK cluster with a broker_logs block specified.
Observe the error message indicating that broker logs are not supported for clusters with Express instance types.

Alternatively, if you do not specify a broker_logs block, you will encounter the error message indicating that the required field CreateClusterInput.LoggingInfo.BrokerLogs is missing.

Q: What are the consequences of the MSK Express brokers not being supported issue?

A: The consequences of this issue are that users are unable to create clusters with Express instance types and broker logs enabled. This can lead to increased costs and decreased performance.

Q: How can I avoid the MSK Express brokers not being supported issue?

A: To avoid this issue, we recommend the following:

Use a different instance type, such as kafka.m5.xlarge.
Specify a broker_logs block with all logging types disabled.
Use the Terraform MSK module, which has been updated to include the fix for this issue.

Q: Is there a fix for the MSK Express brokers not being supported issue?

A: Yes, a fix has been implemented in the Terraform MSK module, which allows for the creation of clusters with Express instance types and broker logs enabled. The relevant pull request can be found on GitHub:

https://github.com/terraform-aws-modules/terraform-aws-msk-kafka-cluster/pull/48/files

Q: How can I get help with the MSK Express brokers not being supported issue?

A: If you are experiencing issues with the MSK Express brokers not being supported, we recommend reaching out to the Terraform community or the AWS support team for assistance.

Q: What are the related issues to the MSK Express brokers not being supported issue?

A: The related issues to this issue are:

Conclusion

In conclusion, the MSK Express brokers are not supported issue is a critical problem that prevents users from creating clusters with Express instance types and broker logs enabled. We hope that this Q&A article has provided you with the information you need to address this issue and get back to work.

Glossary

MSK: Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that makes it easy to run Apache Kafka clusters in the cloud.
Express instance types: Express instance types are a type of instance that offers significant cost savings and improved performance.
Broker logs: Broker logs are a feature that allows users to log and monitor the activity of their MSK clusters.
Terraform AWS MSK Kafka Cluster module: The Terraform AWS MSK Kafka Cluster module is a powerful tool for provisioning and managing Amazon MSK clusters.