AWS MSK 3.8 migration

Period  — 

Support for production MSK 3.5 was set to end in 2025.10, so I built a new 3.8 cluster using KRaft metadata and recreated the existing topics on it.

Configuration for about 160 topics recreated automatically on a new cluster

Team
Solo
Role
Architecture designKRaft adoption in development and productionSASL authenticationTopic and replica configurationMessage sending testsKafka UI monitoring

AWS MSK 3.8Kafka KRaftSASL/SCRAMAWS Secrets ManagerKafka UI

Choosing the authentication method

Problem
The new cluster needed server authentication added. MSK supported IAM, TLS and SASL, among others, so the choice had to weigh compatibility with the internal environment against operating complexity.
Solution
After reviewing the options, I chose SASL/SCRAM. User accounts are stored in AWS Secrets Manager. Each server is wired to Secrets Manager and reads the account from it to authenticate.

Automating topic recreation

Problem
Every topic on the old cluster had to be recreated on the new one with the same settings. Partition counts, replicas and options differed by topic, so copying them by hand risked omissions and errors.
Solution
Exported the topic settings of the old cluster to a CSV file in one pass and reworked the replica configuration for topics that needed changes. A script reads the CSV and creates every topic on the new cluster. Topic names, partition counts, replicas and options carried over as they were, which greatly cut the time spent copying them by hand.