How we Built an Asynchronous Messaging Solution by Deploying “Kafka-as-a-service” for a Premier Global Media & Entertainment Company

Our client used an existing messaging platform that offered minimal to zero asynchronous communication between services.

The existing app was built on a manual, non-serviceable architecture. Due to this, simple tasks like creating topics became a pain area. Over time, it became susceptible to downtime risks.  

The client reviewed the issue and recorded the following performance concerns that needed to be addressed immediately:

• The platform required a redesign/release from scratch with a self-serviceable and scalable infrastructure.
• The Observability team intended to use Kafka primarily to design a perfect HA solution.
• The existing platform required to be migrated to a fully automated KaaS platform. So that the teams could simply write a yaml file with details like the topic name and list of publishers and consumers, and the KaaS platform automatically creates the topics.
• The crucial part was boosting the platform's overall performance of slow-running processes.  

The client needed a strategic engineering partner who could effectively brainstorm a practical roadmap with their observability team and technologically lead them to a better platform with faster release cycles.

Global media and entertainment joined hands with Symv as its strategic engineering partner because of our expertise in Kafka Cluster in production at a large scale.

However, a big challenge was the critical timeline of the project. The client needed to develop and migrate to the new platform in less than 2 months. Symv took on the challenge of automating the existing platform to spin up the infrastructure within the expected timeline.

Symv team deployed an engineering team of 1 Tech Lead, TPM (Technical Project Manager), and 3 Engineers to start working on the project in the mid of September 2021.

Our team designed and built 2 KaaS platforms for the client:

• One is to address the business data communication the service teams use.
• The other KaaS platform set up for the Observability team was used to send metrics from logs and client telemetric data. This Kafka addresses a scale of 1 GB/s and peaks during major streaming events.

Our strategic approach and agile methodologies helped the client build an all-new platform to enhance performance. The team worked in two phases. ‍

Phase 1: Managing the existing Kafka Clusters

• Our team helped the client set up Kafka environments in the development and integration environment with the existing vendor AWS MSK.
• Kafka would be the messaging bus that every service would use.
• The service teams wanted the self-service feature of Creating topics and configuring the ACLs. The service teams would have to use this Kafka-as-a-Service platform for their async messaging requirements.
• Symv team built HA Kafka clusters on AWS with mirroring and cross-account replication. The setup was done using Infrastructure as Code (IaC) using Terraform and deployed on top of the Kubernetes cluster. Helm charts, Jenkins, GitHub Actions, and Flux, were used in the automation.

• The Kafka clusters have 150 topics and are used by over 80 applications that are producing 340M events per day on average.
  

Phase 2: Expanding the Kafka Usage for inter-service communication as well as operational data load

• In this phase, we could choose from multiple options for Kafka vendors.
• Therefore, detailed research was done regarding features, cost, etc. After brainstorming for around 2 months, we finally chose - Confluent-based Kafka. This choice was going to impact their global team of 1000s of engineers.
• Our team worked on improving the overall Kafka architecture. Build an all-new long-term Kafka Platform, using Confluent and migrating all the development/integration to Confluent, which was previously built on AWS MSK.
• The team worked on all the automation required to spin up the infrastructure. It made all the teams get started with Kafka. It took 15 mins to get onboarded on Kafka.
• In this phase, the team also set up a staging/production environment and a global and regional strategy for Kafka data replication.