Kafka: Building scalable, secure, fast applications.

Photo by Florian Krumm on Unsplash
This is how your system would work if you used Kafka
  1. Producer: These are apps that are reponsible for pushing data into the Kafka System. It sends data asynchronously to the Topics.
  2. Broker: Instances of Kafka that commit message exchances are called Brokers.
  3. Topics: Topic are the table equivalent of Kafka systems. All inputs to the systems are stored as part of some topic.
  4. Consumers: The published messages are then used by consumer apps if the consumer has subscribed to the required topic.
Beautiful drawing of a Kafka system, innit?
  1. Scalability: You can amp up the number of producers: Multiple producers on one topic. You can partition your topics. You can group consumers to have them consume single partitions.
  2. High Throughput: Kafka can handle some high volume of data.
  3. Fault Tolerant: Clusters are usually fault-tolerant because if a node fails, the other ones take up the load.
  4. Durability: Data is persistent and message replication makes it more durable.
Courtesy: Lourenço, João & Cabral, Bruno & Carreiro, Paulo & Vieira, Marco & Bernardino, Jorge. (2015). Choosing the right NoSQL database for the job: a quality attribute evaluation. Journal of Big Data.
  1. Linkedin: Linkedin developed Kafka internally as a part of the infrastructure. I am not sure how Linkedin uses Kafka, but newsfeed, jobs, feed- all seem like a huge pubsub function.
  2. Twitter: Twitter uses Kafka to stream posts, typical pubsub architecture.
  3. Netflix: Event Processing and RT monitoring systems.




Building: Ignite Tournaments / Former: DeFi Alliance, Persistence, Eth India Co

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

3.4 Visual programming quick start

Create a Flutter Plugin To Write Platform-specific Code for Windows

Computer Vision for Busy Developers: Image Processing

StrapiConf 2022: Our favorite talks

Deploying your Spring Boot Java app to AWS and using Azure AppInsights for monitoring?

6 Easy Ways to Manage and Harden VM Images in Azure

A Day in the Life: Anna Holland Smith, THG’s Accelerator

Monitoring Laravel’s background queues in realtime

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Abhinav Srivastava

Abhinav Srivastava

Building: Ignite Tournaments / Former: DeFi Alliance, Persistence, Eth India Co

More from Medium

Knowing When to Stop Is Part of the Victory

Notes on “Paxos made moderately complex”

Braze — Customer Engagement Platform for OTT and TV Display

Memcached Keys Expirations