Microservices Integration Patterns with Kafka
10 likes•9,074 views
The document discusses microservices integration patterns using Apache Kafka, emphasizing both active/synchronous and reactive/asynchronous compositions. It explores how services can interact over networks, highlighting the importance of APIs, events, and streams in microservices architectures. Additionally, it covers hybrid composition models, Kafka Streams, event sourcing, and relevant technologies for implementing these integration patterns.
Microservices Integration Patterns with Kafka
- 1. Microservices Integration Patterns with Kafka Kasun Indrasiri Director - Integration Architecture, WSO2 Bay Area Apache Kafka Meetup @ Lyft HQ, San Francisco
- 2. ● Microservices Integration/composition? ● Microservices Composition Patterns ○ Active/Synchronous Composition ○ Reactive Composition ● Kafka for building Reactive Composition. ● Hybrid of active and reactive composition. Overview
- 3. Rise of Cloud Native Applications
- 4. Integrating Microservices and Cloud Native Apps Integration is essential to build any real world use case with cloud native apps.
- 5. ● APIs: ○ Commands - Actions, synchronous ○ Queries - Request for information, no state change, synchronous ● Events: Facts and notifications, asynchronous ● Streams: A sequence of events/data elements made available over time Microservices Integration/Composition How services can interact over the network?
- 6. Active Composition/Orchestration Pattern Synchronous/request-response communication Consumer 1 API Service P API Service Q API Service R API Service S Consumers Consumer 2 Consumer 3 μ Service A Proprietary & Legacy Systems Web API / SaaS API Services/ Edge Services Composite Service/ Integration Services Core Services/ Atomic Services μ Service I μ Service H μ Service J μ Service E μ Service C μ Service D μ Service G μ Service B μ Service F API Management
- 7. ● RESTful Services ● gRPC/HTTP2 ● GraphQL ● WebSockets ● Thrift Active Composition/Orchestration Pattern Synchronous/request-response communication
- 8. Reactive Composition/Choreography Pattern μ Service A μ Service B μ Service C Event Bus Application Asynchronous event-driven communication Produce Consume Produce Consume Produce Consume Produce Consume
- 9. ● Kafka ● AMQP ● JMS Reactive Composition/Choreography Pattern Asynchronous event-driven communication
- 10. Kafka in a nutshell Source: “Microservices for Enterprise” https://goo.gl/p3rWF3
- 11. ● Consumers from different consumer groups subscribe to the same topic. ● Consumers of a given CG can only consume from one partition. Reactive Composition with Kafka Publisher-subscriber Source: “Microservices for Enterprise” https://goo.gl/p3rWF3
- 12. ● Single consumer group and a consumer per each partition. ● Polling and sequential processing of consumer records using offset. Reactive Composition with Kafka Single Consumer/Queuing Source: “Microservices for Enterprise” https://goo.gl/p3rWF3 topic
- 13. Delivery Semantics: Exactly-Once/Atomic Broadcast Reactive Composition with Kafka ● You can publish messages and they will be delivered one time exactly by one or more receiving application. ● Producer side ○ Producer writes to the log but fails to get the ack. over network. ● Consumer side ○ Consumer consumes message from the log but fails before updating its offset value. Source: https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f
- 14. Delivery Semantics: Exactly-Once/Atomic Broadcast Reactive Composition with Kafka ● Producer side ○ Idempotence : Producers can retry until it is successful ○ Kafka will detect the duplicates and remove them ● Consumer side ○ Store the offsets in the same DB as the derived state and update both in a transaction. ○ Write both state updates and offsets together in a way that is idempotent. Source: https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f
- 15. Event Sourcing ● Persist each state changing events of an entity as a sequence of immutable events. ● All such events are stored in an event bus and consumers can derive the state by processing a sequence of events. Shipping
- 16. ● Stream Processing in Microservices ○ Services that execute a business logic against a sequence of events/data elements made available over time. ● Kafka Streams : a client library for building stream processing on top of Kafka. Reactive Composition with Kafka Kafka Streams
- 17. ● Command Query Responsibility Segregation. ● Split the common data model into query and command models. ● Kafka Stream - Interactive queries. CQRS Command Source: https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Query
- 18. ● Kafka does not look at your data but you need consistent across your usage. ○ Avro, ProtoBuf, JSON, XML, ASN.1 ○ Avro preferred schema for Kafka, Kafka schema registry ● Schema in practice ○ Associate an Avro schema with a topic. ○ Declare requirements for data that is produced into the topic ○ Provides the instructions on how to interpret data read from the topic Using Schemas - Avro and Schema Registry
- 19. ● A tool for scalably and reliably streaming data between Apache Kafka and other data systems. Reactive Composition with Kafka Kafka Connect Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/
- 20. Kafka, Kafka Connect and Kafka Streams Source: https://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams/
- 21. Active or Reactive? ● Active Composition ○ Ideal for interactive services ○ Service are NOT autonomous ● Reactive Composition ○ Ideal for asynchronous interactions ○ Composition logic is opaque
- 22. Hybrid of Active and Reactive Composition Consumer 1 API Service P API Service Q API Service R API Service S Consumers Consumer 2 Consumer 3 μ Service A Event Bus Proprietary & Legacy Systems Web API / SaaS API Services/ Edge Services Composite Service/ Integration Services Core Services/ Atomic Services μ Service I μ Service H μ Service J μ Service E μ Service C μ Service D μ Service G μ Service B API Management
- 23. ● Create a queryable materialized view using Kafka Streams or a database with Kafka Connect. Hybrid composition Bridging request-response and event driven messaging Source: https://github.com/confluentinc/kafka-streams-examples/tree/5.0.0-post/src/main/java/io/confluent/examples/streams/microservices
- 24. ● A queryable materialized view using Kafka Streams’ interactive queries. Hybrid composition Bridging request-response and event driven messaging
- 25. Implementation Technologies ● Java ○ All of the Kafka, Streams and Connect APIs are supported. ○ Confluent microservices example. ● Go, Python, .NET ● Ballerina (ballerina.io) ○ Cloud-native programming languages that makes it easy to write microservices that integrate APIs. https://ballerina.io/learn/by-guide/messaging-with-kafka/ ● Integration Frameworks: Apache Camel, Spring Boot, WSO2 Integrator etc.