Introduction
To achieve high performance system need to execute multiple tasks in parallel. It overall increases the throughput and makes system highly responsive. Not all operations should be asynchronous but there are many opportunities where asynchronism can be a better choice. For example, after sending email you can immediately send another without waiting first one to be delivered this is asynchrorous operation.
Message Queue
Message queue is used to achieve asynchronous communication between the components of the system. Producer sends the message to the 'Queue' and at the end of the queue Consumer receives it and processes as per the capacity.
We can consider it as a buffer where sent messages are kept until receiver processes them. Messages are processed in FIFO (First in first out) order by default but can also be configured to behave differently for example, priority based processing.
Components
- Producer (Sends the message)
- Queue (Stores the message)
- Consumer (Receives the message)
Architectures
- Point to Point (One producer one consumer)
- Pub/Sub (One producer (publisher) multiple consumers (subscribers))
Examples
- Kafka
- SQS
- RabbitMQ
Streaming
Streaming is a process of continuous transfer of bytes over the network in real time. Streaming is generally setup with message broker and stream processing component. Audio, Video, Data and Event can be streamed.
Examples
- Producer: Application Logs, IoT Sensors, Financial Transactions
- Message Broker: Apacke Kafka, RabbitQ, AWS Kinesis
- Stream Processing: Apache Kafka Stream, AWS Kinesis, Spark Streaming
- Consumer: Database, Data Lakes
.png){kind=small}
.png){kind=small}
