Public subscribe system

Pub Sub

• Publisher and Subscriber

• To handle many to many situations

• can swap the server side API easily with these middle layers

• Kafka, RabbitMQ

• improve performance through asynchronous dealing with the tasks, which can extract tasks with longer processing times

Requirements

Functional requirements

• Create a topic: The producer should be able to create a topic.
• Write messages: Producers should be able to write messages to the topic.
• Subscription: Consumers should be able to subscribe to the topic to receive messages.
• Read messages: The consumer should be able to read messages from the topic.
• Specify retention time: The consumers should be able to specify the retention time after which the message should be deleted from the system.
• Delete messages: A message should be deleted from the topic or system after a certain retention period as defined by the user of the system.

Non-functional requirements

• Scalable: The system should scale with an increasing number of topics and increasing writing (by producers) and reading (by consumers) load.
• Available: The system should be highly available, so that producers can add their data and consumers can read data from it anytime.
• Durability: The system should be durable. Messages accepted from producers must not be lost and should be delivered to the intended subscribers.
• Fault tolerance: Our system should be able to operate in the event of failures.
• Concurrent: The system should handle concurrency issues where reading and writing are performed simultaneously.

First design

• Topic queue: Each topic will be a distributed messaging queue so we can store the messages sent to us from the producer. A producer will write their messages to that queue.

• Database: We’ll use a relational database that will store the subscription details. For example, we need to store which consumer has subscribed to which topic so we can provide the consumers with their desired messages. We’ll use a relational database since our consumer-related data is structured and we want to ensure our data integrity.

• Message director: This service will read the message from the topic queue, fetch the consumers from the database, and send the message to the consumer queue.

• Consumer queue: The message from the topic queue will be copied to the consumer’s queue so the consumer can read the message. For each consumer, we’ll define a separate distributed queue.

  

Second design

High-level design

• Broker: This server will handle the messages. It will store the messages sent from the producer and allow the consumers to read them.
• Cluster manager: We’ll have numerous broker servers to cater to our scalability needs. We need a cluster manager to supervise the broker’s health. It will notify us if a broker fails.
• Storage: We’ll use a relational database to store consumer details, such as subscription information and retention period.
• Consumer manager: This is responsible for managing the consumers. For example, it will verify if the consumer is authorized to read a message from a certain topic or not.

Besides these components, we also have the following design considerations:

• Acknowledgment: An acknowledgment is used to notify the producer that the received message has been stored successfully. The system will wait for an acknowledgment from the consumer if it has successfully consumed the message.
• Retention time: The consumers can specify the retention period time of their messages. The default will be seven days, but it is configurable. Some applications like banking applications require the data to be stored for a few weeks as a business requirement, while an analytical application might not need the data after consumption.