Distributed logging

Requirements

Functional requirements

• Writing logs: The services of the distributed system must be able to write into the logging system.
• Searchable logs: It should be effortless for a system to find logs. Similarly, the application’s flow from end-to-end should also be effortless.
• Storing logging: The logs should reside in distributed storage for easy access.
• Centralized logging visualizer: The system should provide a unified view of globally separated services.

Non-functional requirements

• Low latency: Logging is an I/O-intensive operation that is often much slower than CPU operations. We need to design the system so that logging is not on an application’s critical path.
• Scalability: We want our logging system to be scalable. It should be able to handle the increasing amounts of logs over time and a growing number of concurrent users.
• Availability: The logging system should be highly available to log the data.

Building blocks we will use

The design of a distributed logging system will utilize the following building blocks:

• Pub-sub system: We’ll use a pub-sub- system to handle the huge size of logs.
• Distributed search: We’ll use distributed search to query the logs efficiently.