SESSION details

Name: Processing Docker Containers Logs and Metrics at Scale

Date/time: Thursday, October 5, 2023, 11:35 AM - 12:20 PM Pacific Time


Containerization has transformed the software development landscape, and deploying applications is one part of the journey; the need for continuous monitoring and processing of logs and metrics in large-scale environments has become a critical challenge. In this session, we will dive deep into how logging works for Docker containerized applications, the pros and cons of filesystem versus network-oriented events under different transports and formats, the complexity of multiline log messages, and how to deal with them. In addition, we will demonstrate why in-motion log parsing and filtering are essential for scalable environments and the best practices to move that information reliably to different backends for data analysis that considers various strategies to deal with back pressure, network outages, data loss, and general unresponsive systems. From the tooling aspect, we will use Fluent Bit, an open-source telemetry agent, to demonstrate the data collection and processing aspects applied to containerized environments and general best practices for a healthy logging layer. The target audience for this session includes DevOps engineers, system administrators, developers, and anyone involved in cloud-native environments looking for a deep understanding of applied telemetry to deal with highly scalable environments. By implementing the strategies and techniques shared in this session, attendees enhance the operational efficiency and stability of their logging layers